The ‘Go to Configuration’ button will take the User directly to the relevant Integration to enable the User to address the Warnings prior to re-attempting the ‘Build’ process.

There are times when some warnings may not be of material concern to the User. For example if the warnings relate to a transaction that is not the primary focus at that point in time or it may still be under configuration … whereas other transactions in the Build are not throwing any warnings. In this case the option to ‘Proceed’ can be used to create the GLU.Engine Build not-withstanding the ‘Warnings’. In this way the User can proceed with testing the ‘working’ transactions within the Build but those associated with the ‘accepted’ “Warnings’ will likely fail.

It must also be noted that some warnings relating to Environments, Connectors, Auth Collections or Static / Reusable Parameter Collections may impact all transactions within a GLU.Engine. Some of these warnings will prevent a build from occurring. As such proceeding with building a GLU.Engine that is flagging ‘warnings’ is generally not advised as there is no certainty that any transactions will actually work.

In other words – if there are ‘Warnings’ – ‘Proceed’ with your Build with caution.

Throughout the GLU.Console – validation logic has been added to enable configuration errors to be ‘flagged’ to the user. These ‘flags’ are reflected as ‘Warnings’, a summary of all ‘Warnings’ can be seen by using the ‘Warnings’ button within the tool options at the top of the Configuration screens. See example in the screenshot below:

Warnings are generated in near real-time i.e. as configurations are added, warnings will pop-up to guide the user. In the example below, the Parameter Name is flagged as ‘mandatory’ due to the * … the warning shows up to prompt the user to add the mandatory field:

Build Manager Validations

Where errors that have not been addressed in the Configuration will be highlighted and listed prior to allowing a ‘Build’ to proceed. Additionally, any errors during the actual ‘Build’ process will be displayed after a build fails (these errors will be visible in the ‘Build History’ – see sample below – for later reference).

The GLU.Console provides the capability to configure in the INFO request and response logs a tag which can be used to trace transactions as they occur. These tags can consist of both hardcoded text labels and/or variables generated in the running GLU.Engine.

In the example below we show how 3 tags have been setup for each API. In the log below we can see how the tags are reflected in the logs:

As can be seen tags appear on each INFO line after the TraceID.

Also note the first 2 transactions in the log. These show values for the “responseCode“ as this contains a value at this point in the transaction execution. The log filter is set as below for this API:

These tags and values are associated with all INFO Level logs for each unique transaction.

This enables the payloads of Inbound and Outbound Connectors to be included in the INFO Level Logs. By selecting the checkbox to ‘Add to INFO Logs’ within the Request / Response Manager Panels, this data can be controlled to be included or not in the INFO logs. 

Considerations

Writing additional data to logs does have a performance impact which must be kept in consideration. Performance testing following such changes is advised.

Sensitive data that is flagged to be excluded or masked from the logs by unchecking of the ‘Make visible in logs’ setting will apply to the tags.

To alert users before their session expires, a pop-up notification with a countdown timer (20 seconds) appears as the inactivity time approaches the timeout interval. Users have the option to stay connected, refreshing their session, or to log out. If there is no response from the user, the session is terminated, and they are automatically logged out.

Note: Any configuration changes not saved will be lost. It is important to regularly save configuration changes that you want to keep to avoid potentially losing configuration work that you may be busy with.

If you encounter an error on the login page stating “User is currently logged in” when trying to log back in after a session timeout, you can resolve this by forcing your user to be logged out using the following URL: https://www.console.glu.global/glu-console/logout.

Startup Warning

If your engine starts up, but there is a warning which looks like this:

This is likely to be linked to the wrong java version running. Please see here GLU.Engine Server Specifications for the correct java version. 

Startup Error

This error will occur if the port value is missing in the database connector. See below screen shot from the connector screen:

You will see this error in the Database logs:

To resolve, simply update your Connector configuration to include the Port

The logging level of a GLU.Engine can be altered through three mechanisms.

The first method involves using the Application Settings within the GLU.Console, however, this requires a complete rebuild and redeployment of the GLU.Engine.

The second method allows for the logging level to be altered without the need for a rebuild and redeployment. This is done through the use of the “changelogginglevel.sh” script located in the GLU.Engine deployment directory:

This shell script actually makes use of one of the GLU.Engine APIs to make the change.

This is the third method, which is to call the JMX APIs directly on the server port. This method should be used when using a docker container to run the GLU.Engine.

Changing Log Level via the GLU.Console

In Build Manager, Select GLU.Engine Settings: 

Edit “logging” to the desired level:

The “Path“ and “Name” of the GLU.Engine logs can be altered to meet specific requirements. To specify the log path as the root directory of the GLU.Engine folder, utilise a full stop “.” in the relevant configuration setting “Path“.

Why have different Log Levels?

The logging framework utilised by GLU, the de-facto standard in the Java world, classifies log messages into five categories:

• ERROR
• WARN
• INFO
• DEBUG
• TRACE

These log levels are assigned based on the urgency of the log message, allowing log filtering by level of importance.

In a production environment, it is crucial to have an efficient filtering mechanism for log messages to quickly identify urgent issues leading to potential losses. Mixing urgent log messages with non-urgent ones which hampers efficient log analysis.

It is important to understand when to categorise log messages into each level of urgency.

ERROR

The ERROR level should only be used when the application really is in trouble. Users are being affected without having a way to work around the issue.

Someone must be alerted to fix it immediately, even if it’s in the middle of the night. There must be some kind of alerting in place for ERROR log events in the production environment. Often, the only use for the ERROR level within a certain application is when a valuable business use case cannot be completed due to technical issues or a bug.

Take care not to use this logging level too generously because that would add too much noise to the logs and reduce the significance of a single ERROR event. You wouldn’t want to be woken in the middle of the night due to something that could have waited until the next morning, would you?

WARN

The WARN level should be used when something bad happened, but the application still has the chance to heal itself or the issue can wait a day or two to be fixed.

Like ERROR events, WARN events should be attended to by a dev or ops person, so there must be some kind of alerting in place for the production environment.

A concrete example for a WARN message is when a system failed to connect to an external resource but will try again automatically. It might ultimately result in an ERROR log message when the retry-mechanism also fails. The WARN level is the level that should be active in production systems by default, so that only WARN and ERROR messages are being reported, thus saving storage capacity and performance.

If storage and performance are not a problem and our log server provides good search capabilities we can actually report even INFO and DEBUG events and just filter them out when we’re only interested in the important stuff.

INFO

The INFO level should be used to document state changes in the application or some entity within the application. This information can be helpful during development and sometimes even in production to track what is actually happening in the system.

Concrete examples for using the INFO level are:

• the application has started with configuration parameter x having the value y
• a new entity (e.g. a user) has been created or changed its state
• the state of a certain business process (e.g. an order) has changed from “open” to “processed”
• a regularly scheduled batch job has finished and processed z items.

DEBUG

DEBUG level logging shows the detail of every transactions.

Including:

• error messages when an incoming HTTP request was malformed, resulting in a 4xx HTTP status
• variable values in business logic.
• Transaction details
  

Due to performance implications DEBUG mode should be used in Production environments sparingly. Sparingly means 2-3 hours in a controlled test, and switched back to INFO or higher straight after.

TRACE

Compared to DEBUG, it’s pretty easy to define what to log on TRACE. As the name suggests, we want to log all information that helps us to trace the processing of an incoming request through our application.

This includes:

• start or end of a method, possibly including the processing duration
• URLs of the endpoints of our application that have been called
• start and end of the processing of an incoming request or scheduled job.
  

Log File Settings & Limits

Multiple methods are available for modifying the logging settings within the GLU.Engine:

• All settings can be set through the console when building through the  GLU.Engine Settings.
• A subset of settings can be configured in the applicationSettings.yml file.
• A further subset can be set through the GLU.Engine APIs.

It is important to note that changes to log levels within the GLU.Engine can have a significant impact on system performance. As such, it is recommended to only run the GLU.Engine in DEBUG mode during periods of low traffic, or to increase system resource allocations to compensate for the increased logging output. The utilisation of the DEBUG mode will result in the generation of substantial log files, which can rapidly consume the allotted log storage space. Given these considerations, it is not advisable to operate the GLU.Engine in a production environment under heavy load for extended durations.

Log file Settings

The GLU.Engine generates a single log file, with historical data being transferred to older log files. The attributes of the file, including the naming convention, can be regulated.

The dynamic name, can be customised in the Application Settings file.

The log file is capable of being processed by any log management tool that can handle text-based log files. GLU has ratified the consumption of GLU.Ware logs through the use of the Elastic Stack, Dynatrace, and AWS Cloudwatch.

See GLU.Analytics Deployment Guidelines for an example of how to integrate to Elastic Stack.

The integration screens provide the capability to configure response payloads to be displayed in the logs as a single line. This enhances the readability of the text-based log files.

See  Masking Sensitive Data/Parameters in Logs section for detail on how to hide any sensitive data from your Logs.

ISO8583 – Q2 Logs

GLU uses the Q2 component to generate ISO8583 logs to provide more detailed information about the processing of ISO8583 messages than is available in the standard GLU.Engine logs. The Q2 logs contain information such as the message type, message format, message fields, and any errors that occurred during processing.

Q2 logs are useful for troubleshooting issues with ISO8583 integrations using the Q2 protocol. They can be used to identify errors in message construction, data formatting, or other issues that may be causing problems with the integration.

The appSettings.yml file

It is possible to configure the various settings in a file which is include with the GLU.Engine.

The file can be found here:

../engine/config/applicationSettings.yml 

If this is changed then this will require the GLU.Engine to be restarted for settings to be applied. The table below describes the parameters which can be changed in the file.

Log file Detail

The GLU.Engines produce 1 types of log file. 

GLUware.log

The Initial lines in the log file will contain the information in the table below. Once the GLU.Engine is running it is also possible to pull the same information through a GLU API request see Engine Info.

Log Levels

See the GLU.Engine Log Levels page for details.

As an integration expert, these are the steps I would follow to troubleshoot an integration:

1. Gather Information: The first step is to gather as much information as possible about the problem. This includes:
   • GLU.Engine configuration and build version details.
   • GLU.Engine logs in DEBUG mode, for ISO8583 be sure to gather the Q2 logs too.
   • Postman Test Pack with sample messages to enable the issue to be repeated in a test environment.
   • Collect information about the transaction that is causing the issue, the message format, the communication channel used, and any error messages that are generated.
   • If errant payloads have elements that are encrypted, one should get those sample messaged decrypted to enable parameter values to be understood. Sometimes a failure may be caused by a parameter value that breaks a validation rule.
   • Documentation such as API Specs, solution design diagrams, Sequence diagrams etc.
   • Time of day may also be relevant, is there any unusual system behaviour happening at the time the issue is observed (e.g. peak load or perhaps system maintenance activities).
2. Analyse the Data: The next step is to analyse the data that was gathered in step one. This includes reviewing the message payloads and checking for any inconsistencies or errors. It is important to understand how the message was constructed, what data was sent, and what data was received.
3. Check the Network Connection: The third step is to check the network connection between the two systems. This includes reviewing network logs, checking for packet loss, and testing the connection to ensure that it is stable and reliable.
4. Verify the Message Format: The fourth step is to verify that the message format is correct. This includes checking the message structure, message type, message class, and other relevant parameters / fields. If you are experiencing protocol-related issues, such as incorrect message formats or sequencing, TCPdump is a command-line packet analyser that can help you identify the source of the problem. By capturing and analysing the network traffic, you can determine if the issue is related to the protocol itself or to the specific implementation of the protocol in your system.
5. Check the Configuration: The fifth step is to check the configuration of both systems. This includes reviewing system settings, ensuring that the correct protocols and ports are being used, and verifying that the necessary libraries and drivers are installed. Check that the message format, data elements, and fields are correctly configured.
6. Perform functional testing: Conduct functional testing to identify any issues in the message flow. Verify that the message is being sent and received correctly and that the message content is correct.
7. Collaborate with Stakeholders: The sixth step is to collaborate with stakeholders, including developers, system administrators, and end-users. This includes sharing information, discussing possible solutions, and determining the best course of action to resolve the issue.
8. Test the Solution: The seventh and final step is to test the proposed solution to ensure that the issue has been resolved. This includes verifying that the message is now being processed correctly and that all data is being transmitted and received as expected.
9. Document findings: Document your findings and the steps taken to resolve the issue. This documentation can be used as a reference for future troubleshooting efforts.

Diagnostic Tools

Here are some diagnostic tools that you can use to troubleshoot system integration issues:

1. Wireshark: Wireshark is a popular network protocol analyser that can capture and analyse network traffic in real-time. You can use it to analyse the message flow and identify any issues with the message format, data elements, and fields.
2. Log analysis tools: There are many log analysis tools available, such as Splunk, ELK Stack, and Logstash. These tools can help you analyse the system logs and identify any errors or exceptions that may have occurred during the transaction.
3. Performance monitoring tools: Performance monitoring tools such as Nagios, Zabbix, and Prometheus can help you monitor system performance and identify any issues that may be affecting system performance.
4. Network monitoring tools: Network monitoring tools such as PRTG, SolarWinds, and Nagios can help you monitor network performance and identify any issues that may be affecting network performance. TCPdump is a command-line packet analyser that is used to capture and analyse network traffic. If you suspect that there is a problem with the network connectivity between systems, TCPdump can help you identify the source of the problem. By capturing and analysing the network traffic between systems, you can determine if packets are being dropped, delayed, or corrupted, and then take the appropriate corrective action.

By using these diagnostic tools, you can effectively troubleshoot system integration issues and resolve them in a timely and efficient manner.

Timeout Troubleshooting

When troubleshooting integration issues where timeout errors are suspected, the following steps and tools can be used to assist the troubleshooting process:

1. Identify the Symptoms: Identify the symptoms of the timeout error. This can include slow response times, failed requests, or error messages related to timeouts. It is important to understand the nature of the issue and the specific symptoms that are being experienced.
2. Check Network Connectivity: Check the network connectivity between the systems. This can include checking for packet loss, latency, or other network issues. Tools such as ping, traceroute, or network diagnostic tools can be used to identify and diagnose network connectivity issues.
3. Review System Logs: Review system logs to identify any errors or warning messages related to the integration. This can include logs from both the sending and receiving systems as well as the GLU.Engine logs. It is important to identify any errors or warning messages that may be related to the timeout issue.
4. Check Timeout Settings: This can include both the sending and receiving systems, as well as any middleware or other components that may be involved in the integration. It is important to ensure that timeout settings are configured appropriately and are not set too low.
5. Assess System Performance: Performance assessing an integration is not a trivial exercise as it can include setting up a dedicated performance testing environment to monitor CPU usage, memory usage, and disk I/O when to simulated high volumes of traffic are being processed. Load testing tools can be used to simulate various load scenarios and identify any performance or scalability issues that may be causing the timeout errors.Performance monitoring tools can be used to identify any bottlenecks or performance issues that may be causing the timeout errors. There is more detail on how to performance test your GLU.Engines here and here.
6. Work with Vendors: The final step is to work with vendors or service providers to identify and resolve any issues related to the integration. This may involve working with the vendor’s support team or development team to identify and resolve any issues that may be causing the timeout errors.

ISO8583 Troubleshooting

Here are some diagnostic tools that you can use to troubleshoot ISO8583 system integration issues:

ISO8583 message viewer: An ISO8583 message viewer is a tool that can help you view and decode ISO8583 messages. It can help you understand the message content and identify any errors in the message flow. Here are some ISO8583 message viewer tools:

1. ISO8583 Message Decoder by Sarel: This is a free web-based tool that allows you to decode and view ISO8583 messages in various formats such as HEX, ASCII, and BINARY. It supports a wide range of ISO8583 message versions and can display detailed information about each field.
2. MessageWare ISO8583 Analyser: This is a paid software tool that provides a user-friendly interface for analysing ISO8583 messages. It can decode and display the message content, perform field validation, and highlight any errors or issues in the message.
3. ISO8583 Message Editor by Sarel: This is a free web-based tool that allows you to create, edit, and send ISO8583 messages. It supports a wide range of ISO8583 message versions and can generate messages in various formats such as HEX, ASCII, and BINARY.
4. ISO8583 Toolkit by Proxymitron: This is a paid software tool that provides a comprehensive set of ISO8583 tools for message creation, editing, and analysis. It includes a message editor, message analyser, message simulator, and message validation tool.
5. ISO8583 Message Viewer by Pantor: This is a free and open-source software tool that allows you to view and analyse ISO8583 messages. It supports a wide range of ISO8583 message versions and can display detailed information about each field.
6. TCPdump can be used to capture and verify the integrity of the binary messages being sent and received as ISO8583 messages. By analysing the binary data in the captured packets, you can ensure that the messages are being transmitted correctly and are not being altered in transit.

ISO8583 simulator: An ISO8583 simulator can help you simulate transactions and test the message flow. It can help you identify any issues with message routing, message format, and message content.

There are several ISO8583 simulator tools available, each with their own unique features and capabilities. Here are some popular ISO8583 simulator tools that you can consider:

1. Postman: Postman is a popular API development tool that can also be used as an ISO8583 simulator. It allows users to create and send ISO8583 messages using its API testing capabilities.
2. JPOS: JPOS is a Java-based framework that provides ISO8583 message processing capabilities. It also includes a simulator tool that allows users to create, send, and receive ISO8583 messages.
3. ISO8583 Simulator: ISO8583 Simulator is a standalone simulator tool that can be used to create, send, and receive ISO8583 messages. It supports various message types and can be configured to simulate different network scenarios.
4. M365 ISO8583 Simulator: M365 ISO8583 Simulator is a web-based simulator tool that can be used to test ISO8583 message processing. It supports various message types and can be customised to simulate different network scenarios.
5. Paragon Testing Tool: Paragon Testing Tool is a comprehensive testing tool that includes ISO8583 message processing capabilities. It allows users to create, send, and receive ISO8583 messages and also includes advanced testing features such as load testing and stress testing.

These are just a few of the many ISO8583 simulator tools available. It’s important to evaluate your specific requirements and choose a simulator tool that best fits your needs.

ISO8583 Packager Issues

If a sending system uses a different packager than the receiving system, there may be issues with the encoding and decoding of the message. The receiving system may not be able to properly interpret the message because the fields may be in a different order, have different lengths, or be missing altogether.

This can result in errors or rejected transactions, which could cause delays or even financial losses. In some cases, the receiving system may be able to detect the issue and reject the message outright, while in other cases, the message may be processed incorrectly, leading to further issues down the line.

To prevent such issues, it is important to ensure that both the sending and receiving systems use compatible packagers that adhere to the same version of the ISO 8583 standard. This can be achieved by using standardised libraries or tools that are commonly used in the financial industry, or by ensuring that both systems are properly configured to communicate with each other.

Unmatched or corrupt ISO8583 packagers used by sending or receiving systems or by your GLU.Engine, this can result in various issues, including:

1. Message Rejection: The receiver system may reject the ISO8583 messages sent by the sender system because the message format is not correct. This can lead to communication failures between the systems, causing delays and errors in the transaction processing.
2. Data Corruption: If the packager is not configured correctly, it may incorrectly process or interpret the message data, leading to data corruption. This can cause errors in the transaction processing or result in incorrect transaction data being recorded.
3. Misinterpretation of Message Fields: If the packager is not configured correctly, it may misinterpret the message fields, leading to incorrect data being recorded or processed.

SOAP Troubleshooting

Prior to launching the GLU.Engines, ensure that the SOAP connectors are operational. The startup process for the GLU.Engine will pause at each SOAP connector until a connection has been established. In the event that a connection is unavailable, the startup process will be terminated.

Non-sensical log error:

If you see this error in the logs:

or something similar, this means you have incorrect settings in the Application Settings, see Logging level tab – Max file Size, Max History or Total Size Cap.

Performance Troubleshooting

Extracting thread dumps

You will need to install the developers version for the JDK to get jstack & jcmd, use this command on linux environments with yum, or something similar.

“jstack” is an effective command line tool to capture thread dumps.

The Jstack tool is included in JDK since Java 5. If you are running in older version of java, consider using other options. The jstack tool is shipped in: “JDK_HOME\bin” folder.

Here is the command that you need to issue to capture a thread dump:

Where:

• pid : is the Process Id of the application, whose thread dump should be captured.
• file-path : is the file path where thread dump will be written in to.

For example:

As per the example the thread dump process would generate a file here:

Capturing Heap Dumps

“jmap“ is a tool to print statistics about memory in a running JVM. We can use it for local or remote processes. To capture a heap dump using jmap we need to use the dump option:

Along with that option, we should specify several parameters:

• live : if set it only prints objects which have active references and discards the ones that are ready to be garbage collected. This parameter is optional.
• format=b : specifies that the dump file will be in binary format. If not set the result is the same.
• file : the file where the dump will be written to.
• pid : id of the Java process.

An example would be like this:

Remember that you can easily get the pid of a Java process by using the jps command, o

Keep in mind that jmap was introduced in the JDK as an experimental tool and it’s unsupported. Therefore, in some cases, it may be preferable to use other tools instead.