GLU ISO8583 Protocol Enabler
The GLU.Ware ISO8583 Connectors make use of the opensource JPOS ISO8583 framework. More information can be found on JPOS here. GLU makes use of JPOS which is distributed under the GNU Affero General Public License V3.
JPOS is the de-facto OpenSource implementation of the international ISO8583 standard. JPOS is used in highly demanding production environments in over 80 countries processing literally thousands of transactions every single second, 24x7x365. JPOS provides an open-source software bridge between ISO 8583-based messages exchanged between processing networks and systems. ISO8583 is well suited to real-time, low-latency, and continuous communication scenarios. As such, it serves as the messaging foundation for systems that exchange electronic transactions made for example by cardholders using payment cards.
Connections, Session and Persistence
The GLU ISO8583 connector uses a persistent session-based protocol, it relies on establishing persistent connections meaning that the connection between the client and the server is established once and maintained throughout the processing of multiple transactions. Once a connection is established, the client and server can exchange multiple messages within the same session. The session remains open until either the client or the server terminate it explicitly or if a predefined session timeout occurs. This approach enables a faster and more efficient flow of data and reduces the need for frequent connection setup and teardown.
Multiplexing the ISO Channel with a MUX
Imagine an acquirer implementation that receives several requests at a time from several POS terminals and that has to route those requests to an issuer institution by means of an ISO Channel. While you can establish one socket connection per transaction, it is common practice to setup just one socket connection (handled by an ISO Channel instance) and multiplex it. A ‘MUX’ is basically a channel multiplexer.
The default multiplexing interface used by GLU.Ware is the QMUX component. QMUX is an asynchronous, powerful, Q2 service that implements the MUX interface to Multiplexing an ISO Channel.
In order to match requests to responses, fields that have unique values that are echoed back are used. For QMUX to return the response matching the request, it needs to know which fields are always identical between the request and response. QMUX by default uses DE11 and DE41 (default field11 + field41 is used) as the matching keys.
When a message arrives on a MUX’s underlying ISO Channel, the MUX implementation checks to see if that message’s key is registered as a pending request. Should that key match a pending request, the response is handed to the waiting thread. If the key was registered as a request, or the response comes in too late then that response is (depending on the configuration) ignored, forwarded to an ISORequestListener or to a well-defined Space queue. Under many situations, the same channel that a client application may use to send requests and wait for responses may also receive requests coming from the remote server.
Here’s how the QMUX works:
- Device Connections: The QMUX manages the communication channels (connections) between the GLU.Engine Client application and the Server-side system. It abstracts the low-level communication details and provides a higher-level API for the application to interact with the devices.
- Queue Management: Each device connected to the QMUX has its own input and output queues. These queues ensure that communication between the application and the devices is managed in an orderly and efficient manner, preventing data loss and ensuring proper synchronization.
- Asynchronous Processing: JPOS follows an event-driven architecture. When a device generates an event (e.g., a barcode is scanned, a paper roll is low), the corresponding event handler in the application is notified asynchronously. The QMUX manages these events and ensures they are properly dispatched to the application’s event handlers.
- Concurrency: The QMUX handles concurrent access to the device queues and ensures that requests from multiple threads are processed correctly and efficiently.
- Device Independent: The QMUX provides a unified API for interacting with various types of devices. This abstraction allows developers to write applications that are not tightly coupled to specific device implementations.
MUXPools
As an alternative to the QMUX component, GLU Connectors support MUXPools. MUXPools provide failover as well as load balancing options at the MUX level. The mechanism desired is referred to as a Strategy.
Client side MUXPools establish connections to the Server. If you have set MUXPools to be = 4, then the Client side will create and use up to 4 connections in parallel. These can all connect to the same IP/Port on the Server side. Each MUXPool connection is like a pipe. The GLU.Engine will use those pipes to process transactions within, the pipes once connected stay open to process transactions within until they are disconnected.
Disconnects can be triggered by the GE (client side) or by the Server side.
In the Q2 log if you see:
- <io-timeout/> … that means the GLU.Engine is initiating a connection closing event.
- <peer-disconnect/> … that means the GLU.Engine has received a connection closing event from the far side.
The GLU.Engine could trigger a connection to be closed if for example the timeout threshold is reached.
When configuring your GLU.Engine with an ISO outbound connector, you can set the timeout for those connections. That timeout setting will be applied to all 4 connections. If there is no traffic over a specific connection for longer than the timeout setting, the GE will disconnect that MUXPool connection. It will re-establish that connection within 5 seconds.
To ensure there is always a Client-Sever connection active, in particular if you have low traffic volumes, you can use an application level Echo message (0800/0810’s) and / or a network level Keep Alive to maintain the Client – Server connection. The frequency of your Echo messages should be configured to ensure that connection timeouts don’t occur.
If using an Echo message along with say x4 MUXPools, the Echo message will be sent on any of the x4 available MUXPool connections. A successful response, with no other traffic being processed will result in that MUXPool connection remaining active until the expiry of the GLU Connector Message Timeout setting (defaulted to 45s). Thereafter that MUXPool connection will timeout.
In the meantime, if your Echo message delay is say 55s (it is recommended to make this repeat frequency lower than the connector timeout) so that the next Echo message will be sent on any of the available active MUXPool connections. When a MUXPool connection times out, the system will re-establish the connection within 5 seconds. The Echo messages will ensure that there is always at least one active MUXPool connection.
MUXPool Strategies
PRIMARY SECONDARY (this is the default strategy)
- This strategy divides the MUX channels into two groups: primary and secondary. The primary channels are used for processing normal transaction requests, while the secondary channels serve as a backup for handling fallback scenarios. When a transaction is initiated, the MUXPool will attempt to use a primary channel first. If a primary channel is available and working, the transaction is processed using that channel. However, if a primary channel is unavailable or fails to respond within a specified timeout, the MUXPool will switch to using a secondary channel to process the transaction.
- Typically, the primary and secondary channels are connected to different backend systems or network paths. The idea is to provide redundancy and ensure that transactions can be processed even if there is a failure in one of the primary channels.
ROUND-ROBIN
- This strategy is a load-balancing strategy that distributes transactions evenly across the available MUX channels in a round-robin fashion. When a transaction is initiated, the MUXPool selects the next available channel in a cyclic order to process that transaction. Each new transaction is assigned to the next MUX channel in the sequence, creating a balanced distribution of transactions across all channels.
- This strategy that distributes transactions evenly across the available MUX channels in a round-robin fashion. A MUX Pool is a collection or pool of multiple MUX Channels. The MUX Pool is responsible for managing these channels, making them available for transaction processing, and ensuring that the transactions are evenly distributed or processed based on the defined strategy. The number of MUX Channels in a MUX Pool is determined by how you initialise and configure the MUXPool class in your application. When creating a MUX Pool instance, you have the flexibility to specify the number of MUX Channels you want the pool to manage.
ROUND ROBIN WITH OVERRIDE (BETA – Contact GLU.Support prior to using this Strategy)
- To be used if you need to force completions and reversals to go out the same route as their corresponding originals.
SPLIT BY DIVISOR (BETA – Contact GLU.Support prior to using this Strategy)
- By using a splitField, ISO Messages can be distributed to different channels within the Pool.
If making Strategy selection by configuration from the GLU.Console the entry format is handled, however if making changes directly in the GLU.Engine application.yml then you need to set the Strategy option as one of the below, note these are case and format is sensitive. The Default strategy must be in CAPS, others all lower-case i.e., options are as follows:
- PRIMARY-SECONDARY: Use this option for primary-secondary strategy.
- round-robin: Use this option for round-robin strategy.
- round-robin-with-override: Use this option for round-robin strategy with override.
- split-by-divisor: Use this option for split-by-divisor strategy.
For further context on MUXPool see: http://jpos.org/doc/javadoc/org/jpos/q2/iso/MUXPool.html
Channels, Connections, Keep Alive’s and Echo’s
Given the session based nature of ISO8583 protocol, ISO Server and Client systems employ two mechanisms to maintain session connections that might otherwise expire / end when there is no other traffic being processed.
Channel:
A channel in jPOS represents a logical connection to a remote endpoint, typically a payment terminal or another financial system. It abstracts the communication protocol being used (e.g., ISO 8583) and provides an interface for sending and receiving messages. Channels handle the encoding and decoding of messages and manage the flow of messages between jPOS and the remote endpoint.
TCP Connection:
A TCP (Transmission Control Protocol) connection is a lower-level network connection that provides a reliable, ordered, and stream-oriented communication between two devices over an IP network. In jPOS, a TCP connection refers to the underlying network connection established between the jPOS instance and the remote endpoint (such as a payment terminal or another financial system) to exchange ISO 8583 messages.
A single Channel can manage multiple TCP Connections simultaneously.
Keep Alive:
Primarily a network-level mechanism to prevent connection closures due to inactivity by periodically exchanging messages at the network level. The addition of the Keep Alive is configured in Step 2 when configuring your ISO8583 Connector.
Echo:
An application-level mechanism to validate the operational status of a remote component by sending a message and expecting an identical response over a Channel.
Both mechanisms can be beneficial in different scenarios and can even be used together to ensure both network-level stability and application-level operability in a financial transaction system.
Connector Settings
Packagers
An ISO8583 packager is responsible for encoding the data into the correct message format before sending it over the network, and for decoding the received message into usable data. It also performs tasks such as error checking, data validation, and data formatting to ensure that the message is transmitted accurately and efficiently.
The diagram below illustrates the GLU ISO8583 Connector components and Packager functions.
ISO8583 Packagers handle the following:
- Message Structure Definition: A packager defines the structure of an ISO 8583 message, specifying the layout and order of fields within the message. ISO 8583 messages consist of a fixed set of data elements, each representing different aspects of the transaction, such as the transaction type, amount, cardholder data, and more.
- Data Encoding and Decoding: When you configure an ISO 8583 message in your GLU.Engine application, the packager is responsible for encoding the data from the application’s fields into the binary format required by the ISO 8583 standard. Conversely, when an incoming ISO 8583 message is received, the packager decodes the binary data into a structured format that the application can understand.
- Field Formatting and Validation: The packager handles data formatting and validation for each field in the ISO 8583 message. It ensures that data is correctly formatted according to the standard’s specifications and performs basic checks to ensure data integrity and consistency.
- Reusable Templates: The packager allows you to define reusable templates for various message types. This means you can define the structure and format of different ISO 8583 message types once and reuse them across your application, reducing redundancy and improving maintainability.
- Customisation: While ISO 8583 defines a set of standard fields, financial institutions and payment processors often have specific requirements that go beyond the standard. Packagers can thus be customised to accommodate these specific needs by adding or modifying fields as necessary.
It is important to understand what packager the system you are connecting to uses, GLU supports a wide range of packagers that can be selected from the Console. Support for bespoke / custom packagers is also provided via support request.
Channel Type
Refers to the type of communication channel used for transmitting ISO8583 messages between systems.
Send Message Length
Adds the length of the message to the request.
ISO8583 Messages
Data elements in an ISO8583 payload are identified by their position within the message string. The position of each data element within the message is specified by its field number and its position within that field.
Sample Raw ISO8583 Message Payload
This is an incoming MTI 1200 ISO8583 message with binary values on the left and the message string on the right. When unpacked, this MTI 1200 looks like this:
Those values can be seen in the message string. GLU.Ware needs to be configured to unmarshall those values so the payload can be unpacked.
Sample ISO8583 MTI 0800 Message Payload:
0000 30 38 30 30 82 38 00 00 00 00 00 00 04 00 00 00 0800.8……….
0010 00 00 00 00 30 33 32 32 31 31 30 35 34 30 32 37 ….032211054027
0020 30 35 31 35 31 33 30 35 34 30 30 33 32 32 33 30 0515130540032230
0030 31 1
An ISO8583 message is made of the following parts:
- Message type indicator (MTI)
- One or more bitmaps, indicating which data elements are present. It consists of primary bitmap and secondary bitmap. This isn’t human readable.
- Data elements, the actual information fields of the message- can be readable.
The placements of fields in different versions of the standard varies; for example, the currency elements of the 1987 and 1993 versions of the standard are no longer used in the 2003 version, which holds currency as a sub-element of any financial amount element.
https://www.ibm.com/docs/en/integration-bus/10.0?topic=formats-iso8583-messaging-standard
Unmarshalling of ISO parameters from the above data element section of an ISO payload relies on the specified position of the parameter in the data payload. The above data payload prints out with this value:
- 03221105402705151305400322301
This is unmarshalled in GLU by specifying the position and length of each parameter. Field lengths are specified in the packager:
<isofield id=”7” length=”10” name=”Date and time, transmission” class=”org.jpos.iso.IFA_NUMERIC”/>
<isofield id=”11” length=”6” name=”Systems trace audit number” class=”org.jpos.iso.IFA_NUMERIC”/>
<isofield id=”13” length=”4” name=”Date, Effective” class=”org.jpos.iso.IFA_NUMERIC”/>
Example above unmarshalls as follows:
- Field[(7) Value (0322110540)] to Param :[transmissionDateAndTime]
- Field[(11) Value (270515)] to Param :[systemTraceAuditNumber]
- Field[(12) Value (130540)] to Param :[timeLocalTransaction]
- Field[(13) Value (0322)] to Param :[dateLocalTransaction]
- Field[(70) Value (301)] to Param :[NetMngInfoCode]
Unmarshalling Example in GLU.Console
Field 11 has a Field Number and some Field Data Settings:
Note: Subfields are referenced as field.subField: 63.2
This tells the GE that the transmissionDateAndTime parameter comes from Field #7 and is 10char.
This gives us the value above: 0322110540
That leaves the remainder of the payload to be broken down into the remaining parameters.
The parameters can then be used in requests to other systems in different protocols/content types as needed. Standard handler options are available for these values if required.
Unmarshalling from Content of ISO8583 values
Further unmarshalling of the contents of an ISO8583 value is also possible. For example, XML sent inside ISO8583 can be unmarshalled by the path of the required value.
Example:
<isomsg direction="incoming">
<!-- org.jpos.iso.packager.GenericPackager[packagerISO8583.xml] -->
<field id="0" value="1210"/>
<field id="3" value="310000"/>
<field id="4" value="000000002000"/>
<field id="7" value="0308134849"/>
<isomsg id="62">
<!-- org.jpos.iso.packager.GenericSubFieldPackager -->
<field id="3"><![CDATA[18IFSFData3584<IFSFData xmlns:ns0="http://www.nrf-arts.org/IXRetail/namespace">
<ProductSets>
<MessageText>~0011* Terms and Conditions Apply ** For queries contact Us ** Customer Care 0860 30 40 50 *\</MessageText>
<BalanceOfPoints>000000020</BalanceOfPoints>
<PointsForTransaction>00000020</PointsForTransaction>
<BasePointsEarned/>
<AvailablePointsBalance>0000002725</AvailablePointsBalance>
<NewPointsBalance>0000002725</NewPointsBalance>
<PointsToMoneyRatio>000100</PointsToMoneyRatio>
<RedemptionRefNumber>167827</RedemptionRefNumber>
</ProductSets>
</IFSFData>]]></field>
</isomsg>
<field id="63" value="1CLRG4112033663407SLEV"/>
</isomsg>
The path for RedemptionRefNumber is IFSFData.ProductSets.RedemptionRefNumber- this would return the specific value for that XML parameter: 167827
Marshalling Example in GLU.Console
To respond to an ISO8583 request, the Marshalling is similar for simple payloads:
Specify the Field #:
Specify the parameter to be sent with parameter length:
For parameters with variable length, the value can be left-padded with leading characters (usually) zero OR the length can be calculated and added to the header of that parameter.
Create a value from other parameters:
To build up a single ISO parameter from multiple GLU parameters you add them in sequence and specify the position each value has for the ISO parameter:
This creates an ISO amount format with additional ISO info: accountType, currency etc:
- Field[54] : 1002800C000002079748
Broken down from the above:
- 10 = accountType
- 02 = amountType
- 800 =currencyISOUGS (ISO currencyCode)
- C = amountSign (Credit or Debit)
- 000002079748 = amountTransaction – this is an amount of 2079748, left padded with zeros to fill the required char count for the field. This value was previously rounded in the Derived Parameter.
Length Calculation
NOTE: There’s a function for this- see here: https://glu.global/glu-guide/glu-functions/#get-iso-message-with-lengths-function
- Get the Length:
2. Get length of the length (how many char for fieldLength):
3. Marshall these values into the required parameter:
Template:
18IFSFData${lengthField12722Length}${Field12722Length}${Field12722AllData}
Value:
Field[127.22] :
18IFSFData3588<IFSFData xmlns:ns0=”http://www.nrf-arts.org/IXRetail/namespace”><ProductSets><MessageText>* Terms and Conditions Apply ** For queries contact Pick n Pay ** Customer Care 0800 11 22 88 *</MessageText><BalanceOfPoints>46047</BalanceOfPoints><PointsForTransaction>0000215</PointsForTransaction><BasePointsEarned>0</BasePointsEarned><AvailablePointsBalance>0000046262</AvailablePointsBalance><NewPointsBalance>0000046262</NewPointsBalance><PointsToMoneyRatio>000100</PointsToMoneyRatio><RedemptionRefNumber>BP0000049184</RedemptionRefNumber></ProductSets></IFSFData>
- The Blue Text is the Value (AllData)
- The 588 is the length of AllData ((number of characters for the AllData value)
- The 3 is the length (number of characters for the 588 length value)
NOTE: There’s a function for this- see here: https://glu.global/glu-guide/glu-functions/#get-iso-message-with-lengths-function
ISO Exception Handling
The GLU Connector settings (Step 2 when configuring your ISO8583 Connector) provide a configuration option that will turn a property – ignoreISOException – on / off. If ‘on’ the GLU.Engine will ignore ISO Exceptions such that those ISO Exceptions do not cause the GE to drop its connection to the Server.
ISO8583 Log Examples
In addition to the ‘normal’ GLU.Engine logs, since GLU.Ware uses Q2 (the QSP version 2 container within jPOS), Q2 logs generated by GLU.Engines that have ISO8583 connectors. The Q2 logs should be used to trace jPOS Channel activity. These can be used in conjunction with the gluware.log files.
Q2 logs (saved a q2.log files) are generated in an ISO8583 GLU.Engine. This happens before ISO8583 payloads are ingested into the GLU.Engine and prior to those events being printed in the gluware.log.
Any investigation into an ISO8583 issue should start with the q2.log. The q2.log uses the specified packager to validate payloads.
<isomsg direction=”incoming”>
<!– org.jpos.iso.packager.GenericPackager[packagerISO8583.xml] –>
<field id=”0″ value=”1200″/>
<field id=”3″ value=”310000″/>
<field id=”4″ value=”000000002000″/>
<field id=”7″ value=”0308131930″/>
<field id=”11″ value=”167827″/>
<field id=”12″ value=”230308011930″/>
<field id=”22″ value=”123456789112″/>
<field id=”49″ value=”200″/>
<field id=”59″ value=”292″/>
<field id=”63″ value=”1CLRG4112033663407SLEV”/>
</isomsg>
When something goes wrong you might see an error packing/unpacking field X:
<session-error>
<iso-exception>
org.jpos.iso.IFA_LLCHAR: Problem unpacking field 12 (org.jpos.iso.ISOException: Field length 98 too long. Max: 25) unpacking field=127
This is saying Field 127.12 is too long. This can be misleading – if any of the fields sent have different length compared to what the packager is expecting, the first field to fail will print in the log. In this case, an extra Field was being sent- this skewed the unmarshalling/packager validation of subsequent Fields.
The best troubleshooting approach is to get the request message that was sent to compare it to what the packager supports/requires and what the config in the GLU.Engine is looking for. If the 3rd party is anable to change their request, the packager can be updated to cater for their payload.
If the extra parameter is required for further use in the GLU.Engine, it can be unmarshalled. If it’s not required, the GLU.Engine can pass the value through without unmarshalling.
If there is no error in the q2.log, troubleshooting can continue with the gluware.log:
Standard Unmarshalling, Handler, Derived values and protocol translation issues or errors will print in the gluware.log.