Message Queues

Asynchronous vs. Synchronous Messaging

Asynchronous messaging refers to a communication pattern where a sender and a receiver do not need to be active or available at the same time. In contrast, synchronous messaging requires both the sender and receiver to be available at the same time for a communication to take place.

The use of message queues in asynchronous messaging can provide several benefits over synchronous communication:

Scalability: Asynchronous messaging allows for communication between loosely coupled systems, which can be scaled independently, leading to better overall system scalability.
Resilience: By decoupling systems, message queues provide a buffer between the sender and receiver, allowing the receiver to process messages at its own pace. This can improve system resilience, as it reduces the risk of cascading failures.
Improved performance: Asynchronous messaging allows for non-blocking communication, enabling the sender to continue processing requests while waiting for a response. This can lead to improved system performance and responsiveness.
Increased flexibility: Message queues allow for the creation of flexible and dynamic communication patterns, where messages can be routed to multiple recipients, transformed, and processed in parallel.

Overall, the use of asynchronous messaging and message queues can provide greater flexibility, scalability, and resilience compared to synchronous messaging.

GLU Message Queue Connectors

GLU.Ware supports integration to both ActiveMQ (Produce and Consume) and Rabbit Message Queues (Produce). Often useful for Asynchronous offloading of a task in a flow, MQ’s can help to speed up transaction round-trip times.

The Client DevOps team will be responsible for the MQ setup and configuration as for any other End-Point that the GLU.Engine Connects to as such configuration of the MQ’s is not covered here.

Based on the setup of the MQ Broker, the GLU Connector can be configured to support the appropriate Protocol e.g. TCP or AMQP.

The GLU.Engine will contain all the libraries to facilitate the connections. So as an example for RabbitMQ all the client components are in the GLU.Engine no additional software is required.

Publish/Subscribe Integration Pattern

ActiveMQ and RabbitMQ can also be used in Microservice Publish/Subscribe integration patterns as ‘Topics’ for consumption by Subscribers. GLU has a Roadmap of Pub/Sub pattern Topics that will be added including Amazon SQS, IBM MQ, RocketMQ, GOOGLE PUBSUB, Apache Pulsar and Apache Kafka – please log any requirements for GLU.Ware to priorise support for any of these via the GLU.Support desk.

Publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorise published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.

Publish–subscribe is a sibling of the message queue paradigm, and is typically one part of a larger message-oriented middleware system. Most messaging systems support both the pub/sub and message queue models in their API, e.g. Java Message Service (JMS).

In the publish-subscribe model, subscribers typically receive only a subset of the total messages published. The process of selecting messages for reception and processing is called filtering. In a topic-based system, messages are published to “topics” or named logical channels. Subscribers in a topic-based system will receive all messages published to the topics to which they subscribe. The publisher is responsible for defining the topics to which subscribers can subscribe.

RabbitMQ

RabbitMQ can be configured to either produce a RabbitMQ transaction on a Queue or consume from a RabbitMQ Queue. If a consumer is to be used then this will need to have a connector set up which has OUTBOUND/INBOUND set to BOTH.

Both consumer and producers can be configured so that the settings for the rabbitMQ can be adjusted from the appSettings.yml file.

Connector Setup

Set Protocol to RABBIT_QUEUE. In the Connector Environment Settings panel, set the Username and Password for access to the RabbitMQ.

Producer

When RabbitMQ is set up in a Connector, it will send transactions of the RabbitMQ as a producer.

Use appSettings.yml

To have the configuration setup so it is possible to configure the RabbitMQ from the appSettings.yml file, you will need to tick the “Use Queue Holder” checkbox.

When this is ticked the default queue holder value needs to be setup. This value will be the initial value which is held in the appSetting.yml file and used for the GLU.Engine interactions with RabbitMQ.

The parameters in the default queue string should use “&amp ;” separator values.

Do not use appSettings.yml

To have the configuration setup so it is not possible to configure the RabbitMQ from the appSettings.yml file, you will need to untick the “Use Queue Holder” button.

When this is unticked the Queue field should contain the rabbitMQ queue which the transaction will go to. The Options field should contain the string of characters which define the settings for the transaction going to RabbitMQ.

The parameters in the default queue string should use “&amp ;” separator values.

Consumer

When RabbitMQ is set up as a connector it will receive the incoming transactions.

When the GLU.Engine is run then then this API will listen on the Queue from RabbitMQ and if there is transactions in the Queue it will pull them in and start the orchestration process.

Use appSettings.yml

To have the configuration setup so it is possible to configure the RabbitMQ from the appSettings.yml file, you will need to tick the “Use Queue Holder” button.

When this is ticked the default queue holder value needs to be setup. This value will be the initial value which is held in the appSetting.yml file and used for the integrations interactions with RabbitMQ.

The parameters in the default queue string should use “&amp ;” separator values.

Do not use appSettings.yml

To have the configuration setup so it is not possible to configure the RabbitMQ from the appSettings.yml file, you will need to untick the “Use Queue Holder” button.

When this is unticked the Queue field should contain the rabbitMQ queue which the transaction will go to . The options field should contain the string of characters which define the settings for the transaction going to RabbitMQ.

The parameters in the default queue string should use “&amp ;” separator values

Build Manager Errors

The Build Manager provides an additional layer or ‘error’ assistance. At build time the Build Manager will present a pop-up with the list of ‘Warnings’. See screenshot below.

The ‘Go to Configuration’ button will take the User directly to the relevant Integration to enable the User to address the Warnings prior to re-attempting the ‘Build’ process.

There are times when some warnings may not be of material concern to the User. For example if the warnings relate to a transaction that is not the primary focus at that point in time or it may still be under configuration … whereas other transactions in the Build are not throwing any warnings. In this case the option to ‘Proceed’ can be used to create the GLU.Engine Build not-withstanding the ‘Warnings’. In this way the User can proceed with testing the ‘working’ transactions within the Build but those associated with the ‘accepted’ “Warnings’ will likely fail.

It must also be noted that some warnings relating to Environments, Connectors, Auth Collections or Static / Reusable Parameter Collections may impact all transactions within a GLU.Engine. Some of these warnings will prevent a build from occurring. As such proceeding with building a GLU.Engine that is flagging ‘warnings’ is generally not advised as there is no certainty that any transactions will actually work.

In other words – if there are ‘Warnings’ – ‘Proceed’ with your Build with caution.

GLU.Console Configuration Errors

Integration Builder Validations

Throughout the GLU.Console – validation logic has been added to enable configuration errors to be ‘flagged’ to the user. These ‘flags’ are reflected as ‘Warnings’, a summary of all ‘Warnings’ can be seen by using the ‘Warnings’ button within the tool options at the top of the Configuration screens. See example in the screenshot below.

Warnings are generated in near real-time i.e. as configurations are added, warnings will pop-up to guide the user. In the example below, the Parameter Name is flagged as ‘mandatory’ due to the * … the warning shows up to prompt the user to add the mandatory field.

Build Manager Validations

Where errors that have not been addressed in the Configuration will be highlighted and listed prior to allowing a ‘Build’ to proceed. Additionally, any errors during the actual ‘Build’ process will be displayed after a build fails (these errors will be visible in the ‘Build History’ – see sample below – for later reference).

GLU.Console Sessions and Timeouts

Session timeouts occurs when a user does not perform any action on the GLU.Console during a specified interval. Timeouts when working within the Integration Builder and USSD Menu Builder tools are set to 30 minutes and when working within all other tools is set to 2 minutes. When this happens, the user session status is changed to ‘invalid’ on the server side, and the web server is instructed to terminate it, deleting all unsaved data.

If the inactivity time approaches the timeout interval, a pop-up notification with a count-down timer (20 seconds) will appear, giving the user the option to stay connected (thus refreshing their session) or to log out. If the user does not respond, the session will be terminated, and the user will be logged out.

Note that any configuration changes not saved will be lost. It is important to regularly save configuration changes that you want to keep to avoid potentially losing configuration work that you may be busy with.

If your session times such that when you attempt to log back in again you get an error on the login page which says “User is currently logged in”, you can fix this by forcing your User to be logged out using this URL: https://www.console.glu.global/glu-console/logout

Orchestration Response Handlers

Orchestration Response Handlers enable business rules / logic to be applied to the flow and based on those rules the transaction flow can take different paths or actions can be triggered in the flows to ultimately conclude with the ‘Response’ to the Initiating System.

How do response handlers work?

Handlers can be added to any Inbound API Request or Response or any downstream system Response.

A Handler on an Inbound API Request could for example trigger the routing of the request to another Inbound API on the same GLU.Engine.
A Handler on an Inbound API Response could trigger the flow to be re-routed
A Handler on an API response could based on a parameter matching a certain value overwrite the parameter with another value.

A Handler on the Response to an outbound Connector call is the most widely used mechanism for orchestrating your flows.

Within the Orchestration Manager, Handlers can be configured based on response Content, HTTP Status Codes or response conditions. When the handler conditions specified in the handler are met, the defined Handler Action will be triggered. The appropriate configuration box is displayed depending on the action specified.

The Response Handler can be used to end a transaction or result in a different outcome or Response to the Initiating System.

Handler Conditions

Handlers can be configured to run based on:

Parameter values
HTTP Status Codes
Expressions – used to trigger the processing of a Expression in a flow
Or ‘none’ – if you just want to trigger a routing event at a point in the flow use the ‘None’ category for you Handler.

Important notes on handlers:

Handlers are always executed in order first to last, so ensure your logic handlers are ordered correctly.
The value in the condition cannot contain a function or formula. If required, use a Derived Parameter to apply a function or formula, then reference the Derived Parameter in the Handler Value.
Note that if for example you have a set of 3 status codes all of which should result in the same handler condition being applied, you can list all three codes in the ‘Value’ field e.g. 1000, 1001, 1002

Parameter Handler

If the Parameter specified via the ‘Name’ dropdown box passes the comparison check (or other boolean operation) specified by the ‘Condition’ dropdown box, it’ll trigger the ‘Action’.

HTTP Status Codes

The GLU Engine needs to specifically be told to handle unsuccessful HTTP status codes (e.g. 400, 401, 500 etc). This is done by adding a status codes handler. Without this handler, the GLU Engine is unable to unmarshall the response payload. Ensure that you create the required HTTP status code handlers in order to orchestrate or unmarshall.

None Handler

A ‘none’ handler means that no triggering event, code or parameter value needs to be present but that the handler will ALWAYS be executed.

Handler Action Types

Route To

Routes the request to a specific flow code.

Send to Many Routes

Routes all requests to the specified flow codes.

Parallel Processing: If enabled then sending messages to the recipients occurs concurrently. Note the caller thread will still wait until all messages have been fully processed before it continues. It’s only the sending and processing the replies from the recipients which happens concurrently.
Stop On Exception: If checked Requests will fail if there’s an exception on any of the flow codes. In all situations, the recipient list will stop further processing. This is the same behaviour as in the pipeline, which is used by the routing engine. The default behaviour is to not stop but continue processing till the end.
Timeout: Sets a total timeout specified in ,milliseconds, when using parallel processing. If the Recipient List hasn’t been able to send and process all replies within the given timeframe, then the timeout triggers and the Recipient List breaks out and continues.

Stop Process

This forces the process to stop and the message gets processed back through the Response Manager. Either the Stop Process can trigger a ‘successful’ response back, alternatively, it can trigger an error template. The error response is comprised on an ‘Error Code’ and an optional ‘Error Message’ describing the error in detail. The error code and error response values will be mapped into the parameter values are that are specified as the response parameters as below.

Overwrite Parameter

‘Overwrite Parameter’ Action will always take place at the step in the flow at which the Handler resides BEFORE any other Handler rules are applied such as a ‘ROUTE TO’ redirection handler. This means that even if the sequence of Handlers has ROUTE TO handlers pre-ceding ‘OVERWRITE’ handlers, the ‘OVERWRITE’ handlers will always be run and only thereafter will the ‘ROUTE TO’ handlers be run – in the order in which they are placed in the list of Handlers (top-down).

The parameter can be overwritten with a value or with other parameter, below some examples:

Overwrite parameter with a value, e.g. text: HelloWorld or Int: 123 ..etc
Overwrite parameter with GLU static parameter, e.g. ${GLUstaticParam}
Overwrite parameter with other parameter, e.g. ${otherParam}
Overwrite parameter with derived parameter,e.g. ${derivedParam}

Save in the Cache

If the tick box is selected, then the data value of the Parameter will be stored in memory for the duration of the time the GLU.Engine is running. If the GLU.Engine is restarted then the data will be lost. This can be used in conjunction with the “Run on startup” feature to populate data to be used for all API/Transactions in the GLU.Engine.

Data which is saved in the cache will be available to all APIs in the GLU.Engine.

To reference cached values, use the following GLU Reserved Words in a Derived Parameter:
SET_DATA_TO_CACHE
GET_DATA_FROM_CACHE
GET DATA FROM CACHE CONTAINS
GET MAPPED ARRAY FROM CACHE

Connector Retry function

There are different ways to trigger a retry. A retry can be triggered on a timeout as well as on a certain response received from a third party.

Retry on Timeout

Step 1: Set the timeout value

On the outbound request as below, the timeout value needs to be set in milliseconds.

Step 2: Set the Handler rules

On the Outbound Response a Handler can be added to determine the retry rules. When the timeout is triggered, this Handler will facilitate the retry.

The name of the retry Handler.
Select the ‘none’ option in the Response Handler Control Panel.
Select the ‘Retry on timeout’ Action in the dropdown.
Enter the Retry Count: The number of times the retry will attempt to send the transaction
Enter the Retry Delay: Each retry will be sent with a delay of x milliseconds.

Step 3: Response after retry attempts

During the retry process, the GLU.Engine will either receive a response from the third party, or it will complete the retry attempts and then send back a timeout response. The timeout response that is sent back will be as per the configuration in the response control panel. The Timeout Response Code and Timeout Response Message are configured as below.

Retry on a specific response parameter value

Step 1: Set the handler rules

An outbound response handler can be used to determine the retry rules for a transaction. When a particular condition is met, this handler will facilitate the retry.

Once the retry handler is processed, the rest of the handlers in the orchestration will be processed (i.e. Retry handlers are first to be processed.)

Note: the condition’s value can be one value or multi values separated by a comma e.g. ValueOne,ValueTwo,ValueThree….. etc

Route to a new flow code on Timeout

One is able to have two orchestrations setup for timeout such that you first ‘Retry’ upon TIMEOUT and thereafter you can ‘Route to’ upon TIMEOUT. To route to a different Flow code upon a timeout, configure the Handler as outlined below.

Step 1: Set the timeout value

On the Outbound Request as below, the timeout value needs to be set in milliseconds.

Step 2: Set the handler rules

The name of the retry handler
Select the ‘none’ option in the response handler control panel
Select the “Route to on timeout” Action in the dropdown.
Flow Code: Select the flow code of the transaction that you want to route to when a timeout is triggered from within that transaction.

Retry delivery counter variable

GLU_REDELIVERY_COUNTER – this increments for each retry that is sent. This counter is available as a GLU variable and can be used to count the number of iterations which the retry completes. This needs to be set up in the derived parameter, see example below.

Handling the Receiving System Response

The receiving system response is handled by firstly creating the response parameters you require and then by processing any logic required on the response.

Depending on the Connector type, for quick generation of response parameters, the response payload sample can be used in the ‘Generate Parameters from Template‘ tool which will auto-generate the entire Response payload along with any associated Object/Collection path mappings. This will create the full parameter set and unmarshalling paths automatically unless otherwise specified.

Response handlers can be created in order to perform logic on the response received from the downstream system.

Orchestration Manager Overview

Within the Integration Builder use the Orchestration Manager section in ‘Green’ of the Transaction pattern to define the downstream Receiving System transaction step or steps for each transaction being configured. There is no limit to the number of receiving systems that can be addressed within the definition of a single transaction.

Each step in the flow to a downstream system is defined by the Request that the GLU.Engine will deliver and the Response it will receive via the pre-defined Receiving System Connector.

Each such step has several possible configuration fields:

Flow Code

Used by Handlers to redirect the flow of a transaction (to specific Flow Code). Thus, the flow of any orchestration is not necessarily sequential in line with the sequence of that the transactions steps with Receiving Systems are defined in the Console. In the example above ‘db1’ is the assigned Flow Code.

Exclude from Build

This can be used to stop the connection defined by this Flow Code from appearing in the build. If this is ticked then this connector will not be included in build and it will not be possible to route to this flow code. It should be used when building a GLU.Engine and the connector is not needed in this version. Also if this is selected it will mean the connector will not be included in the GLU billing.

Async

If selected, this route will be performed asynchronously to other routes. By default, all routes are performed synchronously. Note: The Flow between routes can be altered with the use of Handlers – Flows marked as ‘Async’ will be invoked irrespective of any Handlers.

Process

An orchestration can be marked as ‘Process’. This is an orchestration that just handles logic and routing through the user of handlers. There is no outbound request and response available on a Process orchestration.

Is StreamCall

This process is in Beta development with GLU.Ware team. Further descriptions will follow in later releases of GLU.Ware, however if you are interested in using streaming connectors please speak to GLU support.

Connector

Refers to possible pre-defined Connectors for the Receiving System.

Content Types

Content Type refers to the format and structure of the data being sent or received as part of a message. Connector types and protocols, refer to the mechanism used to transmit the message from one system to another.

The relationship between content types and connector types/protocols comes into play when sending or receiving messages. For example, if you want to send a message containing JSON data over HTTP, you would need to use an HTTP connector and set the content type of the message to application/json so that the receiving system knows how to interpret the data.

GLU connectors support various content types for messages, which can be specified using the Content-Type selector. The following are the content types that GLU supports:

text/plain: This content type is used for plain text messages, such as email or SMS messages.

application/xml: This content type is used for XML documents, which are commonly used for data exchange between systems. The correct Content type for XML documents is application/xml or text/xml. In GLU, you can use either of these Content Types to specify the content type of XML documents being sent as part of a message to an external system or API. The application/xml type is more commonly used, and it indicates that the content of the message is an XML document. The text/xml type is also valid, but it is less commonly used and has the same semantics as application/xml.

application/json: This content type is used for JSON documents, which are commonly used for data exchange between systems.

application/octet-stream: This content type is used for binary data, such as images or audio files.

multipart/form-data: This content type is used for submitting data that includes binary data, such as file uploads.

application/x-www-form-urlencoded: This content type is used for submitting form data encoded as key-value pairs, such as HTML form submissions. Parameters being passed in to this content type cannot include special characters. In order to encode these, you can apply the following function to the parameters before sending them on an outbound request. URL_ENCODER

application/soap+xml: This content type is used for SOAP messages, which are commonly used for web services communication.

application/atom+xml: This content type is used for Atom feeds, which are commonly used for syndicating web content.

application/rss+xml: This content type is used for RSS feeds, which are commonly used for syndicating web content.

application/pdf is a legitimate content type used for Portable Document Format (PDF) documents. PDF is a popular file format used for document exchange, and it is widely supported by various software applications and platforms. In GLU, you can use the application/pdf content type to send PDF documents as part of a message to an external system or API.

text/xml; charset=UTF-8 is a content type that indicates that the content of the message is an XML document encoded in the UTF-8 character set. The text/xml media type is used to indicate that the content of the message is a text-based XML document, and the charset parameter is used to specify the character encoding of the document. It’s worth noting that the application/xml content type is more commonly used to indicate that the content of the message is an XML document, and it also allows you to specify the character encoding using the charset parameter. Therefore, you can also use application/xml; charset=UTF-8 to specify that the content of the message is an XML document encoded in UTF-8.

application/xml; charset=UTF-8 is a valid content type that indicates that the content of the message is an XML document encoded in the UTF-8 character set. The application/xml media type is used to indicate that the content of the message is an XML document, and the charset parameter is used to specify the character encoding of the document.

application/jose is a valid content type that is used for representing encrypted and/or signed JSON data. JOSE (JSON Object Signing and Encryption) is a set of specifications that define a way to secure data using JSON format. It provides a way to digitally sign, encrypt, and/or authenticate JSON data, which can be useful for secure communication between systems and services. In GLU, you can use application/jose as the content type for sending and receiving JOSE-formatted messages. GLU supports working with JSON Web Tokens (JWTs) and JSON Web Encryption (JWE) messages. GLU allows you to create, verify, and decode JWTs, as well as encrypt and decrypt JWE messages. For signed messages, the alg parameter is used to specify the signing algorithm, while for encrypted messages, the enc parameter is used to specify the encryption algorithm.

It’s worth noting that there are several other media types related to JOSE, such as application/jwt for JSON Web Tokens, application/jwe for JSON Web Encryption, and application/jws for JSON Web Signatures. These media types are used to indicate specific types of JOSE messages and may have additional parameters for specifying the algorithm and other metadata.