Splitting an array and parallel processing are commonly employed in to enhance efficiency and optimise resource utilisation when dealing with large datasets or complex tasks.
Splitting an Array
It is possible to take a large payload and use the Splitter function to break it up into multiple elements to be orchestrated.
In the example below we will take a payload and store each element in a table of a database.
Body Payload
See below this simple array:
| { “report”: [ { “reportLine”: { “glcode”: “AG82652-3635”, “name”: “Sticky Buns”, “balance”: “500” } }, { “reportLine”: { “glcode”: “AG82634-3635”, “name”: “Iced Donuts”, “balance”: “2800” } } ] } |
This is an array called “report” containing elements made up of a class called “reportLine”.
Target Table
The intended table structure is as follows:
Each item from the array will occupy a distinct row within the table. Furthermore, the attributes of each class instance will be appropriately assigned to the corresponding column names.
| glCode | Name | Balance |
|---|---|---|
| AG82652-3635 | Sticky Buns | 500 |
| AG82634-3635 | Iced Donuts | 2800 |
Console Configuration
Request Parameters setup
To configure the marshaling set up the Request Parameters as per the example below. The Object / Collection path stipulates the path within which to find and store the class element being unmarshalled called ‘glcode’.
Properties
Unmarshaling
Handler setup
To orchestrate the splitting of each element in the incoming payload, you need to set up a Handler specifically for splitting the payload. Below are the steps outlined based on the numbered diagram and accompanying table:
Handler Parameters
| Parameter | Console Label | Example Value | Description |
|---|---|---|---|
| 1 | Parameter, None, Expression | None | To show the splitter options this needs to be set to ‘None’. |
| 2 | ACTION | ROUTE TO | Indicates how the splitter will be used. |
| 3 | Not from list | Not ticked | BETA description to follow. |
| 4 | FLOW CODE: Choose the flow code of the orchestration which will be triggered for each element of the array | splitterRoute944482 | Flowcode of the orchestration which will be executed |
| 5 | ENABLE THREAD POOL: FALSE | FALSE | See Throttle Type 2: Concurrent Requests using Thread Pools |
| 6 | Splitter | Select this to call the flow code for each element in the array | |
| 7 | Parallel Processing | Not ticked | If this is ticked then each orchestration will be run in parallel. To be used to speed up the processing. |
| 8 | Streaming | Not ticked | BETA description to follow. |
| 9 | Stop on Exception | Not ticked | BETA description to follow. |
| 10 | Skip First | Not ticked | BETA description to follow. |
| 11 | TIMEOUT | 0 | BETA description to follow. |
| 12 | DELIMITER | Blank | To be used for SFTP, FTP, and FILE functions see FTP / SFTP Connectors & FTP/S |
| 13 | Concatenate response | UN TICKED | Tick if the response from each of the Orchestration of the elements needs to be returned back as a Response to the API |
| 14 | unlabelled | report | Provide the name of the array (as defined in the Request Parameters |
| 15 | COMBINED ROUTE | BLANK | BETA description to follow. |
ENDPOINT MANAGER PANEL
Each element within the array will be stored in the database, with each element occupying a single row. Below outlines the configuration steps for the ENDPOINT MANAGER PANEL:
REQUEST MANAGER PANEL
The below shows the setup of the REQUEST MANAGER PANEL, specifically configured with a mySQL DB connector to insert rows into a table:
| insert into report(glCode,Name,Balance) value (${glcode},${name},${balance}) |
RESPONSE MANAGER PANEL
Ensure to include the appropriate Response Manager properties to avoid failure in case the Response Manager panel is incomplete.
Please refer to Response Manager for further details on how to complete the response manager panel.
Parallel Processing
When parallel processing is enabled, the Splitter uses a thread pool to process the messages concurrently. The thread pool is, by default, configured to use 10 threads, making it up to 10 times faster.
