Exactly Once, In Order Message Delivery with Boomi Event Streams
Our application integration team has a long history of work with the soon-to-be-legacy SAP integration product called Process Orchestration or Process Integration (PO/PI). This product offers developers a way to make sure that messages are sent in the same order they were received. It was called “Exactly Once, In Order”, or EOIO, for short. As SAP PO/PI reaches its end of life in 2027 customers are migrating their interfaces and finding out that EOIO doesn’t translate to other integration platforms, like Boomi.
Out-of-the-box, Boomi integration can retry messages up to a limit before stopping and discarding the message, and for sensitive interfaces this can cause out-of-order issues.
Boomi Event Streams largely addresses these requirements, and promises to “support multiple delivery modes, including FIFO, Pub-Sub, and Queuing with guaranteed message delivery, ordering and availability”. But there is a small gap in Event Stream that prevents it from fully meeting the expectations of EOIO, and we’ll address it in this article.
Goals
- Guarantee delivery: Once a message is received by Boomi, it must persist until Boomi
sends it to its destination and gets a confirmation. Only when the receiver confirms
successful delivery does the message queue remove the message. - First in, first out ordering: Messages will be sent in the order it is received.
- Automatic retries: If a message fails, it will retry until successful before letting the next
message process
Event Streams Listen Operation
Boomi Event Streams lets you set up a message queue pattern, where messages from a publisher can be consumed either by a Listen operation or a Consume operation.
The Listen operation offers three subscription types: exclusive, fail-over, and shared. The Listen operation also includes a “transacted acknowledgement” (TA) option, which removes messages from the topic only after receiving a positive confirmation.
We tested these parameters and concluded that processes with the Event Streams Listen operation either couldn’t keep messages in order in failure scenarios or failed to keep messages in the system for continuous retries.
Event Streams Consume Operation
To use the Event Streams Consume operation, an asynchronous “Consumer” process reads messages in batches from a “Main” topic, which is populated with messages from a “Producer” process. This process uses Boomi’s flow control features to maintain message order and sends the messages downstream to its intended destination.
After messages are read from Main by the Consumer, Event Streams no longer tracks them. To handle errors, a “Dead Letter” (DL) topic is used. Messages that encounter failures are written to this DL topic.
Every execution of the Consumer process will check the DL topic for messages to consume. If messages are present, these are read and processed instead of from the Main topic. Messages are read from the Main topic only if the DL topic is empty.
The sequence diagram highlights the conditional check of the DL topic performed at the start of the Consumer process.
In failure cases, the sequence diagram below illustrates that any messages that haven’t been confirmed by the target system are written to the DL topic.
Conclusion
In conclusion, while Boomi Event Streams offers powerful features for handling message delivery and ordering, it doesn’t quite replicate the behavior of SAP’s EOIO in Process Orchestration/Process Integration. Our recipe provides a solution to achieve this quality of service. By using a new topic as a dead letter queue, the ordering of failed messages can still be maintained and additional error handling controls are provided to administrators compared to the default dead letter options provided by Event Streams.
