Beginners Guide to Advanced Event Mesh
Share

Introduction:

In this blog post, I will show the common scenarios of Advanced Event Mesh. To get familiar with basic concepts, go through the below links as a prerequisite.

Advanced Event Mesh: Basic Concepts 

Advanced Event Mesh: Create your first event broker

Advanced Event Mesh: Queue 

Advanced event mesh is powered by Solace and throughout the blog post I will use a solace trial account. All the steps remain same if you are using AEM (Advanced Event Mesh).

In this blog post, the following topics will be covered.

  • Publish Message to Queue using AEM adapter
  • Consume Message from Queue using AEM adapter
  • DMQ Setup and Error Handling
  • Direct Message
  • Durable topic endpoint
  • Use of Selector
  • Send NACK to Publisher
  • Time-to-Live
  •  Rest Delivery Point
  • Replay Log
  • Parallel Consumers
  • Request Reply Pattern

Scenario 1: Publish Message to Queue using AEM adapter

Step 1: 

Create a queue in AEM

Pic1.png

Step 2:

Create an iFlow

 pic2.png

Configure HTTP sender adapter

pic3.png

Configure AEM receiver adapter. Here, the password secure alias is of type Secure Parameter.

pic4.png

The queue name is populated based on the value of Camel header – CamelHttpPath

pic5.png

The message is set as DMQ eligible.

pic6.png

Step 3:

Send event message using API testing tool to the endpoint: https://<host>/http/generic/order

pic7.png

 

Step 4:

Validate if the message is pushed to the AEM queue

pic8.png

Scenario 2: Consume Message from Queue using AEM adapter

Step 1:

Create an iFlow to consumer message from the AEM queue. Set Allowed headers as * to trace the incoming headers.

pic11.png

Configure AEM sender channel

pic10.png

Step 2:

Check message trace in Cloud Integration and verfiy headers. As the publisher iFlow set the the header DMQ eligible as true, the ‘IsDMQEligible’ header is true.

pic12.png

Step 3:

Verify the consumer status attached to the AEM queue.  Here, it reflects ‘Messages Confimed Delivered’ as 1.

pic13.png

Scenario 3: DMQ Setup and Error Handling

To move a message to DEAD Message queue upon delivery failure, follow the steps below.

Step 1:

Configure AEM queue as below.

pic14.png

pic15.png

Also, the queue is subscribed to the topic expression order/>

pic17.png

 

Step 2:

 

Change receiver channel configuration in order to push event message to a topic. The topic name is dynamically set using the header – CamelHttpPath .
 
pic18.png

The sender channel will consume messages from the queue named ‘order’. The message sent from the publisher iFlow is a JSON message. In order to simulate failure, XML to JSON converter block is added . 

pic24.png

Notice the acknowledgement mode is ‘Automatic Immediate’ at Cloud Integration (Consumer).

 
pic20.png
 

Step 3:

POST message to the endpoint https://<host>/http/generic/order. No message will be pushed to the queue, because the topic subscription order/> expects at least one level after order.

pic21.png

pic22.png

 

 

Step 4:

 

Push message to the endpoint https://<host>/http/generic/order/create
 
pic23.png
 
But, observe the message is successfully deleted from the queue. It is because acknowledgment mode is ‘automatic immediate’.
 
pic22.png
 
Now, the retry happens in Cloud Integration 4 times and then the message is marked as failed. 
pic25.png
 
The below shows the trace of Run 1. The header ‘DeliveryCount’ is set from AEM. The header ‘LocalProcessingAttempt’ refers to processing attempts at Cloud Integration.
 
Pic26.png
 
At run 5,  LocalProcessingAttempt value will be 5. After run 5, the message moves to failed state, as the maximum message processing attempts is set as 5.
pic27.png

Step 5:

Now, change the acknowledge mode to ‘Automatic on Exchange Complete’.

pic28.png

Trigger the event message from API testing tool. After trigger, the message is still in the queue, while retry is going on at Cloud Integration.
pic29.png

Now, after 4 retries, in Cloud Integration, failed status is sent back to the AEM. 

pic30.png

During the 5th run, we can see the local processing attempt as 5.

pic32.png

As we have provided, the re-delivery count as 3 in AEM, the AEM re-delivers it 3 times.

pic31.png

Below shows 1st re-delivery i.e, 2nd delivery from AEM (Run 5 at Cloud Integration)

pic33.png

3rd re-delivery from AEM (Run 5 at Cloud Integration)

 pic34.png

Then the message is moved to DMQ (Dead Message Queue). Please note, the DMQ queue must be created beforehand. DMQs are not automatically created during message failure.

pic35.png

pic36.png

 

Scenario 4: Direct Message

For direct message, if we use topic, no queue or topic endpoint configuration is required at AEM. If consumer is available during publishing of message, the message will be consumed, else the message will be lost.

Step 1:

Configure Publisher iFlow.

pic37.png

Step 2:

Configure Consumer iFlow.

pic38.png

Step 3:

Test and validate.

pic39.png

pic40.png

Scenario 5: Durable topic endpoint

Step 1:

Create a topic endpoint at AEM.

pic41.png

Step 2:

Configure the AEM receiver adapter which acts as a publisher.

pic42.png

Configure the AEM sender adapter which acts as a subscriber.

pic43.png

Durable Topic endpoint supports a single subscription. Once a consumer is attached, the topic name gets populated.

pic44.png

Now, currently the topic endpoint has 2 messages queued.

Step 3:

Change the topic name at Cloud Integration side to ‘product/create’.

pic45.png

After changing the topic name to ‘product/create’ and deploying the consumer iFlow, all existing messages are deleted automatically without delivering it to the consumer and the recent topic name gets reflected.

pic46.png

Scenario 6: Use of Selector

Step 1:

Configure Publisher iFlow as below.

pic47.png

 Set event type, priorty and status as user properties.

pic48.png

pic50.png

Step 2:

Populate the selector field at AEM sender adapter as below.

 pic51.png

Step 3:

Test with payload field values as below:

data/eventType order.updated
data/status delayed
data/priority high

pic52.png

Check the trace of consumed message at Cloud Integration.

pic53.png

Now, test with payload field values as below:

data/eventType order.updated
data/priority high

pic54.png

The message is not consumed from the queue as the selector does not match.

pic55.png

Scenario 7: Send NACK to Publisher

General behavior is 

  • If topic subscription does not exist for delivery mode persisted(guaranteed) or direct, no error response is sent to the publisher
  • If queue does not exist for delivery mode persisted(guaranteed), error response is sent to the publisher

pic57.png

  • If the queue does not exist for delivery mode direct, no error response is sent to the publisher.

The profile related to used client username (solace-cloud-client) is ‘default’ at AEM.

pic58.png

 Go to ‘default’ profile at AEM.

pic60.png

After ‘Reject Messages to Sender On No Subscription Match Discard’ is enabled, if no matching topic subscription is found for persistent delivery mode, an error message is returned to the publisher.

pic61.png

Now, if we stop the incoming calls to the queue,

pic62.png

we will get the error message as below.

 pic63.png

Now, we change the property ‘Reject Messages to Sender on Discard’ on the queue to ‘Always’,

pic64.png

we will get the error message as below,

pic65.png

Scenario 8: TTL (Time to Live) 

Enable TTL feature of queue at AEM.

pic66.png

If the message is not consumed, the message will be moved to DMQ after 1 min. Here, no TTL specific settings are configured in the receiver channel at Cloud Integration.

If we don’t want to store it in the DMQ and discard the message after TTL, set DMQ eligible to false at publisher iFlow.

pic67.png

After TTL, the stats will reflect the same at AEM.

pic69.png

If TTL is set at receiver adapter, the minimum of the queue level TTL and the publisher level TTL takes effect.

pic70.png

 

Scenario 9: RDP (REST Delivery Point)

We have a common DMQ queue for all processes and the Rest Delivery points to Cloud Integration endpoint. Based on the endpoint, Cloud Integration forwards it for relevant processing of the DMQ messages.

Business Object Endpoint
Business Partner redeliver/businesspartner/<BPNumber>
Order redeliver/order/<OrderNumber>

Step 1:

Design the publisher iFlow as below.

pic71.png

User Property named ‘identifier’ is added in AEM receiver adapter.

pic72.png

Step 2:

Create RDP at AEM.

pic73.png

Create Rest Consumer with authentication type ‘OAuth2’ for Cloud Integration

pic74.png

Bind the AEM queue to the RDP endpoint.

pic75.png

On the queue bindings settings, turn on the ‘Substitution Expressions’ for ‘Request Target Evaluation’. The below substitution expression is used for target endpoint genertion. The expression ${topic(1)} represents the 1st level of the topic.

/http/redeliver/${topic(1)}/${userPropertyAsString(“identifier”)}

pic75.png

In case of failure at RDP, it retries forever unless we disable ‘Try Forever’ and set ‘Maximum Redelivery Count’ at DMQ.

pic77.png

Step 3:

The generated endpoint for order message is shown below:

pic78.png

The generated endpoint for business partner message is shown below:

pic79.png

 

Scenario 10: Replay Log

If replay log is enabled, the messages can be replayed from Broker without publisher intervention. To avoid saving the log for all queues, we can enable logging based on topic subscription.

pic80.png

pic81.png

For example, the consumer ‘Cloud Integration’ forwards message to a target but the target system is down. All the messages from that point need to be replayed. For that, we need a start pointer and ‘ReplicationGroupMessageId’ can be used. As a pre-requisite, replication group message id for each message should be logged as a custom header at Cloud Integration.

pic82.png

Now, replay all the messages after the above replication group message id. Please note that selective replay of messages or providing an end replication group message id is not possible. Therefore, idempotency must be implemented on the target side to avoid processing duplicate messages.

pic83.png

pic84.png

Messages after Replication Group Message ID will be replayed.

pic85.png

Similarly, we can create a new queue and attach it to a new consumer and replay the previous log related to its topic subscription from the beginning or from a particular date and time or based on replication group message id.

Scenario 11: Parallel Consumers

No. of consumers = (no. of worker node) * (no. of parallel consumer).

pic87.png

If the tenant has 2 worker nodes and 1 parallel consumer, the consumer attached the queue will show up as below:

pic86.png

If we increase the parallel consumer to 5,

pic88.png

the consumer count becomes 10.

pic89.png

Only 1 consumer is active, the remaining 9 consumers are in inactive state. This is because the queue type is ‘Exclusive’.

Now, if we change the queue type to ‘Non-Exclusive’,

pic90.png

 all the consumers are active. 

pic91.png

E.g., if we want a particular object id (say Business Partner) to be processed by the same consumer, we can utilize the partition concept.

Set the partition count as 5.

pic92.png

pic93.png

Set the parallel consumers as 3

pic94.png

As the tenant has 2 worker nodes, the total no. of consumer will show up as 6. Only 1 consumer is inactive. Remaining consumers are in active state as the number of partition count is 5.

pic95.png

If we set ‘JMSXGroupID’ value, it will be used as a partition key. 

pic96.png

For example, for BP number ‘10001234’, the message was consumed by Flow ID 248 and for BP number ‘10001236’, it was consumed by Flow ID 281.

All the updates related to BP ‘10001236’, are lined up for the same consumer.

pic98.png

pic99.png

Scenario 12: Request Reply Pattern

The publisher iFlow is designed as below:

pic102.png

Configure the AEM receiver channel as below. 

pic100.png

This will create a non-durable queue at AEM when the message is sent from publisher iFlow.

pic101.png

Consumer iFlow receives the headers populated from Publisher iFlow related to response propagation, such as ReplyToDestination, ReplyToEndpointType and CorrelationId

pic103.png

pic104.png

Design Consumer iFlow as below

pic105.png

Please note: The delay is used here only to simulate processing time instead of adding additional pallets. 

AEM receiver adapter is used to send back the response.

pic106.png

pic107.png

After ‘request-reply’ step in the publisher iFlow message trace, we will see the response headers.

pic108.png

 

Lets test the publisher iFlow.

pic109.png

As we can see, the response message is received.

Thank you for reading this blog post. 

 

  Read More Technology Blog Posts by Members articles 

#abap

By ali