Introduction

The pipeline concept was introduced by SAP to mirror the way messages are processed in SAP PO/PI – in a pipeline.  Using this concept in cloud integration offers one potential route for migrating a large number of integrations from SAP PO/PI.  The pipeline concept also reduces the number of JMS queues needed for managing migrated or new asynchronous integration scenarios.

There are a great series of blogs that can be found at https://community.sap.com/t5/technology-blog-posts-by-sap/introducing-the-new-pipeline-concept-in-cloud-integration/ba-p/13639651.

Please note that the original SAP Package for the pipeline concept ‘Process Integration Pipeline – Generic Integration Flows and Templates’ will soon be deprecated and the new integration packages should be used, see blog: https://community.sap.com/t5/integration-blog-posts/new-integration-packages-for-the-pipeline-for-cloud-integration/ba-p/14175339

The aim of this blog is to show an example of manually setting up a simple asynchronous integration using the new pipeline packages.

For reference I have mocked up an example iflow in SAP PO that we will replicate in cloud integration using the pipeline.  The flow starts with a json message being sent from Postman over https, converted to xml in the adapter and then sent via email to 2 recipients each with a single interface.

Setting Up The Cloud Integration Pipeline – Overview

From Discovery Tab download following packages:

• Cloud Integration Pipeline – Generic Inbound: Contains the generic IDoc and XI inbound flows.
• Cloud Integration Pipeline – Generic Pipelines: Contains the main generic pipeline steps.
• Cloud Integration Pipeline – Generic Outbound: Contains the generic outbound processing flow.
• Cloud Integration Pipeline – Templates: Contains all templates for the scenario-specific flows and custom extensions.

Although the complete pipeline (shown below with flow steps numbered) is still supported for Asynchronous messages:

The recommendation is to use the Integrated pipeline (flow steps numbered for reference in diagram below):

For setting up the integrated messaging runtime for asynchronous communication (recommended option for asynchronous communication), you need to deploy the following generic integration flows:

• (Optionally, if you like to use a single entry point for IDoc messages) Pipeline v2 Generic Step01 – Inbound Processing for Idoc
• (Optionally, if you like to use a single entry point for XI messages) Pipeline v2 Generic Step01 – Inbound Processing for XI
• Pipeline v2 Generic Step02 – Integrated Messaging Runtime Async
• Pipeline v2 Generic Step06 – Outbound Processing

Configure Flow Step2

Configure ‘Pipeline v2 Generic Step02 – Integrated Messaging Runtime Async’ and deploy

Note – JMS queues are created automatically when you deploy

CustomXPid_SwitchOffTilde is set to true – this ensures the alternative Partner (setup in next step) is read from partner directory.

PipelineJMSQueuePrefix – enter custom Queue Prefix

Configure Flow Step 6

Configure ‘Pipeline v2 Generic Step06 – Outbound Processing’ and deploy

PipelineJMSQueuePrefix – enter custom Queue Prefix

Initial Partner Directory Setup

In Monitor|Integration and APIs|Manage Partner Directory – create a new Partner entry of type ‘Alternative Partners’.  This information is used by the generic flow steps to identify the interface being processed.

Setup Flow Step 1

For Step 1 of the process you can either create your own iflow of copy a template from the Package ‘Cloud Integration Pipeline – Templates’.  In this example I copied the following iflow ‘Pipeline v2 Template Step01 – Inbound Processing At Least Once’

Configure Flow and set name of Receiver Queue to match Sender Queue name setup in ‘Pipeline v2 Generic Step02 – Integrated Messaging Runtime Async’

For this example set Sender Adapter as HTTPS

In ‘Set SAP headers’ Content Modifier set values below to match ones set in Alternative Partner

Nb: Optionally could pass header ‘partnerID’ with value in this case of ‘INT_0006_Test_Pipeline’ rather than using SAP_Sender & SAP_SenderInterface

The ‘Parse incoming XML’ and ‘Set custom header attributes’ steps are used for setting customer headers on MPL Logs if required.  Can delete if not needed.  See blog for more details https://community.sap.com/t5/integration-blog-posts/new-pipeline-concept-features-supporting-custom-header-properties-and/ba-p/13894048

The ‘Define Service Interface Operations’ and ‘Map Message Type to Operation’ steps are used to replicate the SAP PO functionality of having different operations based on the Message Root Node.  Can delete if not needed. See blog for more details https://community.sap.com/t5/integration-blog-posts/new-pipeline-concept-features-supporting-service-interface-operations-and/ba-p/14049909

Optional – Flow Step 3

If want to call a scenario specific inbound conversion then can create an entry in the Partner directory

Again can create your own iflow or Copy template iflow ‘Pipeline v2 Template Step03 – Inbound Conversion’

Set Process Direct Address to match above setting

This template also contains steps for custom headers and handling service interface operations – again delete steps if not required.

For this simple example I have added the JSON to XML step.

Receiver Determination – Partner Directory Setup

Point-to-Point Scenario (not used for this example build since we have 2 receivers)

For Point-to-Point scenarios, that is, scenarios with only one receiver and one receiver interface, you can bypass the two flow steps receiver determination and interface determination.

In this case, you don’t have to upload XSLT mappings to determine the list of receivers and receiver interfaces, but instead you need to create corresponding string parameters in the Partner Directory.

1.  Define a Partner Directory string parameter with ID receiverDetermination as follows:

2.  define a Partner Directory string parameter with the ID as combination of the constant identifier interfaceDetermination and the receiver system name, separated with an underscore “_”. As value, enter the ProcessDirect endpoint of the scenario-specific integration flow for outbound processing

Receiver And Interface Determination

For scenarios with more multiple receivers and or multiple receiver interfaces an xslt file can be created listing these.  This is saved to the Partner Directory and then used by the Generic integration flow.

ReceiverNotDetermined can be set to Default,Ignore or Error

These option replicate those found on the SAP PO IFlow

Save as a xsl file

Load as a Binary Parameter in Partner Directory – Note the user interface encodes the file as base64 as it loads (if using the API need to manually convert to base64 before loading)

Flow Step 7 – Outbound Message

For the final outbound flow you can either create your own iflow or copy one of the ‘Step07’ ones from the ‘Cloud Integration Pipeline – Templates’ package as a starting point.  To keep things simple I created my own – one for each receiver.  It is also at this point you could add any message transformation logic if required ie: mappings.

Testing

Send Message from postman

2 emails receiver with different subjects to differentiate the receivers

Notes On Retries

By default the send from the respective JMS queues will retry 5 times

After the max retries the message will go to the DLQ

Once cause of fail is corrected message can be moved from the DLQ back to the normal queue to process again

Can specify scenario specific retries in partner directory if required

  Read More Technology Blog Posts by Members articles 

#abap