SAP Integration Suite – External logging to Datadog Using OpenTelemetry

[[{“value”:”

Introduction

SAP Cloud Integration (CPI) with Software version update 8.42.* has introduced support for OpenTelemetry (OTEL) as an external logging destination, allowing you to stream Message Processing Logs (MPL) directly to OTEL-supported observability platfroms like Datadog, Grafana, Splunk etc for centralized observability.

What is OpenTelemetry? OpenTelemetry is a vendor-neutral framework for collecting, processing, and exporting telemetry data—metrics, logs, and traces. It’s widely adopted across the industry and supported by most observability platforms, including Datadog. This standardized approach means your integration logs can be correlated with other systems and applications in your Datadog environment.

In this guide, I will walk through the step-by-step configuration needed to send CPI logs to Datadog using the OTEL integration, including prerequisites, BTP setup, and validation of the data flow.

Prerequisites

Before you begin, ensure you have the following:

• An active Datadog account with access to the organization settings
• A Datadog API key (available at Organization Settings > API Keys)
• BTP subaccount with Connectivity and Destination access
• Acess in BTP subaccount to create/update service instance of Process Integration Runtime with appropriate roles

Step 1: Configure Your Datadog Setup

First, prepare your Datadog environment to receive logs from CPI.

Get Your API Key

Navigate to your Datadog organization settings and create an API key if you don’t have one already. Go to Organization Settings and select API Keys. Create a new key and copy it—you’ll need this for the BTP destination configuration.

Identify Your Datadog Region and OTEL Endpoint

Depending on where your Datadog organization is hosted, you’ll use a different endpoint. For example:

• EU region: https://otlp.datadoghq.eu/v1/logs
• US region: https://otlp.datadoghq.com/v1/logs

Confirm your organization’s base URL (it appears in your browser when you log in) and append the OTEL log endpoint accordingly.

Step 2: Create the BTP Destination

Now configure the HTTP destination in SAP BTP that will serve as the bridge between CPI and Datadog.

Navigate to Destinations

Log into SAP BTP, open your integration subaccount, and navigate to Connectivity > Destinations.

Create a New Destination

Click “Create Destination” and configure it with the following parameters:

Name: CloudIntegration_MonitoringDataConsumer
Type: HTTP
Authentication: No Authentication
URL: https://otlp.datadoghq.eu/v1/logs

Authentication set to “No Authentication”?  Yes—the Datadog API key will be passed as a header, not as basic auth.

Add Additional Properties

Scroll down and add the following additional properties. These fields tell Datadog how to classify and organize your CPI logs:

Property Name Value Purpose

Important: Replace <Your Datadog API Key> with the actual API key you created earlier. Store this key securely and never commit it to version control.

After entering all properties, click Save.

Step 3: Activate External Logging in CPI

With the destination configured, you now need to activate external logging at the tenant level.

Create a Service Instance

In your BTP subaccount, go to Services > Instances and Subscriptions. Create a new instance of Process Integration Runtime with the following settings:

• Service: Process Integration Runtime
• Plan: api
• Roles: ExternalLoggingActivate, ExternalLoggingActivationRead

Once created, generate a service key and copy the following credentials:

• Client ID
• Client Secret
• Token URL
• API Host (base URL for your CPI tenant)

Authenticate and Call the Activation API

Use Postman (or curl) to authenticate with the token URL and then call the external logging activation endpoint. Here’s the flow:

1. Get an OAuth token:

POST {TokenURL}
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials
&client_id={ClientID}
&client_secret={ClientSecret}

Copy the access_token from the response.

2. Activate external logging:

POST https://{TenantName}.{host}.hana.ondemand.com/api/v1/activateExternalLogging?defaultLogLevel=INFO
Authorization: Bearer {access_token}

The defaultLogLevel parameter controls which log levels are sent. Options are:

• NONE—disable external logging
• INFO—send info, warn, and error logs (recommended)
• ERROR—send error logs only

A successful response confirms external logging is now active. You should see logs flowing to Datadog within a few seconds.

Verify Activation Status

To confirm external logging is active, query the status endpoint:

GET https://{TenantName}.{host}.hana.ondemand.com/api/v1/ExternalLoggingActivationStatus('{TenantName}')
Authorization: Bearer {access_token}

A successful response will show the activation status as true.

Step 4: Verify Data Flow and View Logs in Datadog

Once external logging is activated, CPI will begin sending Message Processing Logs to Datadog in real-time.

Querying Logs in Datadog

In Datadog, open the Logs section and search for logs with the service you configured in BTP Destination OTELServiceName.

service:MessageProcessingLogs

You should see MPL entries streaming into Datadog logs.

Parsing MPL Content as JSON

CPI sends MPL logs in a structured format, but the log content itself may be nested. To extract individual MPL attributes and create reports or dashboards, you can use Datadog’s Pipelines feature:

Option 1: JSON Parser — If your log body is valid JSON, use the JSON parser to flatten the structure and extract fields. 

Option 2: GROK Parser — For more complex log formats, use a GROK parser pattern to extract specific fields.

Note: My Datadog was missing JSON parser, so I created 2 GROK Parsing rule as below after carfully verifing all the log content format ingested by SAP CI for various combination (ApplicationMessageId, SenderId, CustomStatus, LastError, LastErrorModelStepId etc) to parse all MPL logs.

Once parsed, you can facet logs by integration name, message ID, status, and other attributes to build dashboards, set up alerts, and troubleshoot issues faster.

Maintenance and Troubleshooting

Updating the Destination

If you need to modify the destination (e.g., change the API key or region), follow this sequence:

1. Call the deactivateExternalLogging endpoint
2. Update the destination in BTP
3. Call the activateExternalLogging endpoint again

Skipping deactivation can result in not reflect the destinatinn updates.

Known Limitations

When CPI sends logs via OTEL, the timeUnixNano and observedTimeUnixNano fields are currently set to 0 instead of a proper Unix epoch value for custom headers. Datadog doesn’t process these malformed timestamps, so log ingestion still works but only for Run Events and not Custom Headers. I believe SAP will address this in a future update.

Useful API Endpoints

For reference, here are other relevant endpoints for managing external logging:

Deactivate external logging:
POST https://{TenantName}.{host}.hana.ondemand.com/api/v1/deactivateExternalLogging

View external logging events:
GET https://{TenantName}.{host}.hana.ondemand.com/api/v1/ExternalLoggingEvents

Check MPL logs (alternative query):
GET https://{TenantName}.{host}.hana.ondemand.com/api/v1/MessageProcessingLogs

For detailed API specifications, refer to the SAP API Hub documentation on External Logging.

“}]] 

  Read More Technology Blog Posts by Members articles 

#abap