[[{“value”:”

“Most CPI developers configure adapters. Few understand what happens after deployment.”

This independent research project exists to fill that gap: opening the adapter runtime box and helping the SAP Community understand what really happens inside SAP Cloud Integration after an iFlow is deployed. – Ricardo Viana, 03 June 2026

Part II — What Happens After iFlow Deployment: Consumer Runtime, Polling and Offset Commit

Most SAP Cloud Integration developers use adapters every day.

Very few understand what happens inside one after an iFlow is deployed.

This article opens that runtime box.

In Part I, I introduced the motivation and architecture behind EventSmartKafka, an open-source custom Kafka adapter for SAP Integration Suite / Cloud Integration.

Part II does not focus on Maven setup, project generation, OSGi packaging, or how to create the adapter project from zero. That belongs to Part III.

This article focuses on the runtime behavior behind the CPI channel:

• how metadata.xml defines the adapter UI
• how UI fields become Java Endpoint properties
•  how the Consumer is created after iFlow deployment
• how doStart() initializes the Kafka client
•  how doPoll() reads Kafka records
• how payload validation and Avro conversion happen before the CPI pipeline
• how poison messages are handled and why offset commit must happen only after successful processing.

Kafka connectivity is only one part of the story.

The real adapter problem is controlling runtime behavior without silent payload loss, uncontrolled retry loops, hidden offset commits, or generic monitoring errors.

What to Expect: The 7-Part Deep-Dive Journey

To fully unpack the engineering behind an enterprise-grade SAP adapter, this initiative is structured into seven comprehensive technical chapters:

Status: Community Demo / Experimental Technical Preview

Before going deeper, I want to make the positioning explicit.

EventSmartKafka is being released as a Community Demo / Experimental Technical Preview.

It is:

• Not SAP-certified;
• Not enterprise stress-tested;
• Not SLA-backed;
• Not a commercial product;
• Not a replacement for vendor-supported Kafka adapters.

The objective is educational, maybe with future contribution can be something interessting for all.

The goal is to help the SAP Community understand what happens inside a custom adapter runtime after an iFlow is deployed.

The first public baseline focuses on the Consumer side — implemented as a CPI Sender Adapter:

• Kafka event ingestion
• raw passthrough
• Avro to JSON conversion
• Avro to XML conversion
• Schema Registry integration
• Confluent Magic Wire Format
• Fixed Schema ID resolution
• SASL / PLAIN
• SCRAM-SHA-256
• SCRAM-SHA-512
• TLS connectivity
• controlled runtime safety limits
• compact CPI monitoring errors
• poison message handling
• offset commit control.

Producer-side validation, Protobuf, OAuth 2.0, and full mTLS validation remain roadmap items for later parts.

Production usage requires independent validation by the adopting organization.

Package Overview in SAP Integration Suite

Figure 1 — EventSmartKafka package inside SAP Integration Suite, positioned as a Community Demo / Experimental Technical Preview.

SAP ADK – The open box for SAP SCN Community

SAP adapters are often treated as black boxes.

Developers configure them, deploy iFlows, and wait for messages to arrive — but very few see what happens inside the adapter runtime.

With the SAP Adapter Development Kit, that box can be opened.

A custom adapter exposes the real internal layers behind a CPI channel: metadata.xml, Component, Endpoint, Consumer lifecycle, polling logic, payload handling, error strategy, and offset control.

The objective of this article is not only to show EventSmartKafka as a Kafka adapter.

The objective is to open the box and make the adapter runtime understandable for the SAP Community.

Figure 2 — SAP ADK opens the adapter black box: from design-time metadata to runtime Consumer behavior.

SDIA and ODCP — The Invisible Domain Layer

EventSmartKafka is also an implementation example of SDIA — Semantic Domain Integration Architecture — and ODCP — Orchestration Domain-Centric Pattern for SAP CPI.

SDIA does not replace tools, platforms, or existing governance models. It changes the operating model. The same artifacts remain: APIM, CPI, Kafka topics, metadata, security objects, iFlows, scripts, and backend services. What changes is the discipline used to organize and operate them.

When applied correctly, SDIA creates an invisible Domain Layer above the integration stack. This layer gives business meaning, ownership, boundaries, and operational control to technical artifacts across APIM, CPI, Kafka, and backend systems.

SDIA is a free and open architecture released under Creative Commons Attribution 4.0 International — CC BY 4.0.

That is the principle behind SDIA: technology executes, but the domain governs.

Zenodo: Semantic Domain Integration Architecture (SDIA): A Vendor‑Agnostic Semantic Governance Model for Enterprise Integration – v3.0 – FINAL

Zenodo: Orchestration Domain-Centric Pattern (ODCP): A Vendor-Agnostic Domain-Semantic Orchestration Topology for Enterprise Integration Platforms – v3.0 – FINAL

Figure 13 — SDIA defines the invisible governance layer above technical artifacts, keeping domain ownership consistent across adapters, iFlows, metadata, Kafka topics, security, and runtime behavior.

Why the Metadata Comes First

In SAP Cloud Integration custom adapter development, the UI is not a secondary detail.

The metadata.xml file is the contract between the design-time UI and the Java runtime.

It defines:

• adapter variants
• sender/receiver direction
• tabs
• attribute groups
• field labels
• dropdown values
• default values
• visibility conditions
• runtime parameter names

If the metadata exposes a parameter that the Java endpoint does not understand, the adapter archive may still deploy, but the iFlow can fail when the endpoint is created.

Figure 3 — From the medata.xml to adapter UI visual

This is one of the most common custom adapter problems:

• ESA deploys successfully
• Channel appears in the iFlow
• iFlow deployment fail
• Endpoint cannot be created
• Unknown parameter or missing JavaBean property

That is why the metadata and the endpoint class must evolve together.

If the UI exposes this:

<AttributeReference>
<ReferenceName>schemaRegistryHostAddress</ReferenceName>
</AttributeReference>

Then the endpoint.class must expose this:

@UriParam
private String schemaRegistryHostAddress;

public String getSchemaRegistryHostAddress() {
return schemaRegistryHostAddress;
}

public void setSchemaRegistryHostAddress(String schemaRegistryHostAddress) {
this.schemaRegistryHostAddress = schemaRegistryHostAddress;
}

The name must match.The type must be compatible.The getter and setter must follow JavaBean conventions.

No mismatch.No guesswork.No hidden alias unless intentionally supported.

For EventSmartKafka, the current Consumer UI is organized into five major tabs:

• Connection
• Processing
• Conversion
• Conversion Advanced
• Advanced

Each tab exists for a reason.

The objective is not to expose hundreds of parameters. The objective is to expose the right parameters at the right level, with clear separation between connection, polling, conversion, and runtime behavior.

The metadata.xml Opened in the IDE

Figure 4 — Adapter metadata defines the design-time UI exposed inside the SAP Cloud Integration channel.

Endpoint Java Properties

Figure 5 — Endpoint Java properties must match the runtime parameter names defined in metadata.xml.

Result:

Figure 6 — Adapter UI of Authentication Methods that comes from the metadata.xml, read by java SdiaKafkaEndpoint.class after the compilation into .esa ( adapter form) 

The Component Class

The Component class is the adapter entry point.

Its job is not to process Kafka records.

Its job is to connect the adapter scheme with the Camel runtime and create endpoint instances.

Conceptually, it looks like this:

public class SdiaKafkaComponent extends DefaultComponent {

@Override
protected Endpoint createEndpoint(
String uri,
String remaining,
Map<String, Object> parameters) throws Exception {

SdiaKafkaEndpoint endpoint = new SdiaKafkaEndpoint(uri, this);
setProperties(endpoint, parameters);
return endpoint;
}
}

The important line is this:

setProperties(endpoint, parameters);

This is where Camel maps URI/metadata parameters into Java endpoint properties.

If the metadata sends a parameter that the endpoint does not expose, the endpoint creation may fail.

That is why the Component is simple, but critical.

It is the bridge between:

SAP Cloud Integration channel configuration
↓
Camel parameter map
↓
Endpoint JavaBean properties
↓
Consumer runtime behavior

A good Component class should stay boring.It should create the endpoint, apply parameters, and avoid business logic.

• Kafka logic does not belong here.
• Avro logic does not belong here.
• Schema Registry logic does not belong here.

Component Class

Figure 7 — The Component class creates the Endpoint and connects the adapter scheme with Apache Camel.

The Endpoint Class

The Endpoint class is the runtime configuration contract.

It receives, stores, validates, normalizes, and exposes the adapter configuration that comes from the SAP Cloud Integration channel UI.

For EventSmartKafka, the endpoint is responsible for configuration areas such as:

• Kafka bootstrap servers
• topic pattern
• partitions
• consumer group ID
• authentication mode
• SASL mechanism
• TLS flag
• credential aliases
• Cloud Connector Location ID
• conversion format
• Schema Registry host
• Schema Registry credential
• Schema ID resolution mode
• payload safety limits
• poll timeout
• fetch settings
• retry settings
• runtime timing parameters

The Endpoint should not poll Kafka.

The Endpoint should not decode Avro.

The Endpoint should not commit offsets.

The Endpoint should provide the Consumer with a clean, validated configuration model.

A production-minded endpoint should follow three rules:

• Fail fast on invalid configuration
• Normalize values once
• Expose explicit getters for runtime classes

For example, if the user selects PROTOBUF while Protobuf is not implemented yet, the adapter must not silently pass the record as raw bytes.

Avro Convertor possibilities:

It must fail clearly.

A silent fallback is worse than an exception.

This is especially important in event-driven architecture because silent failures can corrupt downstream business behavior while the monitor still looks green.

Endpoint Configuration Blocks

Figure 7 — SdkaKafkaEndpoint.class

Figure 8 — The Endpoint class receives and validates channel configuration before the Consumer runtime starts.

The Consumer Class: Where Kafka Meets CPI

The Consumer is the heart of the first public baseline.

In EventSmartKafka, the first release is Consumer-first.

This means the main validated flow is:

Kafka topic
↓
EventSmartKafka Consumer Adapter
↓
SAP Cloud Integration iFlow
↓
Downstream processing

The Consumer maps the Apache Camel ScheduledPollConsumer lifecycle to the Kafka KafkaConsumer polling model.

At runtime, the Consumer must:

• initialize configuration
• resolve credentials
• connect to Kafka
• subscribe to topics or assign partitions
• poll records
• optionally convert payloads
• create a Camel Exchange
• send the Exchange to the CPI pipeline
• commit the Kafka offset only after successful processing
• handle failures without silent data loss

The simplified runtime flow is:

KafkaConsumer.poll()
↓
ConsumerRecord<String, byte[]>
↓
raw passthrough or Avro conversion
↓
Camel Exchange
↓
Integration Flow pipeline
↓
successful processing
↓
commit offset

Reading from Kafka is not the hard part.

The hard part is committing the correct offset at the correct time.

 Consumer Class

Figure 9 — The Consumer maps the Camel polling lifecycle to Kafka record consumption.

Kafka to CPI Runtime Flow

Figure 10 — Consumer-first runtime flow from Kafka polling to SAP Cloud Integration message processing.

Why Offset Handling Is the Real Problem

A Kafka consumer can read a record in a few lines of code.

That is not adapter engineering.

Adapter engineering starts when something fails.

The key question is:

When is it safe to commit the offset?

For EventSmartKafka, the principle is simple:

Do not commit the offset until the message is successfully delivered to the CPI pipeline.

For raw passthrough, this means:

• Read record
• Create Exchange
• Send payload to CPI pipeline
• Pipeline succeeds
• Commit offset

For Avro conversion, this means:

1. Read record
2. Validate payload
3. Resolve Schema ID
4. Fetch schema
5. Decode Avro
6. Create JSON or XML
7. Send payload to CPI pipeline
8. Pipeline succeeds
9. Commit offset

If Avro conversion fails, the offset must not be committed.

Otherwise, the consumer group moves forward and the failed record is lost from the integration process.

That is exactly what a serious Kafka adapter must avoid.

The current behavior model is:

Scenario Offset Commit

This is why offset management belongs in the adapter layer.

If every iFlow must solve this independently, governance disappears.

Figure 11 — Offset commit must happen only after successful conversion and pipeline delivery.

Poison Pills and Silent Commits: The Two Failure Modes

A poison pill is a bad Kafka record that can be polled but cannot be processed. The consumer remains alive, but if the offset is not committed and no controlled strategy is applied, the same bad record is read again and again, blocking the next valid offsets.

A silent commit is the opposite problem. It happens when the adapter commits the offset even though the record failed. The consumer moves forward, the monitor may look green, but the failed message is lost.

EventSmartKafka is designed to make this behavior explicit: failed conversions are visible in CPI monitoring, poison messages are controlled by the selected error strategy, and offsets are committed only when processing is considered successful.

Figure 12 — Poison pills and silent commits are opposite failure modes: one blocks progress, the other hides data loss. A serious adapter must make both behaviors visible and controlled.

After iFlow Deployment: Runtime Lifecycle ASCII

[ SAP Integration Suite (CPI) ] Deploys iFlow
|
v
Custom Adapter ESA is loaded
|
v
[ Apache Camel ] Creates Component
|
v
SdiaKafkaComponent.createEndpoint(…)
|
v
metadata.xml parameters
are mapped into
SdiaKafkaEndpoint setters
|
v
SdiaKafkaEndpoint.createConsumer(…)
|
v
SdiaKafkaConsumer is created
|
v
doStart()
├── Validate Endpoint Configuration
├── Resolve CPI Secure Store Credentials
├── Build Native Kafka Client Properties
├── Instantiate KafkaConsumer Client
└── Subscribe to Topic / Assign Partitions
|
v
ScheduledPollConsumer Loop Starts
|
v
doPoll()
├── KafkaConsumer.poll() -> ConsumerRecord
├── Validate Payload Size Limits (Max 20MB)
├── Optional Conversion (AVRO/Protobuf -> JSON)
├── Create Camel Exchange & Inject Headers
└── Dispatch Exchange to CPI Pipeline
|
v
CPI iFlow Processing
|
+————+————+
| |
[ YES ]| |[ NO ]
v v
commitSync() Evaluate Error Strategy
| ├── 🔁 Retry Attempt
v ├── ⏭️ Skip Message
Next Poll Cycle └── 🚨 Stop Pipeline

Runtime Class Responsibility

• SdiaKafkaComponent: Creates the Endpoint and maps Camel parameters.
• SdiaKafkaEndpoint: Stores and validates channel configuration.
• SdiaKafkaConsumer: Starts Kafka connection, polls records, creates exchanges, controls offset commit.
• SdiaKafkaAvroConverter: Converts Avro binary payloads into JSON/XML when conversion is enabled.
• SdiaSchemaRegistryClient: Fetches and caches schemas by schema ID.

Compact Error Rendering

A custom adapter should not hide runtime failures behind generic Java stack traces.

For EventSmartKafka, conversion and payload validation errors are rendered in a compact CPI monitoring format.

The objective is to show the relevant runtime context directly in the monitor:

• topic
• partition
• offset
• bootstrap host
•  schema mode
• schema ID when available
• error code
• root cause

The current demo baseline includes visible error rendering for scenarios such as:

• Avro payload without Confluent Magic Byte
• invalid or unexpected Schema ID
• Schema Registry lookup failure
• payload above the configured conversion limit
• payload above the raw passthrough limit

This matters because poison messages must be visible, traceable, and controlled.

A failed conversion should not silently commit the offset and disappear from the integration flow.

Figure 13 — Compact CPI error for Avro payload without valid Confluent Magic Byte.

Figure 14 — Compact CPI error for payload above the configured conversion limit.

What Has Been Tested So Far

The current Consumer demo baseline was validated in controlled non-production environments using trial, free-tier, or limited cloud resources.

Validated scope includes custom ESA deployment, iFlow channel creation, Kafka consumer startup, SASL/PLAIN over TLS, SCRAM-SHA-256/512, raw passthrough, Avro to JSON/XML, Schema Registry integration, Confluent Magic Wire, Fixed Schema ID, compact CPI monitor errors, conversion failure without offset commit, payload limits, and schema response limits.

This is not enterprise stress testing. It would be unrealistic to claim that an independent open-source project, developed and tested by one person, has validated every broker, security profile, network condition, payload size, partition strategy, retry behavior, and failure scenario.

All broker validations used trial, free-tier, or limited environments. These tests prove functional behavior under controlled conditions, not enterprise-scale certification.

Validated connectivity:
Confluent Cloud, Redpanda, Aiven.

Explored / partially tested:
IBM Event Streams.

What Is Explicitly Not in the First ESA

The first public ESA is Consumer-only.

It does not include validated Producer / Receiver role, Protobuf conversion, OAuth 2.0 production validation, full mTLS production validation, enterprise-scale performance certification, long-running endurance testing, vendor SLA, or SAP certification.

This is intentional. A community release should not mix validated functionality with roadmap items.

Figure 15 — EventSmartKafka – Custom Kafka.esa

What Part III Will Cover

Part III will explain how to create the custom adapter project from zero using SAP ADK and will provide the first downloadable .esa package on GitHub as a Community Demo / Experimental Technical Preview.

Users will be able to deploy the ESA in SAP Integration Suite, create a test iFlow, configure the EventSmartKafka Sender Adapter, and validate the Consumer runtime behavior.

The release will include basic documentation, configuration guidance, known limitations, and controlled test instructions.

The complete Java source code will be opened progressively in the next parts of the series.

Figure 16 — From blog to code to integration: Part III will provide the first downloadable ESA package on GitHub for controlled deployment and testing in SAP Cloud Integration.

Conclusion

Part I explained why EventSmartKafka exists.

Part II opened the runtime box and showed what happens after an iFlow is deployed: metadata.xml becomes UI, UI parameters become Endpoint properties, the Consumer starts, doStart() prepares the Kafka client, doPoll() reads records, and offset commit is controlled after processing.

The main lesson is simple: reading from Kafka is easy. Safe runtime behavior is the real adapter engineering challenge.

That means controlling poison pills, avoiding silent commits, exposing compact CPI errors, and committing offsets only when processing is considered successful.

EventSmartKafka remains a Community Demo / Experimental Technical Preview: not SAP-certified, not SLA-backed, and not a replacement for vendor-supported adapters.

Part III will explain how to create the SAP ADK adapter project from zero and will provide the first downloadable ESA package on GitHub for controlled testing.

The journey continues in Part III.

Part I – SAP BTP – CI – How I Built an Open-Source Enterprise Custom Kafka Adapter from Scratch – Part I

Ricardo Viana
Independent Solo Researcher

Creator of DEIP · SDIA · GDCR · DDCR · ODCP · EDCP · DDCP

#SAPIntegrationSuite #SAPCloudIntegration #Kafka #SAPADK #OpenSource #EventDrivenArchitecture #ApacheKafka #Avro #SchemaRegistry #CloudIntegration #SAPCommunity

“}]] 

  Read More Technology Blog Posts by Members articles 

#abap