#DevOps #Transport #Automation #Git
Overview:
SAP Datasphere provides built-in transport capabilities for promoting artifacts between tenants along with cTMS which is another tool that can be leverage for objects promotions across a multi-tier landscape. However, for teams already working with Git-based workflows where peer review, branching strategies, version control and deployment pipelines are standard practice, there is a natural desire to bring that same discipline to Datasphere artifact lifecycle management.
This blog describes how SAP Datasphere CLI (@sap/datasphere-cli) can be used to build a lightweight, Git-native transport pipeline that fits into existing developer workflows and automates this process end-to-end.
Followed Approach:
The pipeline is built around a simple principle: Git as the intermediary or staging layer. Artifacts are exported from a source Datasphere tenant, versioned in GitLab, and imported into a target tenant, all via scripted datasphere-cli calls. No manual file handling, no browser exports.
The workflow runs in two directions:
- DSP → Git (Pull): A dedicated script connects to the source tenant, discovers all artifacts by specified types, and compares the full content hash of each artifact against the locally stored version. Only genuinely changed artifact or new artifacts are written to an export folder. Once the pull completes, the developer reviews the changes in VS Code Source Control, selects which files to stage, and pushes to GitLab giving full control and flexibility over what gets pulled from Datasphere and versioned.
- Git → DSP (Push): A second script scans Git history for changed/new files and presents an interactive numbered list, allowing the user to select exactly which artifacts to transport. It then imports each artifact into the target tenant using datasphere-cli create command, falling back to update on conflict, and verifies each import with a post-import read. A full audit log captures every outcome.
How does datasphere-cli make it possible?
Beyond artifact transport, datasphere-cli exposes a broad set of capabilities that can be leveraged to automate surrounding workflows:
- Artifact lifecycle: discover, read, create, update across all major object types including local tables, views, replication flows and analytic models
- Package management: list, create, update, read and export package definitions for dependency-aware transport
- CI/CD integration: the pipeline can be fully automated as every transport operation runs without manual intervention, making it straightforward to integrate into existing CI/CD workflows.
Workflow diagram:
Pre-requisites:
- SAP Datasphere tenant(s)
- OAuth client credentials configured in SAP Datasphere, with Interactive Usage
- SAP Datasphere CLI installed
- Git repository (GitLab or GitHub)
- Python 3.9 or above
Authentication:
- SAP Datasphere Access is maintained using an OAuth client (client ID and secret) created in Datasphere tenant. These credentials can be stored in the config file or as environment variables which is recommended to avoid storing secrets in plain text. datasphere-cli establishes a session on first run and caches it locally, so subsequent runs reuse the existing session without re-authenticating.
- GitLab Access is authenticated using a Personal Access Token generated in GitLab. The token is stored in Windows Credential Manager and picked up automatically by Git on every push and pull, with no configuration needed in the scripts.
Dependency Ordering
Transporting multiple artifacts in the right order is crucial as a view cannot be imported before the table it references. By leveraging datasphere-cli objects packages list, create, read and export commands, package manifests can be pulled alongside artifacts. These manifests capture which objects belong to each package and their dependencies, enabling the pipeline to resolve the correct import sequence automatically using a topological sort. Teams that organise their artifacts into Datasphere packages benefit from fully automatic dependency ordering with no manual sequencing required.
Change Detection
Two approaches are supported for detecting whether an artifact has changed since the last pull:
- Hash comparison: computes a SHA-256 hash of the full artifact content and compares it against the locally stored version. Only genuine content changes trigger a write. This is the recommended default.
- Timestamp comparison: compares Datasphere’s internal deployment timestamp against the local file. Less precise as redeploying an object in Datasphere without structural changes still updates its timestamp, triggering an unnecessary write.
Audting:
Every action (pull, push) is written to a structured log file with a timestamp, status, and detail, making it straightforward to trace what happened and when across any transport run.
Important Considerations:
- Authentication Expiry: OAuth client credentials configured for each tenant have a maximum validity period of 180 days. Once expired, all pipeline operations (pull, push) will fail to authenticate. It is recommended to track the expiry date of each client credential and renew them in advance to avoid unexpected transport failures.
- Merge requests/pre-import conflict detection: Before importing, the pipeline checks whether the artifact already exists in the target space. Three outcomes are possible: the artifact is new and gets created, it exists with identical content and is skipped, or it exists with different content and the operator is prompted to resolve the conflict before proceeding.
- Credentials handling (e.g connections, access tokens): Connection definitions can be transported as part of the pipeline, credentials are never stored in the exported artifact but can be retrieved securely at import time from a secrets manager (e.g. AWS Secrets Manager, Azure Key Vault, etc) keeping the transport fully automated without exposing sensitive credentials in Git.
- Transport across spaces: The same pipeline logic applies to transporting artifacts across spaces within the same tenant. Rather than pointing the source and target configurations to different tenants, both can be configured with the same host but different space IDs. Pull, conflict detection, and push all behave identically. Git remains a useful intermediary in this scenario, providing a version-controlled audit trail of what moved between spaces and when.
- Scalability and performance: Initial tests covered local tables, views, and flows transported individually and as part of a package with consistent results. For larger spaces, performance scales linearly as artifacts are processed one at a time, and future iterations could address this through parallelisation and batched API calls. For high-volume migrations, a performance comparison against cTMS would be a worthwhile exercise and the right choice ultimately depends on whether the team favours a Git-native workflow or centrally managed package releases.
Conclusion:
This approach is not a replacement for Datasphere’s native transport capabilities or cTMS. It is a solution for teams running Datasphere through automation or DevOps pipelines. Version control, branching, code review, and automated deployment are now part of how Datasphere artifacts can be managed, all made possible through datasphere-cli.
#SAPCHANNEL