[[{“value”:”
First question: is browser inference even real?
It is. Libraries like Transformers.js and ONNX Runtime Web already run models directly in the browser on top of WebAssembly and WebGPU, no server needed.
Here’s roughly what it looks like from a developer’s side:
import { pipeline } from “@huggingface/transformers”;
const generator = await pipeline(“text-generation”, “onnx-community/Phi-3.5-mini-instruct-onnx-web”);
const result = await generator(“Summarize this text in one sentence:”);
Then reality kicks in
The moment you try to build something real on top of these libraries, the gaps show up.
They’re execution engines, not systems. You hand them a model and an input, they give you an output. That’s the whole contract. Everything around it is your problem: where the model comes from, how it’s stored, which variant to pull, what happens when a user cancels a generation halfway through, how to clean up afterward. Every app that uses these libraries ends up rebuilding the same plumbing.
There’s also a bigger constraint. You can’t really go fully local in an enterprise setting. Some tasks need more than an on-device model can do well, and some situations call for a cloud service as a backup. So a real system has to handle both local and cloud, ideally without the application having to know or care which one answered.
Those were the gaps I wanted to close.
What I built
This turned into two separate things. They started from the same question — does AI have to run in the cloud? — but ended up going in different directions.
Exploration 1: running models in the browser (an npm package)
The first thing I wanted to settle was whether the original idea held up at all. Can you actually run a real model inside a browser tab?
You can. What came out of it was a standalone npm package on top of Transformers.js and ONNX Runtime Web. The package adds a task-aware layer over those libraries so you don’t have to wire each one up by hand. Out of the box it covers five task types — text generation, speech recognition, speech synthesis, emotion detection, and image classification — each with its own handler that knows how to set up and run that particular task.
Whatever task you’re on, the API stays the same: initialize → generate → interrupt → reset → dispose. There’s an interrupt() that stops a generation mid-stream without reloading the model, which sounds minor until you have a user staring at a wall of text they want to cut off. This part stands on its own. It was an early experiment, and it isn’t wired into the larger system I’ll describe next.
Exploration 2: a flexible runtime for local and cloud AI
Once the browser idea checked out, I got curious about a broader version of the question. What if “local” didn’t mean the browser specifically? Any model running on local hardware and exposing an API could play the role of the local side — a small LLM under Ollama, a vision model like YOLO, a speech (ASR/TTS) service, whatever the use case needs. And once you treat local models that way, you can build something more generally useful: a layer that sits between whatever’s running on the edge and the AI services in the cloud, and lets you swap between them without rewriting the app.
Here’s how the whole thing fits together:
The split down the middle is the important part. Everything on the left runs on the edge — inside your own network, on your own hardware. Everything on the right lives in the cloud or on SAP BTP. The runtime spans both and lets a single application draw on either side without knowing where the work actually happened.
It breaks down into three layers.
The routing layer is the AI Gateway. Every request lands here, and the gateway decides where it goes. It carries the routing engine plus an admin dashboard for configuring everything. From here, a request can go one of three ways: out to an external provider like OpenAI or Gemini, over to SAP AI Core, or down to a model running locally. Keeping SAP AI Core as a first-class target matters — it means you can keep AI traffic inside SAP’s own infrastructure and still fall back to a local model when you want to.
There are two ways to set up a route. With fixed routing, you point a route at one target, and everything goes there. With fallback routing, you set a cloud target as primary and a local service as backup, and if the cloud call fails for any reason, the gateway quietly falls back to local. The app doesn’t have to know the cloud call failed. Either way, switching a route from cloud to local (or back) is a config change, not a code change.
The gateway also handles credentials for cloud routes. If there’s a valid token cached, it attaches it and forwards the request. If the token’s expired, it fetches a new one behind the scenes and then forwards. The app never touches auth.
That last part is where model management comes in. It handles the lifecycle of models on the edge: downloading, storing, keeping track of what’s available, and cleaning up.
A management UI ties it together: a dashboard for setting up routes, kicking off model downloads, and configuring cloud provider credentials, so you’re not editing config files by hand.
Where it actually stands
To be clear about it: this is a prototype. The whole thing runs end to end in a local demo — routing, inference, model lifecycle, the dashboard, all of it works. But it hasn’t been deployed anywhere in production or tested against a real SAP business scenario.
The point of this stage was to find out whether the architecture is sound and to pin down how the pieces work in enough detail to build on later. I think it holds up, but the honest test is someone putting it in front of a real workload.
Get in touch
If you’re poking at AI integration, edge or offline deployment, or you’re just curious where browser-based inference might fit in an SAP setting, I’d be glad to compare notes. Reach out or drop a comment.
“}]]
const generator = await pipeline(“text-generation”, “onnx-community/Phi-3.5-mini-instruct-onnx-web”);
const result = await generator(“Summarize this text in one sentence:”);All of that runs on the user’s machine. The model downloads once, gets cached, and after that every call is local. No round trip, no API key, no per-request bill.For a lot of enterprise scenarios that’s an appealing tradeoff. The data never leaves the device, so a whole class of privacy and compliance concerns just goes away. It keeps working when the network doesn’t. There’s no inference cost per call. And responses come back immediately.Then reality kicks inThe moment you try to build something real on top of these libraries, the gaps show up.They’re execution engines, not systems. You hand them a model and an input, they give you an output. That’s the whole contract. Everything around it is your problem: where the model comes from, how it’s stored, which variant to pull, what happens when a user cancels a generation halfway through, how to clean up afterward. Every app that uses these libraries ends up rebuilding the same plumbing.There’s also a bigger constraint. You can’t really go fully local in an enterprise setting. Some tasks need more than an on-device model can do well, and some situations call for a cloud service as a backup. So a real system has to handle both local and cloud, ideally without the application having to know or care which one answered.Those were the gaps I wanted to close.What I builtThis turned into two separate things. They started from the same question — does AI have to run in the cloud? — but ended up going in different directions.Exploration 1: running models in the browser (an npm package)The first thing I wanted to settle was whether the original idea held up at all. Can you actually run a real model inside a browser tab?You can. What came out of it was a standalone npm package on top of Transformers.js and ONNX Runtime Web. The package adds a task-aware layer over those libraries so you don’t have to wire each one up by hand. Out of the box it covers five task types — text generation, speech recognition, speech synthesis, emotion detection, and image classification — each with its own handler that knows how to set up and run that particular task.Whatever task you’re on, the API stays the same: initialize → generate → interrupt → reset → dispose. There’s an interrupt() that stops a generation mid-stream without reloading the model, which sounds minor until you have a user staring at a wall of text they want to cut off. This part stands on its own. It was an early experiment, and it isn’t wired into the larger system I’ll describe next.The code is here: github.tools.sap/ICN-China/ondevice-aiExploration 2: a flexible runtime for local and cloud AIOnce the browser idea checked out, I got curious about a broader version of the question. What if “local” didn’t mean the browser specifically? Any model running on local hardware and exposing an API could play the role of the local side — a small LLM under Ollama, a vision model like YOLO, a speech (ASR/TTS) service, whatever the use case needs. And once you treat local models that way, you can build something more generally useful: a layer that sits between whatever’s running on the edge and the AI services in the cloud, and lets you swap between them without rewriting the app.Here’s how the whole thing fits together:The split down the middle is the important part. Everything on the left runs on the edge — inside your own network, on your own hardware. Everything on the right lives in the cloud or on SAP BTP. The runtime spans both and lets a single application draw on either side without knowing where the work actually happened.It breaks down into three layers.The routing layer is the AI Gateway. Every request lands here, and the gateway decides where it goes. It carries the routing engine plus an admin dashboard for configuring everything. From here, a request can go one of three ways: out to an external provider like OpenAI or Gemini, over to SAP AI Core, or down to a model running locally. Keeping SAP AI Core as a first-class target matters — it means you can keep AI traffic inside SAP’s own infrastructure and still fall back to a local model when you want to.There are two ways to set up a route. With fixed routing, you point a route at one target, and everything goes there. With fallback routing, you set a cloud target as primary and a local service as backup, and if the cloud call fails for any reason, the gateway quietly falls back to local. The app doesn’t have to know the cloud call failed. Either way, switching a route from cloud to local (or back) is a config change, not a code change.The gateway also handles credentials for cloud routes. If there’s a valid token cached, it attaches it and forwards the request. If the token’s expired, it fetches a new one behind the scenes and then forwards. The app never touches auth. The local model infrastructure is the runtime on the edge that actually hosts models — the LLMs, vision, and speech services mentioned above, side by side. What’s interesting is where those models come from: they get deployed down into the local runtime, so the cloud isn’t only a place to route inference to; it’s also the source you pull models from before running them locally. Two sources are on the map here. One is a public repository like Hugging Face, and that’s the path the current PoC actually uses — models are downloaded straight from Hugging Face. The other is SAP AI Core, shown in the diagram as a deploy source; that path isn’t built yet. It’s the direction I want to take next, since it would let you stage and version models through SAP’s own infrastructure instead of pulling from a public hub, but for now it’s intent, not implemented.That last part is where model management comes in. It handles the lifecycle of models on the edge: downloading, storing, keeping track of what’s available, and cleaning up.A management UI ties it together: a dashboard for setting up routes, kicking off model downloads, and configuring cloud provider credentials, so you’re not editing config files by hand.The code for the whole system is here: github.tools.sap/ICN-China/edge-explorationWhere it actually standsTo be clear about it: this is a prototype. The whole thing runs end to end in a local demo — routing, inference, model lifecycle, the dashboard, all of it works. But it hasn’t been deployed anywhere in production or tested against a real SAP business scenario.The point of this stage was to find out whether the architecture is sound and to pin down how the pieces work in enough detail to build on later. I think it holds up, but the honest test is someone putting it in front of a real workload.Get in touchIf you’re poking at AI integration, edge or offline deployment, or you’re just curious where browser-based inference might fit in an SAP setting, I’d be glad to compare notes. Reach out or drop a comment. “}]] Read More Technology Blog Posts by SAP articles
#SAPCHANNEL