AWS published a hands-on walkthrough that shows engineering teams how to run a self-hosted open model inside Amazon Bedrock AgentCore and, just as importantly, how to see what that model actually costs and how long it takes. The post focuses on a specific gap: Amazon Bedrock AgentCore Runtime automatically instruments the Claude models it calls, but it leaves token usage for models hosted on Amazon SageMaker AI almost invisible unless you add custom telemetry.
The combination matters because most production agent systems are not built from a single model. Teams want to keep Claude for orchestration and reasoning on Amazon Bedrock while routing cheaper, domain-specific, or data-resident workloads to a model they control on SageMaker AI. The AWS guide demonstrates exactly that split using a personal-finance example, with the full walkthrough published on the AWS Machine Learning blog.
What the reference architecture actually does
The design connects three model-hosting paths through one Bedrock AgentCore container. An orchestrator agent running Claude Haiku 4.5 on Bedrock classifies the user’s intent and routes work to two specialists, according to the AWS walkthrough. A budget agent runs Claude Sonnet 4.6 on Bedrock and returns structured 50/30/20 budget breakdowns. A financial-analysis agent runs Qwen 3.5 9B on a SageMaker AI real-time endpoint through the OpenAI-compatible API and handles stock analysis with tool-calling. The point of mixing them is cost optimization, data residency, and model flexibility in a single production-ready architecture, as AWS describes in the post.
To make that concrete, the guide deploys Qwen 3.5 9B with the vLLM Deep Learning Container image vllm:0.22.1-gpu-py312-cu130 on an ml.g6e.2xlarge instance, a single L40S GPU with 48 GB of VRAM, and a 32,768-token context window, details that appear in the AWS deployment steps. The SageMaker endpoint is reached from inside AgentCore through an OpenAI-compatible base URL, with bearer tokens refreshed on every request via an httpx.Auth subclass.
The observability gap nobody talks about
Bedrock AgentCore Runtime instruments agents with OpenTelemetry automatically. For model calls made to Amazon Bedrock through boto3, that instrumentation produces full generative-AI spans with token counts and needs no extra code, the post explains. The problem is the other path: a SageMaker OpenAI-compatible endpoint reached through the Strands OpenAIModel provider is not recognized as a generative-AI call, so its tokens never show up in traces. The financial-analysis agent calling Qwen 3.5 9B becomes effectively invisible — you cannot monitor its cost, detect regressions, or debug latency.
The root cause is narrow: Strands’ OTEL integration emits spans for tool calls and agent lifecycle events, but it does not emit gen_ai.chat spans with token attributes for the OpenAI-compatible provider, and AgentCore’s auto-instrumentation only treats Bedrock inference as generative-AI work, AWS writes.
The fix: a custom span and one missing parameter
The workaround is small but easy to miss. You manually emit a gen_ai.chat span that wraps the SageMaker agent invocation and pulls token usage out of Strands’ internal AgentResult.metrics.accumulated_usage, which carries inputTokens, outputTokens, and totalTokens, as shown in the AWS code sample. In the example trace the call consumes 1,391 input tokens and 1,432 output tokens for 2,823 total tokens over roughly 37 seconds, numbers that only appear once the custom span is in place.
There is a second, separate trap. By default vLLM does not include a usage chunk in streaming responses, so Strands receives text but never a final usage object and accumulated_usage stays at zero. Adding stream_options: {"include_usage": True} to the OpenAIModel parameters forces vLLM to send a final usage chunk, the walkthrough notes. Skip that parameter and the custom span still reports zero tokens.
Once the spans land in the AgentCore Observability dashboard, the payoff is direct. The custom gen_ai.chat span for the SageMaker-hosted Qwen model appears next to the automatically instrumented Bedrock spans, and both now carry token counts, AWS shows in its trace view. That side-by-side view is what lets a team compare cost per agent, spot a regression when the financial-analysis agent suddenly burns more tokens, and attribute spend to the right model instead of a single undifferentiated Bedrock bill.
Getting the environment right
A few operational details decide whether the telemetry is useful at all. AWS X-Ray samples at 1 percent by default, which drops most traces; the guide sets a 100 percent indexing rule during development so the spans are actually retained, according to the AWS configuration steps. The accompanying code also builds a fresh agent instance per request because singletons cause concurrent-invocation errors, and it sets AGENT_OBSERVABILITY_ENABLED=true plus opentelemetry-instrument as the container command.
Model availability is not uniform, and the guide points readers to Amazon Bedrock’s regional model support documentation for which Regions serve Claude Haiku 4.5 and Claude Sonnet 4.6. The sample deployment itself targets ap-south-1 for AgentCore and us-west-2 for the SageMaker endpoint, choices documented in the AWS post. Python 3.12 or newer is required for the Strands and SageMaker tooling.
The authentication detail that breaks in production
Reaching a SageMaker OpenAI-compatible endpoint from inside AgentCore is not a one-time secret exchange. SageMaker bearer tokens expire, so any long-running agent session needs a way to refresh them on every request, the post notes in its connection setup. AWS solves this with an httpx.Auth subclass that calls generate_token() per request and injects the fresh bearer header before the call leaves the container. Skip this layer and the endpoint works in a quick test but fails intermittently once the first token expires — a failure mode that is painful to debug because the trace shows a generic auth error rather than a design gap.
When the split architecture is worth the overhead
This is not a default architecture. The AWS post frames the SageMaker-plus-Bedrock split around three concrete motivations: cost optimization by routing heavy reasoning to a cheaper self-hosted model, data residency for regulated workloads that must stay on infrastructure you control, and model flexibility to drop in domain-specific or fine-tuned checkpoints, as AWS states in the introduction. If none of those pressures apply, running every agent on Bedrock is simpler and keeps full automatic telemetry. The custom-span work only pays off when the self-hosted model carries real traffic or real compliance weight.
Where the working code lives
The complete reference implementation is published as a public AWS Samples repository that walks through deploying the SageMaker endpoint, building the multi-agent orchestration, and shipping it to AgentCore Runtime, hosted on GitHub. It includes the observability notebook that explains how to assemble the custom spans end to end.
The repository is organized as three graded labs rather than one monolith, which makes it usable as a learning path. Lab 1 deploys and tests the SageMaker endpoint, Lab 2 builds the multi-agent orchestration with the budget and financial-analysis agents, and Lab 3 handles the AgentCore Runtime deployment and the OBSERVABILITY.md reference, all laid out in the GitHub repository. A team can stop after Lab 1 to validate the SageMaker-to-AgentCore authentication and token-refresh path before committing to the full multi-agent build.
How teams can extend the pattern
The architecture is deliberately composable. AWS outlines swapping in fine-tuned checkpoints by pointing SM_VLLM_MODEL at an S3 artifact while leaving the auth layer, OTEL spans, and AgentCore deployment unchanged, and suggests A/B testing base and fine-tuned variants on one endpoint with a variant attribute on the span, per the post’s extension notes. A practical next step is cost-aware routing: check query complexity before dispatch and send simple lookups to Haiku on Bedrock while reserving the SageMaker GPU endpoint for multi-step reasoning, an approach AWS recommends.
Why this belongs in the AgentCore story
This walkthrough sits inside a quickly growing Bedrock AgentCore ecosystem. AWS has also added temporal policies and rate limits that let teams control agent behaviors and cost beyond a single action, covered in our write-up of AgentCore’s new controls. For teams that would rather orchestrate agents from a low-code layer, our guide to running production AI agents in n8n with AgentCore shows a different on-ramp to the same runtime.
Don’t forget to tear down
Because the pattern provisions a GPU endpoint and an AgentCore runtime, the guide includes explicit cleanup: delete the AgentCore agent runtime, the SageMaker endpoint, its endpoint config, and the model, listed in the AWS conclusion. Leaving the ml.g6e.2xlarge endpoint running is the easiest way to turn a demonstration into a recurring bill.
What a technology decision-maker should take away
The durable lesson is not the specific models. It is that mixing a managed foundation model with a self-hosted one inside one AgentCore container is now a documented, repeatable pattern — but only if you close the telemetry gap yourself, as the AWS walkthrough demonstrates. The post proves the architecture works: a 48 GB GPU instance serves Qwen 3.5 9B, Claude handles routing and budgeting, and a few lines of OpenTelemetry recover the token and latency data that the default instrumentation throws away. Teams evaluating AgentCore for cost-sensitive or data-resident workloads should budget for that custom-span work up front, set X-Ray sampling to 100 percent in development, and treat the GitHub sample as the starting template rather than a finished product.
