Amazon Bedrock AgentCore in Two Phases

On our AI platform for business lenders, I proposed running the agents on Amazon Bedrock AgentCore instead of hosting them ourselves. The work then came in two phases:

Phase 1: the first agents, deployed the way the AgentCore CLI suggests: one agentcore.json spec, synthesised by the L3 CDK library. It worked for one developer and one environment, and then stopped scaling.
Phase 2, three months later: we moved the whole chat experience onto AgentCore, on L2 constructs we control. That meant one gateway in front of every tool, a managed knowledge base scoped per organisation, and deciding what should not run on AgentCore.

This isn’t a tutorial. The AgentCore CLI docs cover the happy path well. These are the decisions I’d make again, and the one I’d make sooner.

Why AgentCore

We wanted managed infrastructure for the parts every agent needs and nobody wants to build twice:

Runtime: each agent runs as a container in an isolated session, and AWS handles scaling.
Identity: third-party API keys and OAuth credentials live in AgentCore Identity, not in environment variables.
Gateway: tools are exposed to agents over MCP, with IAM auth in front of them.
Code Interpreter and Browser: sandboxes managed by AWS, so agents can run code and browse without us securing them.
Observability: OpenTelemetry traces show up in CloudWatch without a separate tracing stack.

Phase 1: one spec, the L3 library

The CLI keeps a whole project in agentcore/agentcore.json: runtimes, memories, credentials, gateways and evaluators, all as top-level arrays. agentcore dev runs an agent locally, and agentcore deploy synthesises the spec with the L3 library @aws/agentcore-cdk. For a first agent, that’s a great experience.

Where it broke: more than one environment

We don’t deploy to one place. Every agent had to exist in dev, UAT and production, and in a sandbox stack for each developer, several of them in the same AWS account. agentcore.json describes one project, though, and doesn’t know about any of that. So I wrapped the L3 library in our own construct that read the spec at synth time and rebound it for each stack:

Network: each runtime’s VPC subnets and security group were swapped for the stack’s real resources. Developers shared the dev VPC but each had their own security group.
File mounts: each runtime’s S3 Files mounts were rewritten to the access points created in that stack.
Names: every runtime name got a per-developer suffix (ChatAgent_alice), so sandboxes didn’t collide. Anything that called a runtime by name had to know that suffix too.
Environment variables: the spec only held variable names. The real values came from the stack and were filtered per runtime against those names, partly because a runtime allows at most 50 variables.
Project root: the library finds agentcore.json by walking up from the current directory, and cdk runs from a different folder, so the root had to be pinned explicitly.

Each of those was reasonable on its own. Together they meant a JSON file that was never what got deployed: every environment was a deep clone of it, rewritten in code. Then we hit a library bug. It built an IAM policy’s resource ARN with a string regex, which silently does nothing on an unresolved CDK token, so the policy was scoped to the wrong resource and every file mount was denied. Working around that meant adding the correct IAM statement ourselves, next to the library’s.

At that point the spec wasn’t a single source of truth anymore. It was an input we kept fighting.

Smaller lessons worth keeping

A few things from phase 1 still apply, whichever way you deploy:

Packaging installs wheels only. Zip deployments run --only-binary :all:, so a dependency that ships only a source distribution fails everywhere, even if it’s pure Python. We vendored one wheel for docling’s pylatexenc dependency. Pin Python too (>=3.12,<3.13).
Tracing needs the distro. If aws-opentelemetry-distro isn’t in your agent’s own dependencies, nothing errors, but no spans arrive. If you also export to Langfuse, its default filter drops LangChain’s chain and tool spans until you allow that scope.
Fail fast on configuration. Validate the variables an agent needs at startup, instead of halfway through its first invocation.
One Code Interpreter sandbox per request. Our coding agent uses a fresh session per request, returns structured output (code, files, summary) instead of parsing the tool trace, and copies the files it created to S3 before cleaning up the sandbox.

Phase 2: making it the platform

Three months later we decided to move the whole chat experience onto AgentCore, not just a few side agents. Until then, chat ran as LangGraph workflows inside our API service, with our own vector store next to it. Five things pushed us:

Problems at scale: long agent runs (tool loops, code execution, document reads) shared CPU, memory and timeouts with ordinary API requests. One heavy conversation could slow down everyone else’s. AgentCore Runtime gives each session its own isolated microVM and scales them independently of the API.
Stateful chat: a conversation should keep its context between messages without us rebuilding it on every request. Runtime sessions stay warm between turns (the team settled on a 15-minute idle timeout and an 8-hour maximum), and AgentCore Memory restores history on a cold start. Teammates later had our API prepare the session in the background when a chat opens, so the first message doesn’t pay the start-up cost.
A built-in harness, browser and code interpreter: AgentCore’s harness gave us a working agent loop with managed Browser and Code Interpreter tools, iteration and token limits, and summarising truncation, all from configuration. We prototyped the chat agent as a harness, then exported it to a container runtime once it settled.
A managed knowledge base: a Bedrock knowledge base handles parsing, embedding, reranking and hybrid search, and plugs into the gateway with no code. That’s one less datastore for us to run.
Better security with IAM: every hop (agent to gateway, gateway to tool runtime, gateway to knowledge base) is authorised by IAM roles and SigV4 instead of shared API keys. A tool runtime can only be called by the gateway role, and the user’s own token still travels with each call so the API can apply its permissions.

The question then changed from “can an agent run?” to “how do several agents, MCP servers and a knowledge base get deployed to every environment, and to every developer’s sandbox, safely?”

Moving to L2 constructs

So in phase 2 I removed @aws/agentcore-cdk and agentcore.json from deployment altogether, and wrote a few small constructs on the L2 AgentCore classes in aws-cdk-lib (Runtime, Gateway, GatewayTarget, Memory):

Container runtime: builds a local Dockerfile as an asset, always for linux/arm64, because that’s the only platform AgentCore runs.
Gateway: an MCP gateway with IAM inbound auth, so runtimes call it with SigV4. It starts with no targets; each tool source attaches itself.
Chat agent: a runtime wired to the shared gateway. Everything environment-specific is a prop, so nothing is read from a JSON file at deploy time.
A nested AgentsStack: one shared gateway plus the agents, dropped into each environment stack and each developer’s sandbox stack, behind feature flags.

The environment problems went away because there’s nothing to rebind: subnets, security groups, names and variables are ordinary props, passed in by whichever stack creates the construct. Names put the target first (<env>-ChatAgent, <developer>-AgentsGateway), so sandboxes share one account without collisions and sort together in the console. Removing the old construct and its helpers deleted nearly 500 lines.

We still use the CLI for prototyping. Our chat agent started as a CLI-generated harness and was exported to a plain container runtime once it settled.

One gateway in front of every tool

Every tool an agent can reach goes through the one gateway: MCP servers for our APIs, the knowledge base, and third-party integrations. That gives us one place for auth and one list of what agents can touch.

The chat agent signs every MCP request with SigV4 using its runtime role. But our APIs still need to know which user is asking. Two details made that work:

The gateway owns the Authorization header, and AgentCore only forwards headers named ...-Runtime-Custom-* through to a runtime. The user’s short-lived token travels in one of those, and the API behind the MCP server reads it as a fallback to Authorization.
Each tool runtime keeps IAM inbound auth, so only the gateway role can call it. Per-user authorisation stays in the API, which verifies the forwarded token.

On the agent side, a custom httpx.Auth signs each request and adds the caller’s token and organisation. AWS credentials are resolved once and reused; botocore’s refreshable credentials rotate the role keys themselves. My first version kept the caller on the auth object. A teammate later moved it into a ContextVar set per invocation and reset in a finally, because one runtime session can serve concurrent requests, and that’s the version to copy.

token = gateway_auth.set_caller(GatewayCaller(user_token=..., organization_id=...))
try:
    async for event in adapter.run(run_input):
        yield event
finally:
    gateway_auth.reset_caller(token)

I also set continue_on_error=True on the gateway’s MCP client. If the gateway is down or IAM is misconfigured, the agent starts without gateway tools instead of failing every chat.

A managed knowledge base, scoped by the server

Each lender’s policies used to live in our own vector store. I moved them to a Bedrock-managed knowledge base, with Cohere Embed v4 through a global inference profile at first and managed parsing, embeddings and reranking later. I attached it to the gateway through the native knowledge-base connector, so there’s no code in between:

agent --SigV4--> gateway --gateway role--> bedrock:Retrieve

Only Retrieve is exposed. The hard requirement was that one lender can never retrieve another lender’s documents, and a model can’t be trusted to set its own filter. So:

Every document is ingested with its owner organisation as metadata.
Our API builds the retrieval filter from the session’s organisation and passes it to the agent outside the prompt.
A hook on the agent rewrites every Retrieve call with that filter, replacing whatever the model sent, and refuses the call if there’s no filter.

The model can choose the query and the result count, but it can’t widen its own scope.

Switching storage also meant a migration: backfilling existing documents into the knowledge base, moving legacy files under organisation-scoped keys, and cleaning up records whose files were gone. Ingestion is asynchronous, so the app now shows queued and indexing stages on each document instead of pretending it’s ready.

Not everything belongs on AgentCore

At one point document classification ran as an AgentCore agent, started from the upload request. It worked, but it isn’t an agent. It’s a fixed pipeline: OCR, classify, chunk, embed, store. It needs retries and a dead-letter queue more than it needs a reasoning loop.

So I moved it to an SQS-driven Lambda that processes one document per invocation:

Bedrock does OCR on page images and classifies each document once.
Documents are chunked per page with stable IDs, so re-processing replaces chunks instead of duplicating them.
Document rows are settled with conditional writes, so a retry can’t overwrite a newer result.
Uploads are stored under content-hash keys, so the same file uploaded twice is processed once.
A dead-letter handler marks timed-out documents as failed, and a small script requeues stuck uploads.

That made the line clear: agents and their tools run on AgentCore, and batch work runs on plain Lambda with Bedrock.

What I’d do differently

Deploy a “hello world” agent with tracing on day one. Packaging and observability problems are much easier to find before real dependencies arrive.
Plan for many environments from the first deploy. The CLI and its spec are great for a prototype. Once you need dev, UAT, production and per-developer sandboxes, go straight to L2 constructs with environment values as props.
Decide the auth story before adding tools. “Whose permissions does this tool call use?” is much harder to retrofit, and scoping should be enforced by the server, never left to the model.
Ask whether it’s really an agent. If the steps are fixed, a queue and a function are simpler and cheaper.

AgentCore removed a lot of infrastructure we’d otherwise have built ourselves: sandboxes, credential storage, an MCP gateway, a vector store and tracing. What’s left is mostly ordinary engineering: packaging, configuration, auth and knowing where the edges are.