What We Learned Building Our Own Agent Harness

Getting an agent to demo well is easy. Take a model, give it a few tools, and on your laptop it looks like magic. Getting the same agent through a month of real users, real data and real edge cases is a different job.

We spent months on that job, building an agent harness on top of commercetools. This is the talk I gave at TSAI Conf London 2026, written down: what broke, and what we changed. Three things mattered more than anything else: the tools, the context, and being able to see what the agent did.

The recording is in the Library.

There is no fixed toolset

commercetools is API-first commerce. Hundreds of endpoints across dozens of domains: products, carts, orders, prices, inventory, promotions. The surface is large and it grows with every feature, and each new endpoint is another potential tool.

That leads to the constraint everything else follows from. The agent never has a fixed, known set of tools at runtime, for three reasons:

1. Scale. Hundreds of MCP tools today, more next quarter.
2. Openness. Merchants bring their own agents and tools, and they plug into our loop at runtime.
3. Permissions. The agent acts as the calling user, so the usable set changes per person.

A model plus an API is not a product. The product is the loop between them: planning, calling tools, managing context, recovering from failures, and recording what happened. We build on Mastra and own that loop on top of it, with our own gateway and context layer. Most frameworks assume you know your tools up front. We never do.

Lesson 1: tools are the surface area

When an agent underperforms, the first instinct is to reach for the model: a bigger one, or a better prompt. In our case the tools mattered more.

What we tried. Expose the whole toolset to the agent.

What broke. Two things. Hundreds of tool definitions do not fit in a context window, and when they do fit, the extra options make the model’s choices worse.

Then a worse failure. Someone asked for total revenue last quarter. The agent found the orders tool, ran it, summed the results and returned a number with a chart. It had summed the first page only: twenty orders out of a few thousand that matched. Nothing errored. No warning. The number was wrong and looked exactly like a correct one.

What we changed. We started treating failure behaviour as a property of the tool, not an afterthought. A powerful tool that fails quietly is worse than a limited tool that fails loudly. Tools break in production for boring reasons: bad arguments, slow APIs, data in a shape nobody planned for. What matters is what the harness does next. Does it retry? Does the model get a usable error? Or does it get a plausible half-answer?

Recovery beats raw capability.

Skills: pick the domain first

Hundreds of tools need structure, and the first cut is by domain. A skill is a playbook for one domain: orders, promotions, pricing. The agent loads the relevant playbook, or several when a question spans domains, and searches for tools only inside them. It never starts from a flat list.

A gateway: four tools instead of a hundred

Inside a domain the agent discovers the tool it needs, then invokes it. It never carries the catalog.

// find the tool: semantic search over a pgvector index
mcp_search_tools({ query, scope?: "read" | "write" })
  → [{ id: "platform::orders_query", inputSchema, score }]

// run it: the result is stored, the model gets a handle back
mcp_invoke_tool({ id: "platform::orders_query", args })
  → { storedResultRef, rowCount, columns }

Two details matter here. mcp_invoke_tool returns a handle rather than the data, which is the subject of the next lesson. And a tool id has to come from a search, so the agent cannot invent one. That removed a whole class of hallucinated tool calls. Only the meta-tools are preloaded, so adding tools or skills does not grow the context.

Read and write: safety in the tool layer

// every tool carries a standard MCP annotation → one scope
readOnlyHint     → "read"
destructiveHint  → "write"

mcp_invoke_tool runs read-scope tools only. Writes go through mcp_validate_write first, an offline dry run that checks the arguments against the schema with no network call and no change to data. The model does not decide whether a call is safe. The harness does, before the call leaves the process.

Lesson 2: context is the product

An agent’s answer quality depends mostly on what sits in the context window at the moment it decides. Not the system prompt, and not the model’s reasoning score on some benchmark. Managing that window turned out to be the core of the work.

What we tried. Give the agent everything up front, and put API results straight into the window so it has what it needs.

What broke. The window filled with payload instead of reasoning. Latency and cost went up, answers got worse. More context did not mean more intelligence.

What we changed. Two rules.

Query instead of stuffing. Tool results go into DuckDB, and the agent reaches them with sql_query when it needs them. The window stays about the task.

Accept more steps. The agent does not answer in one shot. It searches for a tool, invokes it, then queries the result. That is slower, and we would make the trade again, because each step is anchored in a real schema and real rows instead of a guess at an API shape. The cost is latency, and making the loop faster is the next piece of work.

One question, end to end

“What was our total order value per store in Q1 2026?”

Step 1. Load the domain playbook.

search_skills() → load_skill("orders")

The agent identifies the domain first, so everything after this is scoped to orders instead of the full catalog of tools.

Step 2. Find the tool.

mcp_search_tools({ query: "order revenue by store", scope: "read" })

// → { id: "platform::orders_query", inputSchema, score: 0.74 }

Semantic search returns a tool id and its real input schema. scope: "read" keeps write tools out of the result, and the id can only come from here.

Step 3. Run the tool.

mcp_invoke_tool({
  id: "platform::orders_query",            // GET /{projectKey}/orders
  args: {
    where:
      'createdAt >= "2026-01-01T00:00:00.000Z" ' +
      'and createdAt < "2026-04-01T00:00:00.000Z"',
  },
  fetchAll: true,                        // follow pagination
})

// → 3,412 orders, written to DuckDB
// the model gets back the table schema and a short preview, not the rows

This is the step that would normally flood the context window. The orders go to DuckDB instead, and the model receives a handle.

Step 4. Aggregate in SQL.

SELECT store.key AS store,
       SUM(totalPrice.centAmount) AS revenue
FROM orders
GROUP BY store.key;

-- → 5 rows, one per store

The aggregation runs in DuckDB, so only these five rows enter the context.

Step 5. Answer the question, using those five rows.

Four meta-tools, 3,412 orders fetched, 5 rows in context. The raw orders never reach the window.

Lesson 3: you cannot debug what you cannot see

Same prompt, same input, a different path. Anyone who has shipped an agent knows the feeling, and it is not a bug you can remove.

What we tried. Trusting runs that looked fine.

What broke. Silent failures. A run went wrong and we were left guessing what the model had been working with five steps earlier.

What we changed. We trace every step: each tool call, the context at each turn, and each decision with its reason.

Both earlier lessons depend on this one. The latency from the extra steps was invisible until we could read a trace. So were the bad tool calls. Tracing is what turned “the agent feels flaky” into a specific step with a specific fix.

Tracing was not a late addition to the project. It is what made the rest of the work possible.

The four tools the agent sees

Tool	Purpose
mcp_search_tools	Semantic search over the tool index. Returns ids and schemas.
mcp_invoke_tool	Runs a read tool. Result lands in DuckDB, model gets a handle.
mcp_validate_write	Offline dry run of write arguments before any mutation.
sql_query	Read-only SQL over DuckDB. Pulls back only what is needed.

That is the whole surface the agent works with.

What it got us

“What was our revenue per store last quarter?” used to be a BI ticket: file a request, wait for someone to write the query, get an answer the next day. Now a merchant asks and gets the numbers and a chart in seconds, using the trace above.

Reads are not the whole story. “Which carts hold items from the Shoes category?” crosses orders and the product catalog without extra code. “Activate the summer sale promotion” is a real write, validated before it touches live data.

I do not have clean benchmark numbers yet. What we plan to measure: step count and latency per task, tool-call success and recovery rate, and task quality.

The one thing

A model plus an API is not a product. The loop is.

Recovery beats capability in the tool layer. The context window is the most expensive resource you have. Neither of those is fixable if you cannot see what the loop did.