What real agents found in our MCP server

One question. 437,000 input tokens.

Not a hard question, and not a big collection. An agent connected to our MCP server, asked something a support engineer answers in one sentence, and got there through twenty tool calls, each one dragging every previous answer along behind it.

Nothing was broken while this happened. The server answered initialize correctly, spoke the 2025-03-26 revision, returned well formed JSON-RPC to everything we threw at it. Every conformance check we had, passed. That is the part worth sitting with: the protocol was right and the thing was still unusable.

Conformance tells you the protocol is right. It says nothing about whether a language model can find your data and pay a sane price for it. Those are different questions, and only one of them ships with a test suite you can download.

So we pointed real agents at production and watched. 18 scenarios, two vendors, real money. Most of what follows is now how the product works.

What the server is

Briefly, because it shapes everything below.

FoxNose stores content as collections: schema-defined records with typed fields, some of them vector indexed. The MCP server is generated from that schema and served from the same URL prefix as the REST API. The catalog is a fixed seven tools no matter how many collections you have, five read and two optional write.

Two things from that list matter for the rest of this post. Collections are what an agent chooses between, so a collection that describes itself badly is invisible. And the agent inherits exactly the rights of the API key it connects with, so there is no second list of allowed tools drifting out of sync with the first.

The harness

A scenario is a question in plain English, a set of tools, and a check.

The checks were the part we got wrong the most, so start there. They do not look at the answer text. Model output moves between runs, and a suite that asserts on wording is a suite you stop trusting by Thursday. They look at the trace instead: which tools ran, in what order, with what arguments, what came back, how many tokens the whole thing burned.

We ran all of it twice, on Anthropic’s hosted MCP connector and on OpenAI’s hosted MCP in the Responses API. Same server, same questions, two clients. That last decision paid for itself, and I will come back to why.

The bill is round trips, not bytes

Back to those 437,000 tokens.

A hosted connector runs the tool loop on the vendor’s side. Every iteration, it re-sends the whole conversation to the model. Everything before, again. So a tool call does not cost you what it returned. It costs what it returned, times every turn that comes after it.

The same question, same data, same model, answered with one narrow search and one full fetch: 78,000 tokens.

Twenty calls against fourteen is not twenty over fourteen. Each extra call drags the whole history with it, so the curve bends rather than slopes. Everyone tunes model choice. Almost nobody counts round trips, and round trips are where the money went.

What the product does now. Two things.

Search results come back shortened. Every text field is capped at 1000 characters by default, and each field that actually got cut carries a marker with its path, its locale, and the original length. So the agent always knows, per field, whether it is holding a fragment. An 11,917 character body arrives as 1000 characters and says so out loud. get_record hands over all 11,917 when the agent decides it wants them.

The pairing is the whole idea. Search exists so the agent can pick a record. Fetch exists so it can read one. When search returns everything, the agent pays full price for nine documents it will never open, and then pays again for them on every turn that follows.

Page size defaults to 5 now, with a maximum of 100, and both numbers live in the tool’s JSON schema. A model reading the catalog knows the range before its first call. Counting something, it asks for 100. Hunting one document, it leaves the default alone. We stopped guessing on its behalf.

There is a shape here that showed up twice on this project. When a setting is hard to set, the setting is probably asking for the wrong quantity. We could not pick a good page size for everybody. We could tell the caller the bounds and get out of the way.

Details are in Retrieval and Cost.

Your error strings are control flow

An agent hit an error, read the hint that came with it, did exactly what the hint said, failed the same way, read the same hint, and did it again. It did not crash. It did not report anything wrong. It spent its entire iteration budget being helpfully obedient to a wrong instruction.

The cause was one line out of place. We return structured errors, an error code plus a message plus a hint saying what to do next, and one of those codes had been handed a different error’s hint. The code was right. The status was right. The sentence belonged to another failure.

For a person, that is cosmetic. You read the wrong sentence, you shrug, you try something else. A model does not shrug. The hint is not documentation to it. The hint is the next instruction.

What the product does now. Every hint points at its own failure. The payoff turned out to be bigger than one corrected string: once hints are trustworthy, guessing gets cheap. An agent invents a collection identifier, gets told the exact recovery step, and is back on track one call later. That is worth designing for on purpose.

The client decides your authentication

We had two auth schemes. Both good. Both required you to write your own Authorization header, which was never a problem, because every client we developed against was one where we wrote the header ourselves.

Hosted connectors do not work that way. You get one token field, and it goes out as Authorization: Bearer <token>. That is the whole interface. No configuration, no workaround, no documentation that fixes it.

So our server could not be used from the two clients most people reach for first. Not broken. Unusable. And invisible from the inside, which is the part I keep thinking about.

What the product does now. FoxNose issues bearer tokens: opaque, 47 characters, no colon in them so they can never be mistaken for a key pair, shown once at issue. A token is bound to an existing API key and is exactly as powerful as that key. Nothing added, nothing taken away.

Revocation is the part I did not expect to care about. Deleting a key, or emptying its role, changes a permission snapshot that sits in a cache for about five minutes. We measured it: 305 seconds one way, 284 the other. Revoking a bearer token skips the cache entirely, because the token stops resolving, so the next request fails and that is that.

Five minutes is a long time when you are cutting off a connector you should not have connected.

Details are in Authentication.

Discovery is one sentence

We asked an agent about FoxNose API keys. It went and searched a Kubernetes documentation collection instead.

Both collections were connected to the same API. Both descriptions read perfectly well to anyone who already knew which was which. The agent did not know and could not: it cannot see your dashboard, your folder tree, or the naming convention that makes the difference obvious to you. It sees a list of identifiers and one sentence each, and it picks.

What the product does now. Descriptions are surfaced properly in discovery, and there is a per connection override, so one collection can introduce itself differently to an internal agent than to a customer facing one.

If you are writing one of these, describe what is inside rather than what the thing is for. “Support articles about billing, refunds and invoices” works. “Support knowledge base” does not, because it fails to tell your four collections apart. More in Preparing a Collection for Agents.

Two vendors, two shapes

We wrote the OpenAI transport after the Anthropic one, and assumed it would be the same code with the nouns changed. It would have been wrong in three places.

Tool call arguments arrive parsed in one and as a JSON string in the other. The OpenAI SDK types the error field as an optional string, while the wire sends a dict. Neither of those is hard once you know.

The third one is worth a warning, because it will bite anyone building on the Anthropic connector. It runs at most 10 tool iterations per request. If the agent has not finished by then, you get back stop_reason: "pause_turn" and the work so far. This is not an error. The request succeeded. A client that checks for end_turn and treats anything else as done will log a success and show a truncated answer, with no exception anywhere to tell you. You continue by sending the response back as an assistant message.

Two things I learned about testing this

A check that forbids all errors forbids recovery. Our check called no_tool_errors failed three scenarios that were completely fine. In each one the agent guessed an identifier, got a hint, corrected itself, and answered the question. That is the system doing exactly what we built it to do. The check called it a defect.

We had spent real effort making guessing cheap, and then written a test that punished guessing. If your surface is meant to be explored, your tests have to let it be explored.

A negative result is not a result. We checked that a key from one environment cannot read another environment. It could not, and I nearly wrote that down as proof of isolation.

Then we checked whether that key could read anything at all. “Access denied” looks the same whether the isolation worked or the credential was simply dead. This caught us three separate times. Every security test needs a positive control beside it, a case you know should succeed. Without one, all you have proven is that broken things stay broken.

Where this leaves us

Everything above is live. Truncation defaults, page size bounds, hints, bearer tokens, all documented in the MCP section of our docs. The final run is 18 out of 18 on both vendors, against production.

If you are shipping an MCP server of your own, the cheapest useful thing you can do this week is take five real questions, run them through a hosted connector, and read the traces. Not the answers. The traces. Count the calls, look at what each one handed back, then ask whether a model that cannot see your dashboard could have done better than that.

And if you connect FoxNose to an agent and it does something expensive or strange, tell me about it. That is how the list above got written.