Our MCP server was fine. Cloudflare was returning HTML.

This is what our MCP endpoint returned to a client speaking JSON-RPC:

HTTP/2 403
content-type: text/html; charset=UTF-8
server: cloudflare

Attention Required! | Cloudflare
Please enable cookies.
Sorry, you have been blocked

A program was being asked to enable cookies.

We did not see that for three days, because that is not what the client reported. The client reported a parse failure, and a parse failure points at your own serialization, not at a machine four thousand kilometres away.

What it looked like from inside

The connector could not talk to our server. The tools never listed. Everything we could check on our side looked correct: valid JSON-RPC, right content type, right status codes, protocol revision we support.

So we did the reasonable thing and started fixing the things that were wrong but adjacent.

GET /{prefix}/_mcp was returning JSON. Per the Streamable HTTP spec it should return 405 if the server does not offer an SSE stream on GET. We changed it. It is a real fix and we kept it.

It changed nothing.

That is the value of shipping one variable at a time. If we had bundled that change with anything else, the next result would have been unreadable.

The thing that actually settled it

We wrote a minimal MCP server that imitated our own response shape exactly: application/json on POST, 405 on GET. A hundred lines, no auth, no database, nothing of ours in it. Then we exposed it through an ngrok tunnel and pointed the same connector at it.

It worked immediately. Tools listed, first try.

That one result eliminated most of the search space. Our response shape was fine, because a server with the same shape worked. The protocol was fine. The connector was fine. What was left was everything between the connector and our application, which is exactly the part we had not been looking at, because it is not in the repository.

If you take one thing from this: when you cannot find the bug in your code, build something that cannot possibly contain it and see whether the problem follows. The mimic server took an hour. It saved the rest of the week.

It was the user agent

Here is the same request, same body, same endpoint, varying only User-Agent:

user-agent                          code  content-type              bytes
Claude-User/1.0; +claude.ai          200  application/json            156
Claude-SearchBot/1.0; +claude.ai     200  application/json            156
anthropic-ai                         200  application/json            156
curl/8.7.1                           200  application/json            156
ClaudeBot/1.0; +claude.ai/bot        403  text/html; charset=UTF-8   4543
GPTBot/1.2; +openai.com/gptbot       403  text/html; charset=UTF-8   4543

A Cloudflare managed rule, on by default, blocking AI crawlers. Nobody on our side had turned it on. Nobody had turned it off either, which is the point.

Look at the fourth row. curl passes. Every tool you reach for when something is broken is on the allowlist, which is why this survives so long. You test the endpoint, it answers, you conclude the endpoint is fine, and you go back to reading your own code.

One limitation worth stating: every row above was sent by curl, so the TLS fingerprint was held constant while only the header changed. That proves the user agent alone is enough to trigger the block. It does not prove the user agent is the only signal, and bot detection also reads the ClientHello. If you allow an agent by name and it still gets refused, that is the next variable to vary.

The taxonomy is where it goes wrong

Anthropic runs three crawlers on purpose, so that a site owner can make three separate decisions.

ClaudeBot collects content that may contribute to training. Claude-User fetches a page because a person just asked Claude a question. Claude-SearchBot indexes for search results. Three names, three robots.txt entries, three different trade-offs. The split exists precisely so you can refuse training and stay reachable.

Now look at how they are categorised under AI Crawl Control, in the Security section:

Claude-User        Anthropic     AI Crawler
ClaudeBot          Anthropic     AI Crawler
Claude-SearchBot   Anthropic     AI Search
GPTBot             OpenAI        AI Crawler
Anchor Browser     Anchor        AI Crawler

Claude-User and ClaudeBot land in the same bucket. Claude-SearchBot gets its own. So the finer categories do exist, and the one agent in that list that is not a crawler at all is filed as a crawler.

It is not only Anthropic. Anchor Browser is an agentic browser driven by a person, and it is an AI Crawler too.

The consequence is that you cannot express “refuse training, serve agents” at the category level. The vendor separated the agents so you could choose. The category puts two of them back together. You have to allow the individual ones by name, and first you have to know that you need to.

Our configuration now refuses the training crawlers and passes the user-initiated traffic. That was a change we made on purpose, not something the default did for us.

Nothing in the response says you were blocked

This is what makes it expensive rather than annoying.

There is no JSON error. No error code. No header explaining the refusal. cf-mitigated is absent. All you get that a machine can read is server: cloudflare and a cf-ray id, and neither of those means anything to a JSON-RPC client that expected an object and got a document.

So the client raises a parse error, and a parse error is a lie about where the problem is. It points inward, at your serializer, your framework, your content type. Every hypothesis it suggests is about code you own.

The part almost nobody uses: this is configurable. The same screen has a Configure Response control that sets the status code and message returned to blocked crawlers. If the thing behind your CDN is an API, a JSON body with an explicit reason costs nothing and turns three days of debugging into one line in a log. Blocking somebody is fine. Blocking them in a format they cannot parse is a choice you probably did not mean to make.

The same thing decides whether agents can discover you

Agent cards, OAuth protected-resource metadata, MCP discovery documents: all of it is converging on /.well-known/. That only works if the thing fetching it is allowed to fetch.

Ours is reachable, and here is how you tell:

/.well-known/oauth-protected-resource     404  application/json
/definitely-not-a-route                   404  application/json

body: {"message":"Route not found","error_code":"route_not_found",...}

The 404 is ours. Same JSON envelope as any unknown route, which means the request reached the application. A block page instead of your own error format means it did not. Check whose 404 it is, not whether you got one.

The five minute version

Take your own endpoint and run the request you care about six times, changing only the user agent:

for ua in "curl/8.7.1" "Claude-User/1.0" "ClaudeBot/1.0" "GPTBot/1.2"; do
  curl -s -o /dev/null -w "$ua %{http_code} %{content_type}\n" -A "$ua" \
    -X POST https://your.endpoint/_mcp -H 'Content-Type: application/json' \
    -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
done

If the codes differ, or the content types do, the layer in front of you is making decisions you did not make. Watch the content type especially: a proxy can answer 200 with an HTML body, and no status check will ever catch that one. In Cloudflare they live under AI Crawl Control, in the Security section, one row per crawler. Then decide which of those decisions you actually want, because refusing training crawlers and refusing your customers’ agents are not the same choice, and by default they are the same switch.