Debugging With an AI That Sounds Certain

This one is a mistake rather than a technique. It is more useful than the techniques.

The symptom

Production was broken. The preview deployment of the same build was fine.

That is a good symptom, because it narrows things fast. Same code, same build, different behaviour — so the difference is environmental, not in the source.

The investigation

First step was comparing the two directly rather than theorising:

curl -sS https://kociaj.com/ > prod.html
curl -sS https://<hash>.pages.dev/ > prev.html
diff prod.html prev.html

Both returned 200 with near-identical byte counts, so the HTML was being served. The difference showed up in the script tags:

<!-- production -->
<script type="57dcbe46da9a5ddda93858c2-module" src="/_nuxt/DOSwq-1a.js">

<!-- preview -->
<script type="module" src="/_nuxt/DOSwq-1a.js">

Production had type="module" rewritten to a nonsense MIME type. There was also an injected script from /cdn-cgi/scripts/.../rocket-loader.min.js.

That is Cloudflare Rocket Loader. It rewrites script types so it can control execution order, and it is well known for breaking ES modules.

The story fit perfectly. Rocket Loader is a zone-level setting on the custom domain and does not apply to *.pages.dev — which explains exactly why preview worked and production did not. Server-rendered HTML with no hydration is precisely what "broken" looked like.

I wrote it up with the dashboard path and an API call to disable it.

It was the cache

The actual fix was a cache purge.

Rocket Loader was a real thing that was really happening — but its shim does restore type="module" and re-inject the scripts at runtime. It was noise. The site was serving stale cached HTML from a previous deploy, and purging fixed it.

I found this out because after the purge I re-checked, and Rocket Loader was still rewriting script tags on a working site:

curl -sS https://kociaj.com/ | grep -o 'type="[^"]*module"'
# type="7cee160ad24d3eb8b9a8f928-module"

Same rewriting. Site fine. The diagnosis was disproved by the evidence that was always available.

Why it was convincing

Worth being precise about this, because "AI hallucinated" is the wrong description and the wrong lesson.

Nothing was fabricated. Rocket Loader was genuinely enabled. It genuinely rewrites type="module". It genuinely is a known cause of hydration failures. It genuinely does not apply to pages.dev. Every fact was true and independently verifiable.

The error was causation from correlation. A real anomaly was found that could explain the symptom, and "could explain" got promoted to "does explain" without the step that separates them: checking whether the anomaly is present when things work.

That step is cheap. It was one curl. It just was not taken, because the story was already coherent — and coherence feels like correctness.

This is a human failure mode too. It is worse with an assistant only because the writeup arrives fast, well-organised, and with a confident tone that a colleague thinking aloud would not have.

What actually helps

Four things, in rough order of value.

Ask what would disprove it. The strongest habit. Before acting on a diagnosis: what would I expect to see if this were wrong? Here — the anomaly present on a working page. One command, would have caught it immediately.

Distinguish anomalies from causes. Any sufficiently complex production system has several odd things going on. Finding one during an incident does not make it the cause. Cloudflare zones in particular have a pile of transforms enabled that mangle HTML and mostly do not matter.

Prefer differences over anomalies. The initial diff was the right instinct — comparing working against broken is far more targeted than inspecting broken alone. The mistake came from abandoning that discipline once something interesting turned up, and stopping the diff there.

Check the boring causes first. Cache, DNS propagation, a stale build, an environment variable. The Rocket Loader theory was more interesting than a stale cache, and interesting is a bias, not evidence. Sequence by prior probability, not by narrative appeal.

What it does not mean

Not "do not use AI for debugging".

The investigation was still faster than doing it alone. Fetching both pages, diffing them, spotting the script-type difference, and knowing what Rocket Loader is and how it interacts with ES modules — that is maybe fifteen minutes of work compressed into a couple of minutes, and the Rocket Loader knowledge is not something everyone has to hand.

The failure was not in gathering evidence. It was in the last step, where evidence became conclusion without a check.

So keep the tool for the part it is good at — gathering and correlating fast — and keep the verification for yourself. The curl that disproved the theory took two seconds. It just needed someone to ask for it.

The honest bit

I published that diagnosis before the fix was confirmed. It read as authoritative — dashboard path, API call, an aside about Email Obfuscation — and it was wrong about the only thing that mattered.

Which is the actual lesson. Confidence in the writeup is not evidence about the world. It is a property of the writing.

Next: permissions, and deciding what should never be automatic.