Metrics

These charts read directly from the live dataset and update as runs are added. A standing caveat applies throughout: the test is single-shot, the sample per cell is small, and each snapshot reflects whatever models were available that date — so the time-based charts describe the evolving field, not a controlled trend in any one system. See the methodology for scoring and limitations.

Dataset overview

Over time

A note on how to read these. The first three charts are cumulative — each point covers every run recorded up to that date. That makes them stable measures of what the whole corpus says, but it also means a small recent batch barely moves them: by July the dataset held roughly two hundred runs, so adding a dozen cannot shift a median, and a cumulative count can only rise or flatten, never fall. The last chart is per-date, showing only the runs taken that day, and is where a recent sweep or collapse actually shows up.

By configuration

By model family

Cross-language comparison

The Carwash Test has been run in four languages, each kept as a separate corpus (the charts above are the English corpus, n=213). This table places the models tested in more than one language side by side. The English column follows the live dataset. A dash means the model was not run in that language. (Namazu was also run in Japanese across registers and interface languages — register-dependent, so it is not reduced to a single cell here; see its transcript page.)

• Fable 5 is the first model with a clean four-language pass record. Before June 9, every model tested in more than one language failed in at least one: Chinese defeated both Opus generations and every GPT; Ukrainian broke Sonnet 4.6's thinking-off states. Whether the clean sweep reflects Mythos-class capability, training-data composition, or the trace-language match documented below is undetermined: one model, one snapshot. On June 22, GLM-5.2 (Z.ai) became the second model to clear all four languages — a clean hold in every Deep Think state — so the four-language sweep is no longer a sample of one, though it remains rare. Carwash III (July 11) doubled the club: ChatGPT-5.6 Sol went eight-for-eight across the four languages in both effort tiers — every answer a winner's-circle one-liner — and Qwen3.7-Max held every state in both modes; Fable 5 and GLM-5.2 re-verified their sweeps, and Opus 4.8 swept all six non-English states under the new toggle+effort UI. Kimi K3 (July 17) makes five: twelve Drives in twelve runs across four languages and both variants — six days after Kimi K2.6 went one-for-eight, the largest single-generation reversal in the dataset. Qwen3.8-Max (August 3) makes six, and it is the first to sweep on a mode selector rather than a toggle: twelve Drives across four languages in Fast, Thinking, and Auto alike. Its Fast mode holds everywhere its predecessors broke — Qwen3.7-Plus Fast failed English, Chinese, and Ukrainian; Qwen3.6-Plus failed in Fast and Auto both.
• Chinese defeats models that pass in English and French. Opus 4.7, GPT 5.5 On, and GPT 5.2 On all hold the constraint in English, hold (mostly) in French, and fail in Chinese. The language is an independent variable.
• The flagship fails Chinese where the mid-tier holds it. Both Opus generations (4.7 and 4.8) fail the Chinese prompt in every toggle state — recommending walking, or naming the constraint and then dismissing it — even though Opus 4.8 passes English and French cleanly. The smaller Claude Sonnet 4.6 holds the constraint in Chinese across all four states. More reasoning budget on the larger model does not help here; in this language it hurts.
• The inverted-logic failure was language-triggered — for four months. Models argue that driving would make the car dirtier in French (Mistral, Lumo, both GPTs), Chinese (GPT 5.5 Off), and Ukrainian (Sonnet 4.6 Off, both consumer Sonnet runs) — and in zero English runs from March 22 through July 8. Carwash III (July 11) broke the pattern: DeepSeek V4-Pro (Expert, thinking off) argued the post-wash drive home would re-dirty the car, and Lumo 2.0 Max (Fast) warned that driving would mean "tracking dirt right up to the entrance." The failure mode is still language-skewed, but no longer language-exclusive.
• The toggle's effect is language-dependent. GPT 5.2 On passes in English but fails in French; GPT 5.2 Off fails in English but passes in French. The same inversion repeats in Ukrainian — GPT 5.2 Off (Instant) holds the constraint while Thinking-On fails — as it does for Vibe (Balanced passes, Think fails). The same reasoning mechanism helps in one language and hurts in another.
• Ukrainian is the most forgiving corpus, and the easy answers cluster there. With the column now filled, several models that fail elsewhere hold the constraint in Ukrainian: Lumo and GPT 5.5 Off both recover, and Opus 4.7 passes cleanly in both toggle states. Ukrainian's 24% failure rate is the lowest of the four corpora.
• Reasoning modes can refuse to answer. Vibe's Research mode produced no recommendation at all on the Ukrainian prompt — it returned four clarifying questions — the only configuration in the dataset that declines to hold the object by declining to answer.
• The test is now in the models' search results. On July 11, Grok 4.5's Expert mode ran a mid-answer web search, retrieved fifteen sources — one of them this site — and answered by naming the benchmark: "This is the classic 'car wash test' that trips up a lot of AIs." Correct, object-holding, and scored pass-adjacent: the pass is retrieval-informed, not reasoned cold, and the same model fails in both of its non-searching modes (Fast and Auto). With Perplexity's earlier search-grounded passes, this marks a methodological turn — the diagnostic is now discoverable by the systems it measures, and search-equipped modes must be read differently from closed-book ones.
• The first response-language mismatch — and a home-language flip. On July 11 Namazu answered the Chinese prompt entirely in Japanese — until now only reasoning traces mismatched the prompt language; here the answer itself does. The same session flipped its home-language result: the Standard-register Japanese run, which reached Drive on June 23, now recommends walking. Namazu finishes Carwash III zero-for-five (English, French, Ukrainian, Chinese, Japanese).
• DeepSeek V4-Pro fails only English — the mirror image of the historical pattern. With thinking off, V4-Pro produced the dataset's first English inverted-logic failure (July 11) while holding French, Chinese, and Ukrainian on the same day. Four months of data said models hold English and break elsewhere; this is the first model to break only in English. Kimi shows a related inversion: its sole hold anywhere is French-with-thinking, while it fails home-language Chinese in both states — with a Chinese inverted-logic flourish ("the wash would be wasted; the drive back splashes mud on the body").
• A Japanese-adapted model holds the object only in Japanese — and only in some registers. Sakana AI's Namazu reaches "drive" exclusively on Japanese prompts (Standard register in both interfaces; Polite only in the English interface) and recommends walking in every non-Japanese language — English, Chinese, French, and Ukrainian all fail. Two new variables move the verdict within Japanese: the register selector (Kansai-ben walks in both interfaces; Polite walks in Japanese but drives in English) and the interface language itself. It is the first model whose grip on the logical object appears tied to a cultural-adaptation target rather than to reasoning budget. Full breakdown on its transcript page.

Simplified Chinese corpus

Token calculation: DeepSeek documents that 1 Chinese character ≈ 0.6 tokens (vs ~0.25 per English character), so token estimates for Chinese runs use the Chinese rate. A 72-token Chinese response and a 72-token English response represent different amounts of visible text.

French-language corpus

Ukrainian-language corpus

Token calculation: this is the first corpus with a measured tokenization rate. API-console runs report real output-token counts (which include hidden reasoning tokens), and Cyrillic text runs at roughly 0.5 tokens per character — about double the English rate of ~0.25. Console runs are flagged distinctly because their token totals include reasoning the consumer interface hides. Consumer-app runs use the measured Cyrillic rate. None of these counts are directly comparable to the English character-based estimates.

Cross-corpus findings

Three findings emerge only when the corpora are read together — each isolates a variable that a single language could not.

Models reason in a dominant internal language, then translate

Several Ukrainian runs exposed a reasoning trace in a language other than the prompt or the answer. The model handled the logical constraint in its dominant internal language and translated only the final output into Ukrainian.

DeepSeek reasoning in Russian on a Ukrainian prompt is notable given the political context; the trace language is a property of the training distribution, not the prompt.

Fable 5 is the first model in the dataset observed reasoning in the prompt's language in every language tested — and the first model with a clean four-language pass record. Every previously tested model that exposed a trace reasoned in a dominant internal language (English, Russian, or Chinese) and translated outward, and every previously tested model failed in at least one language. The correlation supports the trace-language-match hypothesis: language-dependent failures may enter at the translation boundary between the model's internal working language and its output language. One model; correlation only; stated at that weight. Carwash III (July 11) collected trace language across every model that exposes one — and complicated the hypothesis: Qwen3.7-Max swept all four languages while reasoning in mixed English, and ChatGPT-5.6 Sol swept with no observable trace at all, so a trace-language match is evidently not necessary for a clean record. DeepSeek's Russian-on-Ukrainian pairing, first seen in V4-Flash, reappeared in V4-Pro. And Namazu extended the mismatch from reasoning to output, answering the Chinese prompt in Japanese. Kimi K3 (July 17) reasons in English on every non-English prompt at both tiers even while sweeping all four languages — which sharpens a standard this record now tracks: from the operator’s side, the trace is part of the product, and a reasoning trace the operator cannot read fails at its one job of making the reasoning inspectable, whatever the verdict. Trace language does not change a score; it is recorded and weighed here. Qwen3.8-Max (August 3) breaks the pattern in a new way: its French and Ukrainian traces do not pick a language and stay there — they switch between the prompt language and English mid-stream, sometimes several times in one trace, while its Chinese traces stay wholly in Chinese. A trace that changes language partway is readable to no one in particular.

Reasoning effort is a continuous variable, not a toggle

Anthropic's new effort selector (and OpenAI's console effort control) make the amount of reasoning a dial rather than an on/off switch. The same model can pass or fail depending on where the dial sits.

Inkling puts the whole dial in one model. Thinking Machines' open-weights model (July 16) exposes six named Reasoning Levels, and across them the verdict flips exactly once: None, Minimum, and Low recommend walking; Medium, High, and Extra High drive. Below the threshold the distance wins; above it the object does. Per the model card, those six names discretize a continuous effort parameter running from zero to one — the vendor's own benchmarks report effort=0.99 — so the real threshold sits at some value between Low and Medium, and the selector only samples it. This is an open-weight result, excluded from the commercial corpora, and it is the cleanest effort threshold on record.

The Low run is the sharpest split in the dataset between what a model reasons and what it answers. Its visible trace concludes that the car has to be driven regardless; the answer beneath it recommends walking, in text verbatim identical to the Minimum response. On this evidence the trace is not a window onto the deliberation that produced the answer. It is a second output. Qwen3.8-Max (August 3) supplies the mirror case: two of its traces argue their way to Walk in full, under their own headings, before reversing to Drive and answering correctly. The trace and the answer can diverge in either direction.

Effort is pure cost once the answer is correct

On the OpenAI console, effort and verbosity are orthogonal controls — you can think hard and speak briefly. GPT 5.5 produced the same correct Ukrainian answer at three effort settings; the extra effort bought nothing but hidden reasoning tokens.

Low and extra-high effort return the same correct answer at 121 vs. 565 output tokens (real console counts, not estimates) — a 4.7× cost difference for identical quality.

Two July results cut the other way, at least on the visible answer. Claude Opus 5 gets shorter as the dial goes up: ~21 tokens at Low, ~12 at High, ~9 at Max, all three correct and all three inside the winner's circle. Inkling compresses the same way once it is above its threshold — ~186 tokens at Medium, ~152 at High, ~138 at Extra High. Where a model is already holding the constraint, added effort can buy concision rather than padding.

The caveat matters. These are estimates of the visible answer, and consumer surfaces do not report reasoning tokens, so a shorter answer at higher effort is not a cheaper answer. The GPT 5.5 console runs above are the only place in this dataset where the full cost is measured — and there the extra effort bought nothing.

The budget tier is where the test bites, and where it moves fastest

Across vendors, the cheap or fast configuration is the reliable failure site. ChatGPT 5.5 at Instant effort fails English, French, and Chinese. Kimi K2.6 Instant fails all four languages. Qwen3.7-Plus Fast, Vibe Chat Fast, and Lumo 2.0 Lite Fast each fail three of four. Gemini 3.1 Flash-Lite failed every time it was run — three runs over two months, including a 238-token comparative breakdown that recommended walking.

Eleven days after that breakdown, Gemini 3.6 Flash-Lite passed both of its toggle states, and its Extended-Thinking-off answer is a ~17-token winner's-circle pass. Six days after K2.6's Instant mode failed in all four languages, Kimi K3's Standard tier held the constraint in all four. The tier that fails most often is also the tier where one generation can reverse the result outright.

This is not evidence that the frontier is improving. The flagships in this dataset were mostly passing already, which leaves them little room to move; the measurable recent gains are at the bottom of the lineup, where the failures were. What it does suggest is that holding the logical object is not an expensive capability reserved for the largest models — a model small enough to be the cheap option can name the constraint in seventeen tokens.

The winner's circle: concise correct answers

A “winner's-circle” pass names the constraint and stops. Because tokenization differs by script, the brevity threshold is script-specific: ≤30 tokens in Latin, ≤60 in Hanzi, ≤60 in Cyrillic. Every Pass that clears its threshold is listed below — runs #1/#39 and #2/#40 are the same verbatim answer produced in two separate test batches. Claude Fable 5 enters in three of its four languages (English, French, Chinese); its Ukrainian run passes but, at ~100 tokens by the measured Cyrillic rate, exceeds the 60-token threshold. Claude Sonnet 5 enters in four of its five effort modes (Medium, High, Extra, Max); the Low-effort run passes but, at ~46 tokens, exceeds the Latin threshold. One open-weight run also clears the bar and is included for completeness, flagged as such — Gemma 4 31B (Google AI Studio, Chinese, ~24 tokens) — the table's only non-commercial entrant. Carwash III (July 11) adds 31 qualifiers in one day — 28 of them from the Anthropic lineup re-baseline, plus both ChatGPT-5.6 Sol runs and a Copilot GPT 5.6 Think run — including the shortest pass on record: Claude Opus 4.6's six-token "Drive. It's a carwash." The same day's non-English sweep adds 20 more across all three scripts — six of them ChatGPT-5.6 Sol's, whose entire four-language record sits inside the thresholds, and the tersest of all Opus 4.8's ten-token Chinese "开车去。车不在洗车店里就洗不了。" Gemini 3.6 Flash-Lite (July 22) enters at ~17 tokens on its Extended-Thinking-off run — the same tier whose 3.1 predecessor produced the dataset’s longest Gemini failures eleven days earlier. Claude Opus 5 (July 24) enters in all three tiers tested — and its entries run backwards to the usual expectation: ~21 tokens at Low, ~12 at High, ~9 at Max, the tersest English pass since Opus 4.6’s six-token record. Qwen3.8-Max (August 3) enters once, in Chinese at ~59 tokens; its English Thinking answer passes at ~31 and misses the Latin threshold by a single token.

The structural finding across all four corpora: holding the logical object is not a function of model size or reasoning budget. The shortest correct answers come from models that name the constraint immediately; the longest wrong answers come from models that elaborate their way past it.