Google (Gemma)
Carwash Test transcripts · open-weight
My car is dirty. The carwash is 100 feet away. Should I walk or drive?
Gemma 4 is an open-weight model line, not the commercial Gemini product. Most of these runs were taken in Google AI Studio's playground (Thinking Level: High, Temperature 1, Grounding off); the 12B was run locally on consumer hardware via LM Studio. Either way it is the model as a raw artifact — no consumer system prompt, no RLHF-tuned product layer, and no wrapper doing cleanup. Gemma is not reachable through the Gemini consumer app; it lives in AI Studio, on Hugging Face, and in self-hosted deployments. These results are relevant to developers, self-hosters, and small-office deployers, and they are not evidence for what the Gemini consumer product will do, or vice versa. They are kept as a separate deployment class and excluded from the commercial corpora.
Cross-language matrix
Two Gemma 4 models across four languages. No clean main effect for model size or language: each model fails in exactly one language, but different ones. The dense 31B fails in Ukrainian where the 4B-active MoE passes; the MoE fails in English — its best-resourced language — where the 31B passes. The signature inverts the commercial pattern (which tends to hold in English and break elsewhere).
| Model | Params (active) | English | French | Chinese | Ukrainian |
|---|---|---|---|---|---|
| Gemma 4 26B A4B IT | 26B total / 4B active (MoE) | Fail | Pass | Verbose | Pass-adjacent |
| Gemma 4 31B IT | 31B dense | Pass | Pass | Pass | Fail |
The 26B MoE's Chinese reasoning trace, visible in AI Studio, ran in English throughout and even issued itself a "Keep it brief and witty" self-correction — which failed to prevent verbose Chinese output. The English-space self-monitoring did not govern the language-specific output. The 31B dense model produced a concise Chinese answer with no such self-correction.
Gemma 4 12B (Q6_K) — the new encoder-free multimodal model (released June 3, built to run on a laptop) — was tested separately on consumer hardware: a desktop RTX 5060 Ti (16 GB) running it locally in LM Studio (Q6_K GGUF, ~10 GB) at ~29 tokens/sec, rather than in AI Studio. It failed every run — all four languages (English, French, Chinese, Ukrainian) in both thinking states, eight for eight. Each answer recommends Walk, losing the logical object (the car has to reach the wash regardless), and several invent an inverted-logic flourish — driving the dirty car the 35 m would "fling dirt" onto it. Enabling thinking did not help: it produced longer, more confident wrong answers, with the French and Ukrainian thinking-on runs spending 2½ minutes reasoning their way to the same Walk verdict ("95% of the time, walk"). The same surface-feature capture (distance, fuel, engine wear) drives every miss. The English runs appear below; French, Chinese, and Ukrainian are in their language sections.
Quantized local builds (QAT, Q4_0). Quantization-aware-trained 4-bit builds were run locally across three sizes, and the thinking toggle's effect depends on architecture. The dense Gemma 4 31B QAT passes only with thinking on — its trace reasons explicitly to the constraint ("you can't wash the car if you are just standing at the car wash without the car. Therefore, you must drive") — and fails with thinking off. The 4B-active MoE Gemma 4 26B A4B QAT inverts it: only thinking off reaches Drive, while thinking on talks itself into Walk (its trace entertains the correct logic, then abandons it). The smallest Gemma 4 12B QAT fails in both states — its thinking-on trace concludes "Direct Answer: Walk" outright — as the Q6_K 12B did across every language. So enabling reasoning rescues the dense model, sinks the MoE, and leaves the 12B unmoved. All six runs are in the English results below.