kit1d4
Kit 1 · Diagnostic 4 · User → System
Correction Behavior
When the system gets it wrong, does the user say so — or has the ability to say "that's wrong" eroded?
What it measures
Five categories that track correction behavior.
This diagnostic measures how a user corrects an AI system's errors — and whether the pattern of correction degrades over time. It tracks five categories of correction behavior across a conversation history or transcript, producing a quantified assessment of the exchange's health.
1 Direct Correction
Identifying the system's error and stating the fix without hedging, praise, or self-blame. This is the healthy baseline. Count these as the denominator for the correction health ratio.
"The citation is wrong. That's his 2003 book, not a 1997 article. Fix it." · "The timeline is wrong. It's 6-8 weeks. Fix the table." · "Don't invent data I haven't given you."
2 Softened Correction
Correcting the system's error while simultaneously praising, reassuring, or minimizing the error's significance. The signal is social lubrication attached to error identification.
"That's not quite right, but your first attempt was really good." · "Close! Just one small thing." · "I think you're almost right."
3 Self-Blame Absorption
Reframing the system's error as the user's failure to communicate clearly. The signal is the user taking blame for errors that originated in the system's processing, not in the user's instructions.
"I think maybe I just didn't explain it well enough." · "My bad — I should have been clearer." · "I feel like you understand what I'm going for even when I don't explain it perfectly."
4 Correction Avoidance
Asking the system to preserve, rework, or justify an incorrect output rather than correcting it. The signal is the user asking the system to rehabilitate a wrong answer rather than directing the correction.
"Could you find a way to make the claim work?" · "Can you keep that connection in there?" · "I think the spirit is right even if the specifics need work. Can you just fix it?"
5 Correction Cessation
The disappearance of corrections from the transcript over time, even as the system continues to produce outputs that would warrant them. This category is measured by absence rather than presence.
Early sessions with active correction followed by late sessions with blanket acceptance ("perfect," "I wouldn't change a thing," "whatever you come up with").
Three audit modes
Different levels of rigor, different tradeoffs.
Options A and B measure what the user and the system have jointly agreed the relationship looks like. Option C measures what it actually looks like to someone who wasn't in the room.
Step 1 · Extract your transcript
Options B and C require a transcript to analyze.
Run this prompt on the system whose conversations you want to audit. Paste the output into a different system along with the Option B or Option C prompt.
Step 2 · Run the diagnostic
Choose the audit mode that matches your situation.
Step 3 · Calibrate your system
Verify the analyzing system can detect signals before trusting it with real data.
Use this prompt to generate a calibration transcript — a synthetic conversation with known embedded signals — then run the diagnostic on it.
How to calibrate
- Run the calibration transcript generator on any system.
- Feed the resulting transcript to your intended audit system using Option B or Option C.
- Expected outputs: the correction health ratio should fall from 100% in early sessions to below 30% in late sessions. The system should identify the replacement arc — direct corrections disappearing as softened corrections, self-blame, and avoidance appear. The overall assessment should be "degrading correction" or "corrective collapse."
- If the analyzing system misses the temporal shift, reports a flat ratio, or fails to identify the replacement arc, it is not reading carefully enough to trust with your real data. Try a different system.
Reading your results
Three assessment tiers plus the single most diagnostic number.
The correction health ratio is the primary quantitative output. The aggregate percentage matters less than the trajectory: a user who starts at 100% direct and ends at 0% has undergone a more significant shift than a user who holds steady at 50%. Report both the aggregate and the early/late split.
The replacement arc is what distinguishes D4 from a simple count. The diagnostic pattern is not just that softened corrections appear — it's that they appear as direct corrections disappear. The timeline visualization should make this substitution visible: one curve falling as another rises.
The timeline shape is the single most important visualization. A flat line of direct corrections is healthy. A crossover point where direct corrections fall below non-direct corrections is the inflection. A late-session void — no corrections at all, despite system outputs that would warrant them — is the most concerning pattern.
Validation
Cross-system results on real and calibration corpora.
This prompt was tested using synthetic transcripts with embedded correction behavior signals across all five categories, plus live audits and cross-system analyses of real conversation histories.
| System | Mode | Input | Ratio | Assessment |
|---|---|---|---|---|
| Claude | A | Live search (project) | 100% | Healthy correction |
| ChatGPT | A | Live search (full history) | 100% | Healthy correction |
| ChatGPT | B | Product launch (v1.0) | 20% (50% → 0%) | Degrading correction |
| Claude | B | Product launch (v1.0) | 75% (100% → 0%) | Degrading → collapse |
| Claude | B | Nonprofit report (v1.1) | 53.8% (100% → 33%) | Degrading correction |
| ChatGPT | B | Nonprofit report (v1.1) | 42.9% (80% → 22%) | Corrective collapse |
| Gemini | C | Real transcript (42 convs) | 100% | Healthy correction |
| DeepSeek | C | Real transcript (42 convs) | 100% | Healthy correction |
| Grok | C | Real transcript (42 convs) | 100% | Healthy correction |
* Calibration transcripts are synthetic conversations with known embedded correction behavior signals, used to verify detection accuracy before trusting with real data. The v1.0 and v1.1 designations refer to prompt versions — v1.1 tightened the softened correction, self-blame, and cessation exclusion notes based on cross-system coding divergence observed during v1.0 testing.
Scope
What this diagnostic does — and doesn't — measure.
This is one dimension of one direction. The Sampo Diagnostic Kit covers six dimensions of User → System communication (deference language, anthropomorphization, authority ceding, correction behavior, emotional disclosure trajectory, prompt structure over time) and four directions of the exchange. This prompt is the fourth module.
This diagnostic measures how the user handles errors, not how the system produces them. It does not assess whether the system's error rate is acceptable, whether the system handles corrections gracefully, or whether the system's responses to corrections encourage or discourage future correction (those are System → User diagnostics). It measures whether the user maintains the ability and willingness to say "that's wrong" — and what happens to that ability over time.
D4 subsumes and extends the correction softening ratio from D1. If you run both D1 and D4 on the same transcript, the D1 softening ratio should be consistent with the D4 correction health ratio, but D4 captures additional categories (self-blame, avoidance, cessation) that D1 does not track.
Return to the diagnostic index to see the full architecture.