Draft — Andy to refine the specifics marked [expand]. The headline facts and figures below are from your performance record; the colour and quotes are yours to add.
The problem
When reviewers disagree about how serious a translation error is, every downstream number wobbles. Quality scores stop meaning the same thing from one reviewer to the next, and you can’t manage what you can’t measure consistently. On the machine-translation quality work, agreement on the most severe category — critical errors — sat at just 20%. Four reviewers in five were calling it differently.
How I framed it
I treated this as a behaviour problem, not a knowledge problem. People had read the guidelines; they just applied them differently under real conditions. So rather than re-teach the rulebook, I designed for calibration — getting reviewers to make the same call on the same evidence. [expand: the specific MQM severity dimensions you focused on]
What I tried
- [expand: the format — short worked examples? annotated cases? a decision aid?]
- I built the programme to be small and fast, something reviewers could complete inside their working week, not a course that competed with the job.
- I measured alignment before and after on the same material, so the effect was the training, not chance.
The result
Critical-error alignment rose from 20% to 76%, with overall alignment up 11.4% — and the whole thing was designed and delivered in a single week. [expand: what this unlocked downstream — cleaner trend data, fairer reviews, etc.]
What you can take away
- When experts “already know” the content but act inconsistently, the lever is calibration, not more teaching.
- Measure the same thing before and after on identical material; it’s the cheapest way to prove the learning, not the luck.
- Small and fast beats comprehensive and ignored. A module that fits the working week gets finished.