Model drift index
Did the model change under you? Every UTC day, each model answers the same 20 prompts. We publish the pass rate against deterministic goldens and how many answers changed since the day before. No LLM judge, no sampling parameters, no login.
31 days of data · suite drift-suite/v1 · updated 2026-10-22 15:31 UTC · JSON: /drift.json · badge: 
| Model | Pass rate (latest) | Δ vs previous day | Answers changed | Days | Last 30 days | Badge |
|---|---|---|---|---|---|---|
claude-fable-5anthropic | 100% | ±0 | 0/20 | 31 |  | |
claude-fable-5-1anthropic | 100% | ±0 | 0/20 | 31 |  | |
claude-haiku-5-5anthropic | 100% | ±0 | 0/20 | 15 |  | |
claude-opus-4-6anthropic | 100% | — | — | 1 | 1 day of data |  |
claude-opus-4-7anthropic | 100% | ±0 | 0/20 | 16 |  | |
claude-opus-4-8anthropic | 100% | ±0 | 0/20 | 31 |  | |
claude-opus-5anthropic | 95% | ±0 | 1/20 | 31 |  | |
claude-opus-5-5anthropic | 100% | ±0 | 0/20 | 30 |  | |
claude-sonnet-4-6anthropic | 100% | ±0 | 1/20 | 7 |  | |
claude-sonnet-5anthropic | 100% | ±0 | 0/20 | 31 |  | |
claude-sonnet-5-5anthropic | 100% | +5 pts | 1/20 | 24 |  |
Not yet measured: google, openai — adapters exist, API keys do not. They join the index when a key is configured.
How to read it
- Pass rate — share of the 20 prompts whose answer matches its golden exactly (numbers, JSON shape, a regex). It measures instruction-following consistency, not intelligence.
- Answers changed — prompts whose answer text differs from the previous day's answer for the same model. Zero means the model answered identically; a jump on a day with no announced release is what people call "nerfed".
- For your own agent — agentcheck's Pro plan does the same with your checks: when a provider ships a model, it re-runs your suite against your endpoint and emails what broke. Re-run your checks on Pro.
The suite (drift-suite/v1)
arith.mul— What is 17 * 23? Reply with the number only. (exact)string.reverse— Reverse the string 'drift'. Reply with the reversed string only. (exact)count.letters— How many times does the letter r appear in the word strawberry? Reply with the number only. (exact)json.capital— Return a JSON object with keys "city" and "country" for the capital of Japan. JSON only. (json_schema)list.primes— List the first five prime numbers as a comma-separated list, nothing else. (regex)translate.fr— Translate 'good morning' into French. Reply with the translation only. (contains)sort.words— Sort these words alphabetically: pear, apple, mango. Reply with a comma-separated list only. (regex)fact.gold— What is the chemical symbol for gold? Reply with the symbol only. (exact)convert.temp— Convert 100 degrees Celsius to Fahrenheit. Reply with the number only. (regex)compare.decimals— Which number is larger, 9.11 or 9.9? Reply with the larger number only. (exact)sequence.double— Complete the sequence 2, 4, 8, 16, __ . Reply with the missing number only. (exact)format.upper— Write the word yes in uppercase letters. Reply with that word only. (exact)extract.email— Extract the email address from this text: "Contact us at hello@example.com for details." Reply with the address only. (exact)logic.squares— Is the statement "all squares are rectangles" true or false? Reply with one word. (regex)json.colors— Return a JSON object with a key "colors" holding an array of the three primary colours red, blue and yellow. JSON only. (json_schema)fact.moon— In which year did humans first land on the Moon? Reply with the year only. (exact)sql.distinct— Which SQL keyword removes duplicate rows from a SELECT result? Reply with the keyword only. (regex)instruct.ok— Reply with exactly the text OK and nothing else. (exact)arith.speed— A train travels 60 km in 1.5 hours. What is its average speed in km/h? Reply with the number only. (regex)count.words— Count the words in this sentence: "The quick brown fox jumps over the lazy dog". Reply with the number only. (exact)