Intelligence Index v5 Rebuilds Test, Open-Weight Gap Triples
By @sharedot · · 7 pages
- AI Frontier
- AI Benchmarks
- Open Weight Models
- Artificial Analysis
Artificial Analysis' Intelligence Index v5 uses 60% private test data, with the closed vs open-weight model gap tripling from 3 to 12 points.
What happened: a benchmark rebuilt from the inside
Artificial Analysis launched Intelligence Index v5 in late October, its biggest overhaul since the index began, and the leaderboard reshuffled around it.
Claude Opus 5.5 at max reasoning effort now leads with 58, ahead of OpenAI's GPT-6 Astra at 57 and GPT-6.1 Sol at 55 — a two-point spread across the top three.
The decisive change is methodological: for the first time, 60% of the index runs on private test sets or held-out answers that labs cannot study in advance, a direct response to benchmark contamination.
Two new evaluations — Cognition's FrontierCode 2.0 and Terminal-Bench-Science — anchor the shift, alongside three upgraded benchmarks, per OfficeChai's detailed breakdown of the release.
Why it is surprising: open models went from closing to falling back
The reversal is the story practitioners are arguing about. Just months earlier, Moonshot's Kimi K3 had closed to within 6 points of the leader under v4.3 scoring, and Artificial Analysis itself had noted open-weight models halved the gap over the prior year.
Under v5, the gap between the top closed model and the best open-weight model widened from 3 points to 12. Trending Topics reports Kimi K3.1 Pro fell from 55 to 46, DeepSeek V4.1 Flash plunged from 45 to 32, and Xiaomi's MiMo-V2.6-Pro lost 16 points to end at 30.
The best model with weights actually released now sits 26 points behind Opus 5.5 — behind even Claude Haiku 5.5 at 36.
The evidence: where the new tests break open models
The steepest drops land on the new, harder private tests. On the revised HLE-Diamond, Kimi K3.1 Pro scores 37% against Opus 5.5's 68%; on Terminal-Bench-Science, Kimi reaches 34% while GPT-6 Astra hits 63% and most open-weight models stay below 10%, OfficeChai reports.
Yet the picture is not one-sided: Kimi leads the entire field, closed models included, on Terminal-Bench 4.0 at 67%, and no single model wins all nine evaluations — Opus 5.5 tops three, with the other six split among five models.
Trending Topics cautions the ranking depends heavily on which tests are chosen and that secret test data can no longer be independently verified from outside.
The stakes: buyers' math changed overnight
Artificial Analysis sells no models, which is precisely why procurement teams cite it when justifying seven-figure inference contracts.
Startup Fortune notes the uncomfortable part: if vendor selection leaned on the old index, the math behind that decision changed even though the running model did not.
A companion Progress Index v1 places every major model since November 2022 on one capability scale to track how fast the ground shifts.
What comes next: an open question about the ruler itself
Whether the widened gap reflects real capability or an artifact of how the private tests were built is exactly what r/LocalLLaMA has been disputing since the release landed, Startup Fortune reports.
Open-weight labs may have less access to private evaluation partnerships or held-out data partnerships that frontier labs can use, which would penalize them on tests they cannot rehearse.
With 60% of the index now secret, v5 is designed to resist gaming longer than its predecessors — but the same secrecy makes the score harder to audit. Expect the next open-weight releases, including Kimi's already-teased variants, to be judged against a ruler that now changes every few months.