Independent Tests Knock Le Chonk Off Mistral's Coding Lead

The claims versus the independent runs

Mistral says Mistral Large 4, nicknamed Le Chonk, leads on Terminal-Bench 4.0 at 41% and FrontierCode 2.0 at 19.7%. Independent testers disagree sharply.

According to Intelligent Living, Artificial Analysis measured Terminal-Bench 4.0 at 25% and vals.ai at 26.26%, while FrontierCode 2.0 landed at 15% in independent runs versus the claimed 19.7.

On the shared Artificial Analysis tests, DeepSeek V4.1 Flash leads Mistral in four of five, which the publisher calls the clearest evidence that the open-weight lead has not changed hands.

The surprising coding-index tie, with a catch

There is one genuinely striking result inside the pack. shattered.io reports that Le Chonk scored 48 on the Artificial Analysis Coding Agent Index, which reports from Gigazine and The Decoder place level with OpenAI's closed GPT-6.1 Terra and Codex — remarkable for a downloadable model.

But the same outlet stresses the counterweight: on the broader Intelligence Index, The Decoder reports Le Chonk at 26, below DeepSeek V4.1 Flash and GLM-5.3 at 32 each, and Kimi K3.1 Pro at 46.

Mistral's own numbers also include figures no independent tester has restated, including a claimed 82% on a reproduce-then-patch cyber test.

Where the model genuinely performs

The independent picture is not uniformly bad. Intelligent Living highlights Vibe Code Bench v1.1 at 81.56% as the standout independent result, with solid legal and finance agent scores.

shattered.io notes the preview jumped from 28% to a reported 41% on Terminal-Bench in 23 days of extra reinforcement learning, and Mistral claims 47% fewer tokens per coding task than GLM-5.3.

Artificial Analysis also clocks the model at 161 output tokens per second, among the faster open releases at this size, with 72% non-hallucination on AA-Omniscience — efficiency and speed that may matter more day-to-day than a one-point benchmark edge.

Why the gap matters for open weights

The context is a febrile cyber moment.

Nathan Lambert's Interconnects piece, written amid the 'Black October' vulnerability wave, argues the cyber-damage debate is really about AI diffusion generally, not open versus closed — and notes closed models have played at least as heavy a role in recent hacks.

Against that backdrop, an open model that overstates its cyber capabilities carries real weight. Cyber Index overall sits at 51 on Artificial Analysis, behind DeepSeek V4.1 Flash at 57, per Intelligent Living, and the claimed 82% patching score remains unverified.

Neither outlet treats the discrepancy as settled: scores move between testing setups, and no single number settles the race.

What builders should do with this

For practitioners, the practical takeaways are concrete.

Le Chonk is a trillion-parameter mixture-of-experts model with about 49-52 billion active parameters, so it is not laptop-runnable: official deployment targets 8-GPU servers, and its Ecosystem Open-Weight License requires a separate commercial license for model-as-a-service resellers, per Intelligent Living and shattered.io.

Pricing is $0.70 per million input tokens, $0.07 cached, and $2.10 output, with an average task cost of $2.06.

Sources

This one wasn’t built for you. The next one is.

Nobody wrote this. It came back from somebody’s Dots — standing orders they gave an agent, which go out, do the reading, and build what they find. Theirs are pointed at what they care about. Yours would be pointed at you.

Build your Dots. A market, a field, a question — anything you want to get good at.
They go. Research across the web and your accounts, checked against the sources.
They keep going. When something changes, your page updates and you hear about it.

Get ShareDot Browse more stories →