Meta's recent release of Llama 4, its latest AI model, has set the tech world abuzz—though not for the reasons the company might have hoped. The launch of two new models, Scout and Maverick, aimed to position Meta as a serious competitor in the AI arena, but a leaked detail has raised questions about the integrity of the benchmarks used to promote these models.
Maverick's ELO Score and the Benchmarking Controversy
Maverick, one of the Llama 4 models, made waves when it secured the number-two spot on LMArena, a popular AI benchmark site. The model’s impressive ELO score of 1417, which placed it just below Gemini 2.5 Pro and above OpenAI’s 4o, was touted as a sign that Meta's new AI was on par with the industry leaders. This seemingly decisive victory raised expectations about Llama 4's capabilities and the company's place in the rapidly growing AI space. However, AI researchers soon uncovered something unusual. Upon closer inspection, Meta's documentation revealed that the version of Maverick tested on LMArena was not the same one that would be available to the public. Meta had deployed a custom, "experimental chat version" of Maverick, optimized specifically for better conversational performance. This small but crucial detail had been buried in the fine print of Meta’s release, leading to confusion and frustration in the AI community.The Fallout: Meta’s Explanation and LMArena’s Response
After the revelation, LMArena issued a statement clarifying that Meta had not followed the expected protocol for model submissions. The benchmark site made it clear that the customized model used in the tests had not been sufficiently labeled, which could lead to misleading results. “Meta should have made it clearer that ‘Llama-4-Maverick-03-26-Experimental’ was a customized model to optimize for human preference,†LMArena posted on X, pledging to update their policies to ensure fairer evaluations moving forward. Meta’s spokesperson, Ashley Gabriel, responded with an explanation: "We experiment with all types of custom variants." Gabriel emphasized that the version used in the benchmark was indeed an optimized chat version, but that the company had now released the open-source version of Llama 4 for public use. The spokesperson also expressed excitement about seeing how developers would customize the model for their own needs. Despite Meta's justification, the damage had already been done. The gaming of the benchmarks raised doubts about the reliability of results from systems like LMArena, especially when custom versions of models are used for promotional purposes.
Benchmarking AI: A Growing Battleground
The Maverick controversy highlights a growing concern in the AI community: as AI technology accelerates, so do the stakes surrounding benchmark rankings. Models like Llama 4 are increasingly being evaluated not just for their inherent capabilities, but also for how well they perform on widely-accepted benchmarks, which serve as a key metric for developers when choosing AI models for their projects. “LMArena is the most widely respected general benchmark because all of the other ones suck,†said Simon Willison, an independent AI researcher. "When Llama 4 came out, the fact that it came second in the arena — just after Gemini 2.5 Pro — that really impressed me. But I’m kicking myself for not reading the small print." While what Meta did with Maverick is not explicitly against LMArena’s rules, the decision to submit a specialized version of the model makes the benchmarks less meaningful for developers seeking realistic insights. This situation underscores the challenge developers face when navigating a world where benchmarks, once considered a reliable tool, can be manipulated to project an artificial sense of superiority.Meta’s Response to Training Accusations
The controversy didn’t stop at the benchmarks. Shortly after Maverick’s release, rumors began to swirl that Meta had trained its Llama 4 models on specific test sets to improve benchmark performance. However, Ahmad Al-Dahle, VP of Generative AI at Meta, publicly denied these claims. "We’ve also heard claims that we trained on test sets — that’s simply not true, and we would never do that," Al-Dahle posted on X. Despite the clarification, many in the AI community remained skeptical. Independent researchers noted that the timing of the release and the secrecy surrounding certain aspects of Llama 4 added to the confusion. "It’s a very confusing release generally," said Simon Willison. "The model score that we got there is completely worthless to me. I can’t even use the model that they got a high score on."Timing and Transparency: Meta’s Strategic Release
Another unusual aspect of the release was its timing. Traditionally, major AI announcements take place on weekdays when the media and tech world are fully engaged. Instead, Meta dropped Llama 4 over the weekend, sparking further questions about the company’s strategy. When asked why the release occurred at an unusual time, Meta CEO Mark Zuckerberg simply replied, "That’s when it was ready." This lack of transparency only fueled the speculation and suspicion surrounding the launch.