AI evaluations analysis: Solaiman wants open standards

A conference stage is a bold place to tell the AI industry its measuring tape is made of licorice. But that is the useful tension in Irene Solaiman’s TechCrunch Disrupt appearance: before you let an AI do the job, you might want to know whether the test saying it can do the job is itself house trained.

According to TechCrunch’s Disrupt 2026 agenda, Solaiman of Hugging Face appeared in a session focused on how to verify that AI can actually handle a task before assigning it one. That framing is refreshingly practical in a market where benchmark charts can sometimes resemble nutrition labels for candy. The takeaway was not that evaluations are useless. It was that evaluations are becoming too important to leave as vendor theater with a leaderboard attached.

TechCrunch put the evaluator hype on notice

TechCrunch reported that Solaiman, Hugging Face’s chief policy officer, described current AI evaluation methods as "complicated and still maturing" during TechCrunch Disrupt. That matters because TechCrunch also reported that AI labs lean on benchmarking to argue one product is better than another. If benchmarks are the scoreboard, Solaiman is asking who calibrated the scoreboard, who can rerun the match, and whether the scoreboard has been quietly trained on marketing copy.

The counterintuitive lesson is that embedded evaluators and outside reviewers do not automatically solve the trust problem. According to TechCrunch, Solaiman pointed to demand for an embedded evaluator ecosystem while saying the qualifications for effective evaluators across tasks need much more standardization. In plainer builder English: adding an auditor is not the same thing as defining what good auditing means. That is not oversight, it is cosplay with clipboards.

TechCrunch says reproducibility is the trust tax

TechCrunch reported that Solaiman tied trust in AI evaluation results to reproducibility and access to the model being tested. She said part of the question is whether others can reproduce an evaluation and whether they have sufficient model access, adding that "open weights come in in a big way." Open weights are not fairy dust, sadly, because the AI industry keeps refusing to ship fairy dust under an Apache license. But access does change what outside evaluators can verify rather than merely admire from across the velvet rope.

For teams choosing models, that should shift procurement questions from “what score did it get?” to “can we understand how that score was produced?” If a benchmark cannot be rerun, compared, or inspected, it may still be interesting, but it is weaker evidence for deployment decisions. The best evaluation culture looks less like a beauty pageant and more like a lab notebook: boring, repeatable, and hostile to hand waving.

SiliconReport says the target is moving beyond the model

SiliconReport reported that Solaiman warned AI testing is expanding from standalone models to more complex and expensive agent systems. TechCrunch similarly reported that evaluations are moving from testing only the model to testing the broader system around it, including the harness and agent. That is a meaningful scope change. A model score can tell you something, but it does not automatically tell you how the surrounding system behaves once the model is embedded inside it.

TechCrunch reported that Solaiman called agentic evaluations hard and expensive, which is the kind of sentence that makes both researchers and finance departments reach for coffee. The industry has already learned that model benchmarks can age quickly as capabilities, prompts, and products change. Agent systems add another layer of evaluation complexity because the test is no longer just the model in isolation. It is the model plus the system context that turns outputs into actions.

TechCrunch says standards are the missing API

TechCrunch reported that Solaiman described evaluations as an evolving science and said the industry still lacks a meaningful way to compare different evaluation approaches. That is the crux: third party review is useful only if the methods are comparable enough for builders, customers, and policymakers to interpret. Otherwise, every lab brings its own ruler, every ruler has a sponsored podcast, and nobody agrees how tall the table is.

According to TechCrunch, a more uniform industry wide approach may be needed if benchmarking is going to remain useful for leaders. That does not require every organization to run identical tests forever, because that would be how you turn science into a DMV line. It does mean AI teams should demand clearer documentation, reproducible procedures, adequate model access where possible, and evaluator standards that match the task being assessed.

For builders, the immediate move is simple: treat evals as operational infrastructure, not launch confetti. Ask whether results can be reproduced, whether evaluators had enough access, what system was tested, and whether the benchmark actually maps to your use case. The next useful AI eval will not be the loudest chart. It will be the one that survives someone else pressing rerun.