What It Really Takes to Benchmark Frontier AI
Why Humanity’s Last Exam Matters — and Why Workflow Contributed
AI models are improving so quickly that many of the benchmarks used to evaluate them are no longer enough. Once a model consistently scores highly on a benchmark, that benchmark stops being a useful measure of frontier capability. That is exactly the gap addressed by A benchmark of expert-level academic questions to assess AI capabilities, recently published in Nature. The paper introduces Humanity’s Last Exam (HLE) — a rigorous benchmark designed to test AI systems with questions at the frontier of human expertise. (Nature)
At Workflow, we contributed to this effort because we believe one of the most important questions in AI today is not just “What can a model do?” but “How do we know what it can actually do reliably?” Benchmarks shape how the world understands AI progress, and weak evaluation leads to inflated claims, misplaced trust, and poor deployment decisions.
The Problem with Traditional AI Benchmarks
Most well-known AI benchmarks were useful when large language models were earlier in their development. But as frontier models improved, many of these tests became saturated. In fact, the Nature paper notes that state-of-the-art models now exceed 90% accuracy on popular benchmarks like MMLU, making them far less useful for measuring real frontier progress. (Nature)
That creates a serious issue.
If the benchmark is too easy, it stops telling us whether a model can:
- reason deeply,
- generalize to unfamiliar expert-level material,
- remain calibrated under uncertainty,
- or handle difficult, high-context problems beyond memorized internet patterns.
In other words, we stop measuring capability and start measuring benchmark familiarity.
What Makes Humanity’s Last Exam Different
HLE was built to be intentionally difficult — not through trick questions, but through real academic depth.
The benchmark contains 2,500 expert-level questions spanning over 100 subjects, with contributions from nearly 1,000 subject matter experts across more than 500 institutions and 50 countries. It includes both text-only and multimodal questions, along with exact-match and multiple-choice formats designed for automated grading. (Nature)
More importantly, each question was designed to meet a very specific bar:
- it must have a clear, verifiable answer,
- it must be hard for frontier models,
- and it must not be solvable through simple retrieval or superficial pattern matching. (Nature)
That distinction matters. A model that can retrieve facts is useful. A model that can reason through difficult, domain-specific problems under ambiguity is something else entirely.
Why This Matters Beyond Research
This is not just an academic exercise.
If AI is going to be trusted in domains like healthcare, engineering, science, law, or safety-critical workflows, then evaluation needs to move beyond broad consumer-style tests. We need better ways to measure where models are strong, where they fail, and where they are confidently wrong.
One of the most important findings in the paper is that frontier models still perform poorly on HLE and often show poor calibration, meaning they can answer incorrectly with high confidence. (Nature)
That has direct implications for real-world deployment.
At Workflow, this is exactly the kind of issue we care about. Whether we are building AI systems for healthcare, technical reasoning, domain-specific automation, or structured decision support, evaluation is not a side task — it is part of the product itself.
Why Workflow Contributed
We contributed to this work because we see evaluation as foundational to responsible AI development.
Our perspective has always been practical: AI should not just impress in demos — it should perform under pressure, in the real world, in domains where precision matters. That means helping shape better standards for measuring capability, reasoning, and reliability.
Contributing to HLE aligns with how we think about AI at Workflow:
- build systems that are useful in real environments,
- test them against meaningful difficulty,
- and stay honest about what current models can and cannot do.
That mindset is especially important as more companies rush to deploy AI into workflows that affect real people, real decisions, and real outcomes.
The Bigger Shift
Benchmarks like HLE represent a bigger shift happening in AI.
We are moving from the era of “Can models do this at all?” into the era of “How well do they do it when the task is actually hard?”
That is a much more valuable question.
And it is one Workflow is proud to contribute to.
At Workflow, we build human-centered AI systems grounded in real-world utility, rigorous thinking, and meaningful evaluation.