The Wall: Why 15% is the New 100%
MMLU is solved. GSM8K is trivial. So researchers created 'Humanity's Last Exam' (agi.safe.ai). It's a collection of the hardest questions from every field—quantum mechanics, ancient history, abstract topology—designed to be un-googleable. And guess what? It's humiliating the industry.
Current SOTA models (R1, O3, Claude 3.7) are scoring around 15%. For context, a PhD in the specific field scores around 80%. This gap proves that while AI is broad, it is shallow. It can pass the bar exam because the bar exam is memorization. It can't invent a new legal theory because that requires reasoning.
Memorization vs. Reasoning: The Great Deception
The public has been duped. We see a model solve a riddle and think 'AGI'. But 'Humanity's Last Exam' exposes the truth: these models are just really good at pattern matching. When faced with a novel problem that requires multi-step deduction without a training data template, they fall apart. They hallucinate. They guess. They fail.
# The Reality of 'Reasoning' Models
def solve_novel_problem(problem):
if problem in training_data:
return exact_answer
else:
return plausible_sounding_nonsense()The Real Definition of AGI
This benchmark is the new finish line. When a model scores 90% on this exam, we can officially pack it up. Until then, anyone claiming 'AGI is here' is selling you a bridge or a wrapper. We aren't there yet, and the curve is flattening, not vertical.
