Opus 4.8 tops the LLM leaderboard with 95% on skill evals
We added Claude Opus 4.8 to our ongoing model benchmark. It scored 95% with skill context, which puts...
#Benchmark
675 articles tagged with Benchmark
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not b...
When experts grade LLM answers in their own field, how well do the citations hold up?ExpertQA, a 2024 benchmark, has 484...
When experts grade LLM answers in their own field, how well do the citations hold up?ExpertQA, a 2024 benchmark, has 484 experts write questions in their specialty, then judge the ...
More eval traces will not stabilize your kappa. Stratify the ones you have
TL;DR: Our LLM-as-judge agreement (Cohen's kappa against human labels) swung between 0.41 and 0.63...
The Eval Gap: Your Agent Has Observability but No Idea If It's Any Good
89% of teams running production AI agents have observability, but only 52% have evals. That gap is where agent quality dies — and closing it is a human-labeled data problem before ...
ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics
Combinatorics is central to Olympiad-level mathematical problem solving, requiring deep discrete reasoning, creative constructions, and rigorous structural insight. Recent evidence...
inferbench — download, run & benchmark local LLMsRun and benchmark local LLMs without the setup pain. inferbench is a de...
inferbench — download, run & benchmark local LLMsRun and benchmark local LLMs without the setup pain. inferbench is a desktop app that downloads, launches and benchmarks local infe...
I Built an Adversarial Eval Framework and Attacked 5 LLMs — Every Single One Failed
10 adversarial scenarios, 64 assertions, 3-tier evaluation pyramid. Llama, Qwen, GPT-OSS — none scored above 63%. Here's what broke them.
Why We're Changing Our Default Eval Model
We're changing the default solver model in our eval harness from Claude Sonnet 4.6 to GLM 5.1. This...
OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics
Vision-language model (VLM) agents are increasingly deployed in interactive game environments. Yet game benchmarks for VLM agents typically report a single first-attempt score per ...
τ-Rec: A Verifiable Benchmark for Agentic Recommender Systems
As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace. Current benchmarks often rely on "LLM-as-a...
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchm...
MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between...
iOSWorld: A Benchmark for Personally Intelligent Phone Agents
A useful phone agent needs to be personally intelligent. It should reason over a user's identity, history, and preferences as they exist on the device, not just follow isolated ins...
"Skill Is Not Document: A Query-Conditional Benchmark and Two-Stage Retriever for LLM Agent Skill Routing"Skills used by...
"Skill Is Not Document: A Query-Conditional Benchmark and Two-Stage Retriever for LLM Agent Skill Routing"Skills used by LLMs require a different retrieval pipeline than the one us...
Estos LLM son los mejores para resistir la propaganda rusaUn benchmark del gobierno de Estonia evaluó decenas de modelos...
Estos LLM son los mejores para resistir la propaganda rusaUn benchmark del gobierno de Estonia evaluó decenas de modelos de lenguaje grande para medir su capacidad de contrarrestar...
Arena.ai startet die Agent Arena, einen Benchmark für autonome KI-Agenten basierend auf echten Nutzersitzungen statt kün...
Arena.ai startet die Agent Arena, einen Benchmark für autonome KI-Agenten basierend auf echten Nutzersitzungen statt künstlichen Tests.Der Test wertet über 330.000 Sitzungen aus, u...
inferbench v0.1.1 🖥️ download, launch & benchmark local LLM engines (llama.cpp & co) from one desktop app. Real tok/s on...
inferbench v0.1.1 🖥️ download, launch & benchmark local LLM engines (llama.cpp & co) from one desktop app. Real tok/s on your own GPU, no cloud, no guesswork. Now serves models + e...
The Benchmark Lied. Here’s What It Didn’t Measure.https://cariagiovannib.wordpress.com/2026/06/07/the-benchmark-lied-her...
The Benchmark Lied. Here’s What It Didn’t Measure.https://cariagiovannib.wordpress.com/2026/06/07/the-benchmark-lied-heres-what-it-didnt-measure/#AI #AIResearch #llm #mlops #linux ...
China LLM API Benchmark 2026: Prices, Speed, and Setup Guide
Chinese models now account for 61% of global LLM token consumption. DeepSeek, Qwen, GLM, and Doubao...
LLM Wire Format Benchmark: Which Format Can AI Actually Read and Write?
Every LLM wire format claims token savings. Nobody proves whether AI models can actually comprehend...
PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems
Enterprise property graphs vary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. A deployment-relevant T...
My LLM pipeline passed every eval. Then it started lying to users in production.
My LLM pipeline passed every eval. Then it started lying to users in production. Not dramatically....
https://winbuzzer.com/2026/06/06/meta-weighs-equity-raise-as-alphabet-sets-ai-benchmark-xcxwbn/A rumored equity raise by...
https://winbuzzer.com/2026/06/06/meta-weighs-equity-raise-as-alphabet-sets-ai-benchmark-xcxwbn/A rumored equity raise by Meta is making AI infrastructure funding a dilution test as...