I picked a coding agent off a leaderboard. It flopped on our codebase.
Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good...
#Benchmark
674 articles tagged with Benchmark
I built an LLM eval framework from scratch. Here is what I wish I had bought instead.
One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed...
Comparing Two Eval Runs by Their Average Pass Rate Is the Wrong Test
TL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B...
We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.
TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one...
Your RAG Eval Isn't Flaky. Your Retrieval Is Non-Deterministic.
Same query. Same documents. Same model. And the RAG eval can still hand back a different...
📊 Mi:dm K 2.5 Pro — the actual numbers GPQA: 70.1% MMLU-Pro: 80.9% Humanity's Last Exam: 7.7% Long Context Reasoning: 9%...
📊 Mi:dm K 2.5 Pro — the actual numbers GPQA: 70.1% MMLU-Pro: 80.9% Humanity's Last Exam: 7.7% Long Context Reasoning: 9%Measured independently, not self-reported →https://opensourc...
The Leaderboard Is Dead. Here's What I Actually Reach For.
The Leaderboard Is Dead. Here's What I Actually Reach For. Let It Break — part 2 Tags: #ai...
Show HN: Benchmark your eng team's AI agent maturity in 5 minuteshttps://agent-benchmarks.com/software-factory/#ai
Show HN: Benchmark your eng team's AI agent maturity in 5 minuteshttps://agent-benchmarks.com/software-factory/#ai
I Tested 300+ Models. Then I Killed the Benchmark.
I Tested 300+ Models. Then I Killed the Benchmark. Let It Break — part 1 Tags: #ai #llm...
Best AI agents fail 50% of visual tool tasks, Apple benchmark showsA new open benchmark with 500+ tools reveals even fro...
Best AI agents fail 50% of visual tool tasks, Apple benchmark showsA new open benchmark with 500+ tools reveals even frontier AI models can't reliably read an image and act on it, ...
AI music transcription scores 38% on new pop benchmarkA new 572-segment pop music benchmark shows the best AI transcript...
AI music transcription scores 38% on new pop benchmarkA new 572-segment pop music benchmark shows the best AI transcription models still miss most notes in real recordings, affecti...
MedRealMM benchmark: AI matches doctors but fails on safetyNew 5,620-case benchmark finds frontier AI matches physicians...
MedRealMM benchmark: AI matches doctors but fails on safetyNew 5,620-case benchmark finds frontier AI matches physicians on positive clinical steps but triggers more unsafe respons...
Agentic tool-use eval on a local 35B (Q8): trap-tool avoidance is solid, but I can't tell if my failures are the model or my harness
I've been running a small agentic eval harness against a local model and I'd like a sanity check on...
Benchmark an AI Agent Migration Without Believing One Speedup Number
A paired workload protocol for validating latency, cost, correctness, and recovery before changing production models.
Your RAG Eval Is Checking the Receipt, Not the Patient
Clinical RAG can pass grounding checks while citing evidence about the wrong entity.
もうClaudeのモデル選びで迷わない — Anthropicが教える独自Eval（評価テスト）と「成功1件あたりのコスト」の測り方https://qiita.com/nogataka/items/6cd8c6ea583221e45def?...
もうClaudeのモデル選びで迷わない — Anthropicが教える独自Eval（評価テスト）と「成功1件あたりのコスト」の測り方https://qiita.com/nogataka/items/6cd8c6ea583221e45def?utm_campaign=popular_items&utm_medium=feed&utm_source=popula...
LLM agents fail 70% of skill safety tests in SLBench studyA new 86-case benchmark finds Codex and Claude Code agents vio...
LLM agents fail 70% of skill safety tests in SLBench studyA new 86-case benchmark finds Codex and Claude Code agents violate logical constraints in skill files up to 70% of the tim...
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
Large language models (LLMs) have achieved remarkable performance on high-school and olympiad-style mathematics, yet their capabilities on advanced mathematics remain poorly unders...
The benchmark that built the tools
How we measured a markdown knowledge graph as agent memory — lost to grep, rebuilt our search and...
📊 Apriel-v1.6-15B-Thinker — the actual numbers GPQA: 73.3% MMLU-Pro: 79% Humanity's Last Exam: 9.8% Long Context Reasoni...
📊 Apriel-v1.6-15B-Thinker — the actual numbers GPQA: 73.3% MMLU-Pro: 79% Humanity's Last Exam: 9.8% Long Context Reasoning: 50.3%Measured independently, not self-reported →https://...
How to Review an Edge LLM Benchmark Without Fooling Yourself
Record model identity, device state, power, latency distribution, and thermal behavior before turning one local-inference run into a product conclusion.
Simple Benchmark Review: Ollama on Jetson Nano
This particular rabbit hole was created due to a previous conversation I had here with another member...
GPT-5.5 leads EvoPolicyGym: top-two across all 16 environmentsEvoPolicyGym, a new arXiv benchmark, tests whether AI agen...
GPT-5.5 leads EvoPolicyGym: top-two across all 16 environmentsEvoPolicyGym, a new arXiv benchmark, tests whether AI agents can autonomously rewrite executable policies under a fixe...