708 articles tagged with Benchmark
Ivan Fioravanti ᯅ (@ivanfioravanti)평가(Eval)에서 thinking tokens가 결과와 소요 시간에 큰 영향을 주며, 모델이 과도하게 생각하는 경우를 벌점화하기 위해 평가별 최대 토큰 제한이 필요할 수 있다는 제안이다.https://x.com/ivanfioravanti/status/2045...
Gökdeniz Gülmez (@ActuallyIsaak)Apple의 MLX 프레임워크에서 LLM 성능을 평가하기 위한 첫 번째 종합 벤치마크인 MLX-Benchmark Suite가 소개되었습니다. 코드 이해, 작성, 디버깅 능력을 측정하는 CLI 도구와 데이터셋을 포함한 개발자용 평가 도구입니다.https://x.com...
As AI-assisted video creation becomes increasingly practical, instruction-guided video editing has become essential for refining generated or captured footage to meet professional ...
We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates existing multiword expression (M...
Multimodal large language models (MLLMs) have achieved impressive progress on vision language benchmarks, yet their capacity for visual cognitive and visuospatial reasoning remains...
The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of the...
HORIZON Benchmark Diagnoses Long-Horizon Failures in GPT-5 and Claude AgentsA new benchmark called HORIZON systematically analyzes where and why LLM agents like GPT-5 and Claude fa...
🔬 Claude Opus 4.7 vs Opus 4.6 — Real Benchmark ResultsCoding: Opus 4.7 is 2.5x faster (13.4s vs 33.9s)Reasoning: TieContext: TieFull benchmark via Crazyrouter 👇https://crazyrouter....
Voice deepfake losses are projected to hit $40B by 2027, a 6,566% jump from 2023. Modulate's velma-2 now ranks #1 on Hugging Face's Speech Deepfake Leaderboard with a 1.104% averag...
Teams are making critical model selection decisions based on benchmarks designed for someone else's...
📰 Alibaba's Mystery AI "Happy Horse" Crashes the Video Generation Leaderboard — Then Reveals ItselfA stealthy 15-billion-parameter video AI appeared from nowhere, dethroned ByteDan...
Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation r...
The paradigm of agentic science requires AI systems to conduct robust reasoning and engage in long-horizon, autonomous exploration. However, current scientific benchmarks remain co...
Large language models have demonstrated strong performance on general-purpose programming tasks, yet their ability to generate executable algorithmic trading strategies remains und...
Diffusion-based image editing has achieved strong visual fidelity under natural language instructions, yet most existing systems still operate at the level of surface instruction f...
【VAKRAの内部構造：エージェントの推論、ツールの使用、および障害モード】https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (...
Which LLM is best for your AI agent? MiniMax-M2.7 vs Claude Sonnet 4.6📊 Benchmark Results:- Task completion: MiniMax +12%- Response quality: Claude +8%- Tool usage: TieSee full ben...
📰 ClawBench: AI Agents Succeed in Just 33.3% of Real Tasks (2026 Study)ClawBench, a new benchmark testing AI agents on 153 real-world online tasks across 144 live websites, reveals...
LABBench2 Benchmark Shows AI Biology Agents Struggle with Real-World TasksResearchers introduced LABBench2, a 1,900-task benchmark for AI in biology research. It shows current mode...
【Open ASRリーダーボード：新しい多言語トラックと長尺トラックによるトレンドとインサイト】https://huggingface.co/blog/open-asr-leaderboard※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
We audited our own pattern-embedding evaluation and found 53% of held-out samples had same-symbol training neighbors within 20 days. Here's what we changed — and why agent develope...
AI is improving all the time but don't be too worried about AGI, take for example this clockwatching benchmark.Source: https://hai.stanford.edu/ai-index/2026-ai-index-report#AI #Cl...
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studi...