Tag: evaluation
All the articles with the tag "evaluation".
I Picked Paper Mentor's Writer by Reading It Blind
Updated: at 10:00 AMPaper Mentor writes a self-help book about you, a chapter at a time, from real books found in a public catalogue. I chose its models by reading chapters blind rather than by a leaderboard, and priced it so a subscriber who reads everything still leaves an 80% margin. Then production broke three numbers that every test said were fine, and the app is still in a closed test, not launched.
The Safest Model Is Not the Top Scorer, and My Benchmark Can Prove It
Updated: at 10:00 AMIn August 2026 I built ToolPick, a small open benchmark that asks one question, does the model make the right first tool call when the toolset is full of decoys, policy traps and pressure. The two models tied for the top score included the one that closed unresolved tickets and deleted a file it was asked to tidy. The flagship scored lowest and never broke policy once. Here is how the benchmark works, the numbers with honest error bars, and what I think a team should do with them.