I Picked Paper Mentor's Writer by Reading It Blind

On 16 October 2026 I opened a single HTML page with fifteen chapters on it. Five test readers, and for each one the same first chapter written three times from the same plan by three different models. The letters were shuffled per reader, so “A” was never the same model twice, and the key was held back until I had written down my picks. Under each trio was one line: Best for this reader: A / B / C. Any you wouldn’t pay for?

Opus 5.5 won four of the five. Sonnet 5.5 won none. That read, more than any score, decided what writes the chapters in Paper Mentor, my reading app, and the cost of that decision then set every price in it.

Paper Mentor is not launched. As I write, it is in a closed test on Google Play with billing switched off, so nobody can pay for it and there is no revenue in this post. What there is: a model choice made by reading, unit economics worked out before any sale, and three production bugs that slipped past every gate.

Table of contents

What the app does

You answer four questions: what you want help with, where you are with it, the voice you want, and one sentence in your own words. The server, which I call the press, plans a short book and proposes catalogue search queries, never titles, which go to Open Library. A second pass assigns the real books that come back to chapters, using only keys from that pool, so a book the model invents resolves to nothing and is dropped. Then chapters stream onto the phone: an opening, three sections, a margin note, and one small practice.

That is the same instinct as never letting the model do the maths: make dishonesty unreachable by data flow. This post is about the decisions data flow could not make for me.

Why gates could not choose the model

The press has two jobs. Planning is structural: a JSON plan with chapter titles, promises and ten catalogue queries. Prose is the product.

My eval harness has gates, hard invariants a reader could catch me on (every citation in the pool, the block format intact, no markdown on the page), and scores, quality targets (length in the band the reader chose, three decades on the shelf, the reader’s own words echoed back). A model judge also answers three anchored questions per chapter: is it specific to this reader, does the argument earn its next move, is it honest.

Six reader fixtures went through each prose model, and the three current models ran twice more, so each wrote 18 first chapters from the same plans:

Opus 5 was out on the spot: one chapter came back with one section instead of three. Everything else passed every gate in every round. Gates say what is unacceptable; they have nothing to say about which acceptable chapter a person would pay for.

The scores and the judge leaned towards Opus 5.5, but gently: it led on specific and honest and tied Sonnet 5.5 on earns. A third of a point on a five-point scale is a hint, not a verdict. The one hard signal was length: Haiku wrote short in ten of eighteen chapters.

The blind read

So I read them. Five of the six fixtures went onto the page (the sixth, the crisis fixture, is read by hand separately). Each block showed what that reader had asked for: a startup engineering lead who can no longer finish anything, someone weighing a safe offer against an interesting one with a mortgage, a new parent with six minutes before sleep, someone who chose “tell me stories” and wrote nothing, and someone learning the things their late father knew.

I answered one question per trio: which would I pay for, for this person. Not “which is best written”. A beautiful chapter that could have been for anyone fails; a plain one that picks up their exact words and pushes on them passes.

Opus 5.5 won four of five. Sonnet 5.5 won none, so the fifth went to Haiku. The caveat: this is one reader, me, across five chapters, and it is my taste. But it is the right kind of evidence for a product whose whole claim is “this was written for you”, and it agreed with the length data and the direction of the judge.

Planning: the opposite answer

Planning gave the opposite answer. The previous planner was Sonnet 5:

Haiku 5.5 passed all 81 gates in about a third of the wait. Sonnet 5.5 was faster still and lost: one chapter credited a book it had not been given, a gate, and its queries found nothing at all for one reader. Haiku’s planning for six books came to 12,759 input and 23,666 output tokens, about $0.013, or $0.002 a book. So: Haiku 5.5 plans, Opus 5.5 writes, and the reasoning sits above the routing table in the code so whoever changes it has to read why first.

The choice had a cost I fixed separately. Opus 5.5 always thinks, and at its default effort it thought for 2,800 to 4,500 tokens before the first word: 32 to 45 seconds of blank page. Effort “low” brought that to 200 to 400 thinking tokens and first words in 3 to 9 seconds, with the same length and about 60% less output. I picked the writer by quality and bought the latency back with a setting.

It is the ToolPick lesson from the other side: there the top scorer was the riskiest model, here every model cleared the bar. Either way, the leaderboard number was not the decision.

Unit economics, before the first sale

The pricing rule is one sentence: a subscriber who uses every chapter on their plan must still leave at least 80% of what they paid, after the store’s 15%, and in India after 18% GST too. Worst case, not average case.

Opus 5.5 measured $0.057 a chapter across the 18 test chapters. I added 30% for retries and longer later chapters:

measured Opus 5.5 chapter        $0.057
+ 30% retries and longer ones    $0.057 x 1.3 = $0.0741
worst-case budget                $0.074 a chapter
planning (Haiku 5.5)             about $0.002 a book

Two plans, monthly or yearly. The margin is measured on what the store passes on, with the same function the dashboard runs: net = price x 0.85, cost = chapters x $0.074, margin = (net - cost) / net.

My billing notes fold planning in too and land a point lower on Devoted monthly (83%); every plan clears 80% either way. India, once: ₹599 less 18% GST is ₹507.63, less the store’s 15% is ₹431.48. Ten chapters at $0.074 and ₹88 to the dollar is ₹65.12, which leaves 84.9%.

Three consequences fell out of the rule rather than out of taste:

• No free trial. The first chapter is free, once per install. A trial on top would break the worst-case margin.
• Raise VAT countries by about 10%. Converted store prices include VAT; at 20 to 25% VAT they leave only 77 to 81%, and 10% higher restores 80%.
• A cheaper tier is a smaller plan, not a discount. Five chapters at about $2.99 keeps the floor: about $0.37 of cost against about $2.54 net.

Devoted is 11% cheaper per chapter than Reader ($0.533 against $0.599), and yearly saves about 10%. Care chapters and chapters the press failed to finish are refunded automatically.

The weak point is in bold in my billing notes: if the prose model changes, these margins change with it. The $0.057 was measured at Opus’s default effort, before “low” cut output by about 60%, so real cost should now sit under budget. For part of the testing period the deployed press wrote with Haiku through an environment override; I removed it on 3 November so testers read what launch will use. The dashboard shows measured cost beside the worst-case budget and, when chapters run on another model, what they would have cost on Opus 5.5.

Three bugs only production found

Before anyone outside could install it, the offline suite was green: on the demo press, which swaps the model calls for fixtures and keeps catalogue retrieval real, the sweep was 121 gates and 162 scores with no failures. Then I bound one real book against production, and it could not be done.

1. The token cap: 1,500 against about 4,300. The planning call was capped at 1,500 output tokens. A real plan needs about 4,300. Every live bind failed, and the error depended on where the cut landed: an “empty sheet” when the budget went before any text, a “torn sheet” when the JSON stopped mid-object. Both readings blamed the model. The demo press could not see it, because there the plan is a fixture and the model is never asked.

2. The route duration: 60 seconds against about 62. The serverless function had maxDuration = 60, with a comment calling it the platform ceiling and estimating a bind at 15 to 25 seconds. Measured against the live model, the plan alone took 33 seconds and a full bind took about 62. Every reader who gave more than the bare minimum had their function killed in the matching stage. The app said “the connection dropped, try again”, which was exactly right, and that correct message is why it looked like a model problem. The number only means anything on the platform, and no test ran there.

3. The deadline I left behind: 45 seconds, two weeks later. Each upstream model call had its own 45-second deadline, sized for the 60-second route. When I raised the route to 300 seconds, I did not revisit it. The live sweep on 1 September then passed 81/81 gates, with plans taking about 33 seconds when I measured them. A fortnight later, on a day the model was slower, every real bind died at exactly 45 seconds in the plan stage, and the reader’s screen showed one word as a title: deadline.

Why could no gate catch these? A gate checks what came out, and here nothing came out. Gates are written against artefacts: a plan, a pool, a chapter. These bugs were the absence of an artefact, caused by a number sitting between the model and the reader. The first needed the real model, the second the real platform, and the third the real model on a slow day. A sweep is a photograph. Latency drifts.

What changed:

The fixes I trust more than the new numbers are the ones that make a failure say what it is. A reply stopped by the ceiling now fails with stop=max_tokens in its diagnostic instead of pretending to be unparseable JSON. A chapter cut short says so at the foot of the page instead of trailing off mid-sentence. The deadline surfaces as a 504 with copy the app already treats as “try again”. The truncation cases and the deadline each have tests pinning them now.

That diagnostic paid for itself within hours: the assignment call later failed with stop=max_tokens · cap=2500 · out=2500, naming its own cause. In October a crisis note made the model deliberate so long that assignment spent all 4,000 of its tokens, because thinking counts towards the cap. Each time, the error said which number to change.

There is now a script, probe-bind.mjs, that binds one real book for a few cents, prints where the time went, exits non-zero on failure, and warns when the plan passes 75% of the per-call deadline. The runbook says to run it daily for the first weeks. Binding one real book is the check that found all three.

The care path

Paper Mentor is self-help, so some readers will write something that means they are not safe. It is not a crisis service and does not pretend to be one; the point of this path is to get out of the way and point at people who can help.

When a reader’s own words suggest they may not be safe, the first chapter is not a chapter. It names what they wrote without diagnosing it, says plainly that a machine is not the help this needs, gives helplines (the card covers seven countries with checked numbers, and an international directory elsewhere) and “tell one person tonight”, and carries no sections, no practice and no reading list. It is never metered.

This path broke in production too. In the first full live sweep, the crisis fixture was the only one of six that failed. The care rule was in the system prompt for every call, so the planning call correctly answered with care prose instead of JSON, and the bind died on a missing brace before any care reached anyone. My first fix asked the prompt for an exception; it held about half the time. The real fix was to send the care rule only to the chapter and the afterword, never to calls whose output nobody reads, with a test asserting that split.

The harder question is when to raise care at all. Too eager, and someone grieving or burnt out is handed a helpline instead of the book they asked for. So the eval has a care-decision set: eleven readers in a hard stretch (grief, a breakup, a layoff, a figure of speech like “this job is killing me”) who must get their chapter, and seven at risk, most of them saying so indirectly, who must get care. The live run was 18 of 18, twice. That is a fixture set, not a guarantee, and the runbook says to read the care chapters by hand after every change to the prompts or the prose model, because the gates are not the standard here.

Where it actually stands

Not launched. In a closed test on Google Play, with billing built but switched off, so the paywall and margins above are designed and tested, not earned. No paying subscribers, no revenue.

What I took away from it

1. Gates say what is unacceptable; they cannot choose between acceptable options. When every model passes, read the output blind, and ask the question your customer will ask: would I pay for this, for this person?
2. Choose on quality, then buy latency back with configuration. Effort “low” gave Opus 5.5 first words in seconds. Choosing a weaker writer for speed would have been permanent.
3. Price from the worst case and write the dependency down. The margin is a function of the prose model. Say so next to the prices, and make the dashboard show what reality would cost on the model you priced.
4. A number between the model and the reader is a bug waiting for a slow day. Token caps, route durations and deadlines were right in every test and wrong in production, and the only check that found them was binding one real book.
5. Make failures name themselves. The first time, a truncated plan looked like the model’s fault. Once stop=max_tokens was in the diagnostic, the next one was a one-line fix.
6. Safety rules go only where they can be obeyed. A JSON call cannot carry a crisis line. Route the rule to the place that can.