The Hardest Thing for an Editing Agent Is Leaving Things Alone

“It is not the traffic. It is not the traffic. It is not the traffic.”

In a transcript that looks like a stutter, or three takes nobody cleaned up, and an editor working from text would keep one. Said out loud, it is the point of the video.

I planted that line in a test video for ClearCut, the agentic video editor I built between July and August 2026, to see whether the agent would leave it alone. It kept the first two, trimmed the third, and gave a reason: “two hits land the rhythm, the third is one too many.” My answer key said all three stay. In the end I changed the scorer, not the agent, because the agent’s call was defensible and the scorer was the thing being rigid.

That run scored 16 pass, 0 fail, 2 weak. This post covers that eval, a caching mistake that made every edit cost two to two and a half times what it should, and three bugs one real run found after 181 unit tests had missed them.

Table of contents

What ClearCut is

You drop in raw footage: a talking head, a screen recording, a podcast. The agent transcribes it, watches it, pitches a short creative brief (story, target runtime, the cold open it would pick, what it plans to cut) and waits for you to say go. Then it cuts a draft you can play straight away and revise by conversation: “tighten the middle”, “keep the joke at 2:14”, “make it a Short instead”.

The core idea is that the edit is data, not a rendered file: an edit decision list anchored in source time and played virtually over the original video, so every change shows up on the next frame and encoding happens once, at export. The agent runs on Claude Opus 4.8 through the Vercel AI SDK. Transcription runs on-device with Whisper in the browser, so the footage stays on the machine; only the transcript, scene metadata and the frames the agent asks to see go to the model.

How an edit actually happens

The agent has 25 tools. These do most of the work:

The unusual part is where the tools run. The server runs the model loop, but every tool executes in the browser, which owns the media and the edit state: the client runs each call against a zustand store and sends the result back as the next request. Every mutating tool pushes an undo snapshot, and the transcript doubles as the timeline, with cut words struck through and the agent’s reason beside them. A request stops after 24 model steps, and the agent gets at most 30 frames per creator turn, because images are the expensive part.

Export goes through ffmpeg, with FCPXML and EDL for finishing elsewhere. As in my AI harness, the model decides, while state, measurement and undo live in ordinary code.

Why restraint is the hard property

Deleting is easy. Any agent told to “tighten this” will find things to cut, and an eval that only checks whether planted flaws went will reward aggressive cutting. My first eval was exactly that: the tighten fixture planted five flaws (a weak open, a false start, a filler run, a tangent about a cat, the worse of two takes). The agent removed 5 of 5 and kept 7 of 7 load-bearing lines in all three clean runs.

That says less than it seems. A transcript-reading editor fails in the other direction. It sees a quiet line and reads dead air, a deliberate repetition and reads a retake, a pause and reads a gap to tighten, “this will sound unrelated” and reads a tangent, because it has not yet reached the line that pays it off. Each mistake makes the cut look tidier on paper and the video worse.

I also wrote the prompt and the answer keys, so a removal-only eval could easily flatter my design. It is the worry behind ToolPick: the dangerous failure is the confident action that should not have happened, and you only see it when the right answer is to hold back.

The restraint eval

The restraint fixture is a 70 second monologue with four traps that must survive and two real flaws that must go:

The real flaws matter: if the right answer were “cut nothing”, an agent that did nothing would score perfectly.

The footage is synthetic. A build script renders each sentence with macOS say, measures it, and stitches the clips with exact silences, so the answer key holds real timings. Each section also shows N white squares, where N is the section number. That count is nowhere in the audio, so asking how many squares are visible at 0:50 can only be answered by looking at a frame.

The runner drives the real app in Chrome, says “go” after the brief, and captures every tool call off the wire from the /api/agent traffic, with nothing instrumented inside the product. The scorer prints the evidence for every check and reports INCONCLUSIVE rather than passing what it cannot evaluate. The trap check is the interesting part:

const fraction = removedSeconds / Math.max(0.1, t.end - t.start);
add(
  `Kept the ${t.label.replace("TRAP-", "")}`,
  !hit ? "PASS" : fraction >= 0.6 ? "FAIL" : "WEAK"
  // ...
);

Losing most of a trap is a failure of judgement. Trimming part of one is a taste call. The first version of the scorer could not tell them apart.

What the restraint run did

The first restraint run, on 19 August, scored 16 pass, 0 fail, 2 weak:

• It kept the quiet setup, the bakery tangent and the 2.6 second beat.
• It protected the beat with an explicit hold from 56.6s to 59.3s, with the reason “keep the real pause so the 41% reveal hits harder”. Gap tightening would otherwise have eaten it.
• It cut both real flaws with cutRange rather than throwing away whole scenes.
• It counted the squares correctly (5) and named the background colour sequence unprompted.
• The cut went from 69.7s to 51.4s, first word at 0.2s, with 0 critic failures.

The two weak results were the triple, trimmed by 25%, and one console error from a missing favicon.

One caveat: since the first agent commit in July, the playbook has said “a quiet scene that sets up a later payoff MUST stay” and that payoffs get holds. So passing the quiet setup and the pause is partly the prompt doing what it says. The eval shows the instruction surviving real tool calls on real footage, not the model inventing the idea.

A second run on 3 September scored 16 pass, 0 fail, 3 weak, with the same call on the triple (23% this time). The extra weak result was a bug in my critic, which demanded a cropped frame for a format meant to keep the frame you shot. The agent declined to crop, which was right.

The $2.61 edit

The eval also prices every run. The first measured edit of the 78 second tighten fixture cost $2.61, a lot for just over a minute of footage. The cause was one line in the agent route:

const system = context
  ? `${SYSTEM_PROMPT}\n\n${buildContextBlock(context)}`
  : SYSTEM_PROMPT;

Every request appended the live project context (transcript, scenes, current timeline, delivery settings) to the system prompt. Two problems were stacked. First, nothing carried a cache_control breakpoint, and Anthropic only caches what you mark. Second, the obvious place for one was poisoned. The cache is an exact prefix match in render order (tools, then system, then messages), so with the system prompt changing on every request, any breakpoint from there on would write a new entry each time, pay the write premium, and never be read. The conversation renders after the system prompt, so it, contact sheets included, could never be reused either. A long edit resent the playbook and the whole history at full price on every request.

The fix moves the volatile part after the last breakpoint:

const cacheable = {
  anthropic: { cacheControl: { type: "ephemeral" as const } },
};
const history = await convertToModelMessages(messages, { tools: agentTools });
const modelMessages = [
  { role: "system", content: SYSTEM_PROMPT, providerOptions: cacheable },
  ...history.map((m, i) =>
    i === history.length - 1 ? { ...m, providerOptions: cacheable } : m
  ),
];
if (context) {
  modelMessages.push({
    role: "user",
    content: [{ type: "text", text: buildContextBlock(context) }],
  });
}

The static system prompt and the last history message each carry a breakpoint, so each request reads the conversation so far from cache. The context rides at the end, never stored in the history, so changing it breaks nothing.

I then made the scorer read cache usage. On Opus 4.8 a cache read costs a tenth of the $5 per million input price and a five-minute cache write 1.25 times it; the scorer prices each and prints what the same tokens would cost uncached. The two runs whose raw scorecards I kept:

Across the three clean tighten runs, cost per edit was $0.55 to $0.86. Later tighten runs in September cost $0.78 and $0.65.

My notes from the time mix up two comparisons. They call the fix “a 60% reduction”, but 60% is the caching saving on one run against the same tokens at full price. Going from $2.61 to $0.55 to $0.86 is a 67% to 79% drop, and not all of that is caching: the $2.61 run was the same fixture and task but a different conversation, and I no longer have its token counts. The like-for-like number is the one I trust: caching saved 60% on the tighten run and 50% on the restraint run, which wrote more to cache.

The design has costs. The context block, transcript included, is billed at full price on every request, which is why long transcripts are abridged. Dropping old contact sheets from the history (the client keeps three) changes an earlier message, so the cache is rewritten from there. And a five-minute entry only stays warm while requests keep coming: a creator who goes for a coffee pays for a fresh write.

Three bugs the unit tests could not see

Before the first eval run the project had 181 passing unit tests covering timeline maths, word-safe cutting, the critic, captions and NLE export formats. The first run of the real app found three bugs. (The project notes say 189 tests, which includes 8 regression tests I added for the third bug afterwards.)

1. Ingest crashed on every file with speech. prepareFootage copied transcription progress into the progress bar from inside a store subscription. zustand calls every listener on every set(), and setPrep always wrote a new object, so the listener’s own write triggered it again until “Maximum call stack size exceeded”. The fix mirrors only on an actual change, and setPrep now returns the same state when nothing differs:

const unsubProgress = useStore.subscribe((s, prev) => {
  const busy = s.transcribeBusy;
  if (!busy || busy === prev.transcribeBusy) return;
  st().setPrep(busy.label, 55 + busy.pct * 0.35);
});

2. The Whisper worker never started. It uses ES module imports but was constructed as a classic worker, so it threw on construction every time and transcription silently fell back to the main thread: the exact UI blocking the worker exists to prevent. The fix was one missing option, { type: 'module' }.

3. On-device Whisper produced zero words. Word timestamps need the ONNX export to include cross-attention outputs, and the quantised build the app downloads does not. transformers.js threw, and everything built on word timings (filler removal, sentence cuts, captions) was dead on the on-device path. It now falls back to segment timings with word boundaries approximated inside each segment.

None of these could fail in the unit suite. Vitest runs in Node: no test ran real ingest with a live subscription, constructed a browser Worker, or loaded the real quantised model, whose missing outputs are a property of a file downloaded at runtime. Bug 2 also hid itself, since the fallback still produced a transcript, slowly and on the wrong thread.

It happened again in October: on-device transcription was dead on machines without a GPU, and SRT files repeated sentences. A review and real runs caught both; 257 unit tests had not. It is the lesson of my sovereign coding agent again: nothing works until you have watched the real thing run.

What is real, what is demo, and who uses it

Real: the agent loop and its 25 tools, the timeline, undo, preview, both export engines and on-device transcription, running locally with your own API key; the eval harness and the numbers above; and a hosted mode with a spend governor (per-account balance, rate limit, global daily circuit breaker).

Demo, or not proven:

• The evals use synthetic speech, and the “subject” on screen is a disc, not a face. No real footage of a real person has been through them.
• I wrote the prompt and both answer keys. Three clean runs is evidence, not a benchmark. Two other runs were invalidated, one by bug 1 and one by a network drop, and I reported them rather than quietly re-running.
• Nobody has measured a 20 minute edit. Cost scales with conversation length too; runs on the 78 second fixture took 10 to 21 turns.
• The hosted meter’s store is in memory and resets on every deploy, which is fine while I am the only tester and useless for billing. The in-app ledger and the hosted meter also still price every input token at full rate, ignoring cache discounts, so both overstate the real cost.

Who uses it: as far as I can tell, nobody but me. I wrote a document on the 20 kinds of people who might pay for it and what was stopping each one. That is a hypothesis, not a customer list, and the product has never taken a payment.

What I took away from it

1. For an editing agent, restraint is the property worth measuring. Removing planted flaws is easy to score and easy to pass. Only traps that must survive show whether the agent is reading the video or just the text.
2. Let the eval tell a taste call from a mistake. Trimming the third repetition of a triple, with a reason, is not the same failure as deleting the line. A scorer that treats them alike tunes the agent towards one editor’s preferences.
3. Measure caching on identical tokens. The fix was one restructure: static content first, breakpoints at stable boundaries, volatile context last. The honest saving was 50% to 60% on the same tokens. The rest of the fall from $2.61 came from a different conversation, and I nearly credited it to caching.
4. Unit tests show the parts are right; one real run shows whether they fit. Three bugs, each fatal to a core feature, passed 181 tests and did not survive the first run against real footage in a real browser.
5. Know what your eval cannot prove. Synthetic voices, a disc for a face and an answer key I wrote myself put a ceiling on these numbers. ClearCut does not need another feature next. It needs somebody else’s footage.