The Model Gets a Vote, Not a Veto

I run a task board for myself. It watches the places obligations arrive: mail, calendar, chat, issue trackers. It pulls out the things I have committed to and orders my day. Models do work in it I would not want to write by hand. One reads a message and decides whether there is an obligation in it. One embeds tasks so the same obligation seen twice can be recognized. One reranks the open list against a query about what matters today.

Three model steps. None of them decides anything on its own.

That was not an ideology I brought to the project. It is what fell out of asking, at each step, what happens when this call is wrong. The answers are wildly different depending on where in the pipeline you ask, and the design follows the asymmetry.

The Score Is Arithmetic

The board’s priority score is a weighted average over five features, clamped to zero-to-one. Due proximity, decaying smoothly from overdue to far off. Commitment type, where a deadline outweighs an ask, which outweighs a promise I made, which outweighs an informational note. Source weight, because a ticket assigned to me carries more than a chat mention. The extractor’s confidence in its own reading. And an importance signal the extractor emits, which is the differentiator for anything with no date attached.

The reranker enters that blend at a weight of 0.15.

It is a tiebreaker. A generic “what matters to me today” query produces near-uniform relevance scores across a board of tasks that are all, in some sense, things I said I would do. The reranker has genuine signal, and the signal is small, so it gets a small weight. Deciding that was a matter of looking at raw logits, not philosophy.

Two properties keep it honest. The blend is a relative-weighted average that renormalizes over the components present, so the reranker term is included only when a reranker produced a score. A raw weighted sum would have quietly inflated every task toward critical the moment I added the term. And when the rerank backend is down, the four deterministic components renormalize among themselves and ranking proceeds. Reranker failure degrades the score. It never stalls the board.

That is the shape I want from a model component: pull it out and the system is duller, not broken.

Wrong Merge Versus Missed Merge

Deduplication is where the asymmetry gets sharp, and it is the highest-risk correctness area in the whole thing.

The same obligation reaches me through several doors. Someone asks in chat, files an issue, and mails a summary. Three sources, one thing I owe. Collapsing them is what makes the board usable.

Merging is also the one operation that destroys data. A merge reassigns the duplicate’s provenance rows to the surviving task and deletes what is left. Get it right and two entries become one. Get it wrong and an obligation I never saw disappears from the board, and nothing downstream will tell me it was there.

Compare the two failure directions. A wrong merge is unrecoverable. A missed merge is a duplicate on the board that gets caught on a later pass or by me, in about four seconds. Those costs are not close, so the rules are not symmetric:

• Exact URL match is the only path that merges without asking a model. Two sources pointing at the same issue link are the same obligation, and that is a string comparison.
• Cosine similarity never merges on its own. High similarity only gates the question. It buys the right to ask a model “are these the same obligation?” and only an explicit yes merges.
• A missing confirmation function, an exception, or an uncertain answer all resolve to no merge.
• Embeddings from different model backends are never compared, because a fallback to a different embedding provider puts vectors in a different space and a similarity score across the two means nothing.
• The survivor is always the older task. Not the one with the better score, not the one the model prefers. Lower first-seen timestamp wins, so the operation is deterministic and repeatable.

Read that list back and notice where the model sits. It cannot initiate a merge. It can only ratify one that a deterministic gate has already proposed, and its silence counts as no. That is the vote-not-veto arrangement in one function.

Deterministic Where the Headers Already Know

There is a cheaper version of the same principle worth naming, because it is easy to skip.

Mail is the noisiest source. The question “is this actionable” used to be answered from the subject line, which cannot distinguish “can you sign off by Thursday” from “FYI, cutover done.” So the connector now carries the message body into extraction. That part is genuinely a judgment call, and a model makes it.

What the connector also carries is how the message was addressed to me: whether I am a direct recipient, copied only, or on a mailing list, plus a recipient count. That is not a judgment call. It is header parsing. A List-Unsubscribe header or a bulk precedence marker is a fact, and the extraction rules treat bulk mail as never actionable without ever consulting a model about it.

Every fact you compute rather than infer is one the model cannot get creatively wrong, and one you can test. Being a direct recipient does not make a message actionable either, which is the part that still needs judgment. The split runs exactly along the line between what the headers state and what the prose implies.

Failures Have to Terminate

One more piece of the pipeline is deterministic on purpose: giving up.

The worker re-selects anything not yet extracted on every sweep. So a message the extractor cannot handle gets retried forever. A malformed encoding, a body that trips something: each one burns a model call every cycle, quietly, in a loop nobody watches. Extraction attempts are therefore counted, and past a threshold the pipeline stops selecting that item. The failure then surfaces through a health endpoint and a dismissible banner in the UI.

A deterministic failure must not requeue forever. The retry ceiling is a boring counter in a column, and it converts an invisible recurring cost into a visible one-time signal. Cost control and observability turn out to be the same change.

Where the Model Belongs

The rule I would give someone building this: put the model where a wrong answer costs a scroll, and keep it away from anywhere a wrong answer costs a record.

Ranking is a scroll. If the model’s opinion pushes the wrong thing to the top of my board, I look past it and lose two seconds. So the model gets a vote there, and a small one. My own manual priority override still wins over the computed score. The score is kept alongside it, shown for what it is: the model’s opinion, not the board’s decision.

Merging is a record. If the model is wrong there, an obligation vanishes and the failure is silent. So its vote there is strictly negative. It can stop a merge and it cannot start one.

None of this is a claim that the models are bad. They are good, and the board would be a worse tool without them. It is a claim about where nondeterminism belongs in a system you intend to leave running unattended. That is the same instinct as graduating the deterministic parts of a workflow off the model runtime once their shape is clear. Here it happens at the granularity of single features inside one pipeline rather than whole services.

It also compounds with what I know about how these systems fail. A model that cannot do the work will produce a confident, well-formed answer anyway. A confident wrong merge looks exactly like a confident right one. The defense is to leave it no path to act unilaterally.

And where an agent acts on the world, the same reasoning lands one layer down: on what it can destroy and what you can restore. That is confirm before acting is not a safety feature.

If you are wiring a model into a pipeline that runs unattended, reach out at hello@escapecommand.com.