Skip to content

Research

Built on three papers.

The theory behind Mark 1, published as working papers from Mark 1 by Omium Research: what a verified outcome costs, how to verify without a test suite, and how agents can share what they learn without sharing code.

Paper I

Tokens per Verified Outcome

An economic theory and benchmark protocol for agentic coding systems.

What you pay for a coding agent is not the price of a request. It is every token in the session, divided by the outcomes that were actually verified. The paper defines that metric, derives when checking an agent's work is worth what the check costs, and shows why today's leaderboards can't report it.

  • A break-even for verification

    A verifier lowers cost only if its discrimination clears a threshold set by how often the agent fails, what an attempt costs, and how expensive a wrong answer is to find later.

    v* = (1 − p)·D·(α − β) − c·(1 − α)
  • How many repair rounds

    One formula gives the cost-optimal number of self-repair rounds. With nothing fitted it returns 2 at function scale and 6 at repository scale, matching the two knees reported in the literature.

  • Not recoverable from pass@k

    Resolution rate and mean cost per instance cannot be turned into tokens per verified outcome. Two systems can report identical numbers and cost very different amounts.

  • Waste, in four channels

    Session waste splits exactly into iteration, exploration, misallocation and context, so you can see where the tokens went, not only how many.

Cost of one verified outcome, tokens
  • No verifier 300.0k
  • LLM judge · AUROC 0.537 360.0k +20%
  • LLM judge · AUROC 0.65 257.1k −14%
  • Grounded rubric · AUC 0.886 214.7k −28%

Model-implied at p = 0.4, c = 30k, D = 150k tokens. The weaker judge is worse than having no verifier at all, even if it were free. Paper I, Table 1.

Paper II

Verification Without Oracles

Calibrated outcome adjudication for software agents in test-poor environments.

Agents misreport their own success: across roughly ten thousand measured trajectories, 35.6% of task failures on τ²-bench were cases where the agent declared success, and no LLM judge caught them above AUROC 0.65. Published verifiers are tested on repositories that have test suites. Most production repositories don't. This paper combines weak signals into one calibrated verdict, with no labeled data.

  • Recoverable from agreement alone

    With three or more independent groups of checks, the class prior and every check's error rates can be estimated from unlabeled agreement. A check that can only reject correctly, like a compile failure, fixes the orientation.

  • Four verdicts beat two

    A checker that may answer satisfied, violated, refused or error has strictly lower expected cost than any forced yes-or-no rule whenever some cases fall in the uncertain band. Reporting a checker outage as "your agent failed" carries a provable excess risk.

  • Self-certifying labels

    The same estimates give the noise rate of the verdicts themselves, which gates whether they are fit to train other models on.

  • Abstaining pays

    In the worked example, the four-verdict rule costs about 192.6k tokens per verified outcome, against 245.9k for a strict binary rule and 208.2k for a lenient one, and 36% below having no verifier.

One ensemble, every verdict
Rubric agent Reward model Anchor Posterior Verdict
accept accept accept 0.934 satisfied
accept reject accept 0.612 refused
reject accept accept 0.385 violated
reject reject accept 0.065 violated
any any reject 0 violated
checker outage: no signal error
86.1% decided 13.9%

Worked example: a rubric agent, a reasoning reward model and a compile-and-type anchor, prior 0.4. Exactly one pattern lands in the abstention band, and that is the refused share. Paper II, Section 12.

Paper III

Federated Experience

Privacy-preserving, cross-organization procedural memory for coding agents.

Lessons from one company's agent sessions can make another company's agent better. The raw sessions can't move: model vendors are contractually barred from training on customer content, and mixing unfiltered self-generated successes into training has been shown to hurt. The paper shares distilled procedure instead, never trajectories or code.

  • Artifacts, not trajectories

    Workflow templates, pitfall and remedy pairs, invariants and routing hints. A written grammar and static checks keep code, identifiers and free text out. Only those artifacts, with capped statistics, cross the tenant boundary.

  • Privacy you can cite

    The shared pool has no tenant identifier, so every released view is (k, m)-anonymous by construction. Statistics are released with differential-privacy noise, and each tenant's budget is accounted for.

  • Do no harm

    An artifact is admitted only when its measured benefit clears a lower confidence bound. With high probability every admitted artifact has positive utility, so on average the pool does no worse than private memory.

  • Travels only as far as it should

    Retrieval discounts an artifact by the distance between the source and target task distributions. In the worked example, a new tenant gets a guaranteed 3.8% expected saving on day one.

Why curation is not optional

+6.6 pts resolve rate for +7.5% tokens, about 9% fewer tokens per resolved issue, when distilled experience is injected and same-repository experience is excluded.

15.3% → 8.7% what one strong open agent scored after training on unfiltered self-generated successes.

  1. 1 Stays local Sessions, trajectories and verified labels never leave your side.
  2. 2 Abstract Literals, paths, identifiers and free text become typed placeholders.
  3. 3 Admit Kept only if its measured benefit clears a lower confidence bound above zero.
  4. 4 Release Only when k artifacts from m tenants share its coarse profile, with noise on the counts.
  5. 5 Retrieve Used by another tenant only as far as its tasks resemble the source.

Reported figures are cited from the literature in Paper III, Section 1. The pipeline is Sections 4, 5 and 8.

Working papers, v1, August 2026. Anurag Upadhyay, Mark 1 by Omium Research. Papers I to III of the Mark I Protocol series.

These are theory, algorithms and evaluation protocols. Every figure above is a closed-form computation from a paper's stated model, calibrated to values published in the literature it cites. None is a measurement of Mark 1, and the papers don't describe what the current app does.