<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

 <title>Huascar Sanchez Posts</title>
 <link href="http://huascarsanchez.com/atom.xml" rel="self"/>
 <link href="http://huascarsanchez.com/"/>
 <updated>2026-11-11T17:11:30+00:00</updated>
 <id>http://huascarsanchez.com/</id>
 <author>
   <name></name>
   <email></email>
 </author>

 
 
 <entry>
   <title>Who Audits the LLM Judge?</title>
   <link href="http://huascarsanchez.com/posts/notes/who-audits-the-llm-judge.html"/>
   <updated>2026-10-27T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/who-audits-the-llm-judge</id>
   <content type="html">&lt;h2 id=&quot;a-judge-can-be-right-for-the-wrong-reasons&quot;&gt;A judge can be right for the wrong reasons&lt;/h2&gt;

&lt;p&gt;&lt;a href=&quot;/posts/notes/how-do-you-localize-a-fault-in-a-guardrail.html&quot;&gt;The previous post&lt;/a&gt;
asked whether a guardrail’s rules are right. This one asks whether whoever applies
the rules, often an LLM judge, followed them.&lt;/p&gt;

&lt;p&gt;This question matters because so much now depends on the judge’s verdict.
An LLM judge reads a response and returns a verdict, usually paired with a rationale.
Increasingly, those verdicts gate real decisions: whether an answer ships, an agent’s
next step is accepted, or a model gets promoted.&lt;/p&gt;

&lt;p&gt;When we evaluate a judge, we usually just ask: &lt;em&gt;did it get the right answer?&lt;/em&gt; If
we have a gold label, that’s easy to check. But we don’t just give a judge a
question; we also give it an evaluation protocol: rules for how to reach its
verdict. Something like:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Identify every factual claim. Check each one against only the supplied context.
  Do not use outside knowledge. PASS if and only if every claim is supported.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Whether the judge followed that protocol is a different question from whether its
verdict matched the gold label, and the two come apart in ways that matter.&lt;/p&gt;

&lt;p&gt;To see how, consider four test cases from my test suite. In each one, a judge is
evaluating the same response: &lt;em&gt;“Alice joined ACME in 2024 and is its CTO.”&lt;/em&gt; In
case A, the supplied context confirms both claims, so the response should pass.
In cases B, C, and D, the context only says &lt;em&gt;“Alice joined ACME in 2024,”&lt;/em&gt;
leaving the CTO claim unsupported, so the response should fail.&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;Case&lt;/th&gt;
      &lt;th&gt;Verdict correct?&lt;/th&gt;
      &lt;th&gt;Procedure followed?&lt;/th&gt;
      &lt;th&gt;Rationale supports its verdict?&lt;/th&gt;
      &lt;th&gt;Claims grounded in context?&lt;/th&gt;
      &lt;th&gt;Overall&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;A&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;satisfied&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;B&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;&lt;strong&gt;no&lt;/strong&gt;&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;violated&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;C&lt;/td&gt;
      &lt;td&gt;&lt;strong&gt;no&lt;/strong&gt;&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;&lt;strong&gt;no&lt;/strong&gt;&lt;/td&gt;
      &lt;td&gt;violated&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;D&lt;/td&gt;
      &lt;td&gt;&lt;strong&gt;no&lt;/strong&gt;&lt;/td&gt;
      &lt;td&gt;&lt;strong&gt;no&lt;/strong&gt;&lt;/td&gt;
      &lt;td&gt;&lt;strong&gt;no&lt;/strong&gt;&lt;/td&gt;
      &lt;td&gt;yes&lt;/td&gt;
      &lt;td&gt;violated&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;In Case B, the judge gets the right answer, but only by breaking the rules. Its
rationale admits: &lt;em&gt;“I also checked my outside knowledge to confirm this first
claim,”&lt;/em&gt; which the protocol explicitly forbids. A simple gold-label check would
mark this judge as correct, whereas a procedural audit catches the violation.&lt;/p&gt;

&lt;p&gt;Case C is the opposite, and it’s the more worrying failure mode. Here, the judge
follows every step of the protocol, and its verdict follows logically from its
rationale. The problem is that it claims the context supports the CTO role when
the text says nothing of the sort. The judge is internally consistent, yet
completely wrong, and the only way to catch the error is to check its claims
against an independent reading of the context.&lt;/p&gt;

&lt;p&gt;Case D combines both failures. Its rationale says the context does not support
Alice’s title, then draws on outside knowledge to return PASS anyway. It
contradicts itself and breaks the protocol.&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Correctness asks whether the judge landed in the right place. Adherence asks
  whether it got there the way it said it would.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/hsanchez/unmask&quot;&gt;Unmask&lt;/a&gt;’s adherence check is built to tell those apart.&lt;/p&gt;

&lt;h2 id=&quot;what-goes-in-a-replay-not-a-vibe&quot;&gt;What goes in: a replay, not a vibe&lt;/h2&gt;

&lt;p&gt;To understand what Unmask’s output means, it helps to be clear about what goes in.
Unmask doesn’t just read a judge’s rationale and decide whether it sounds careful.
Instead, it checks a &lt;strong&gt;structured replay&lt;/strong&gt; of the evaluation, which consists of five
parts:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;The protocol&lt;/strong&gt;, translated into propositional &lt;em&gt;atoms&lt;/em&gt;. Each atom has one of three roles:
a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;judge&lt;/code&gt; atom represents something the judge asserts (e.g., “the context supports
claim 2”), a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;context&lt;/code&gt; atom captures an independent observation about the source
material, and a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;procedure&lt;/code&gt; atom records an observable step (e.g., “the judge
consulted outside knowledge”).&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;A decision expression&lt;/strong&gt; (a propositional formula) over those atoms that defines the verdict. In our four
test cases, for example, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PASS&lt;/code&gt; holds if and only if both claims are supported.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Requirements&lt;/strong&gt; the judge must follow, each defined by a propositional formula and
an optional &lt;em&gt;coverage&lt;/em&gt; expression indicating whether the requirement was observable
in the first place.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Annotations&lt;/strong&gt; that assign each atom a value of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;TRUE&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;FALSE&lt;/code&gt;, or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt;,
backed by exact evidence spans quoted from the source text.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Grounding pairs&lt;/strong&gt; that link each of the judge’s claims (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;judge&lt;/code&gt; atoms) to the
corresponding independent observation (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;context&lt;/code&gt; atoms).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To make this concrete, here is how Case B encodes the “use only supplied context”
requirement alongside the annotation that violates it, trimmed from the actual
fixture:&lt;/p&gt;

&lt;div class=&quot;language-json highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;id&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;context_only&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;description&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;Use only supplied context&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;expr&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;op&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;not&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;args&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;atom&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;external&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}]},&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;addressed&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;const&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-json highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;atom&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;external&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;value&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;TRUE&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;evidence&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[{&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;source&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;rationale&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;start&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;end&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;182&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;quote&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;There are two claims. Joining ACME in 2024 is supported. I also checked my outside knowledge to confirm this first claim. ...&quot;&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}],&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;explanation&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;Rationale explicitly relies on external knowledge. Hand-annotated semantic assessment; span validity does not prove interpretation.&quot;&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The annotations in these fixtures were written by hand, which is a real limitation
that I’ll return to below. For now, though, notice what this design buys us: instead
of having another model silently re-derive semantic judgments inside the checker,
those judgments are surfaced as explicit, quotable facts. A reviewer can look at
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;external = TRUE&lt;/code&gt;, read the attached quote, and decide for themselves whether they
agree.&lt;/p&gt;

&lt;p&gt;The pipeline:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart LR
    accTitle: Adherence verification pipeline
    accDescr: A replay is validated, then three solver engines classify the verdict, the requirements and the grounding pairs, and the results are aggregated conservatively into a report.
    R[&quot;Replay JSON&quot;] --&amp;gt; V[&quot;Validate&amp;lt;br/&amp;gt;bounded IR, exact spans&quot;]
    V --&amp;gt; E[&quot;Three engines&amp;lt;br/&amp;gt;protocol, evidence, combined&quot;]
    E --&amp;gt; C[&quot;Classify each proposition&quot;]
    C --&amp;gt; A[&quot;Aggregate conservatively&quot;]
    A --&amp;gt; P[&quot;Report with witnesses&quot;]
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&quot;principle-1-unknown-is-not-false&quot;&gt;Principle 1: unknown is not false&lt;/h2&gt;

&lt;p&gt;Real evaluations are rarely complete. For instance, a rationale might never mention
whether the judge used outside knowledge. If we silently treated “not mentioned” as
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;FALSE&lt;/code&gt;, we would end up crediting judges for rules we never actually saw them follow.&lt;/p&gt;

&lt;p&gt;Instead, an annotation can be marked as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt;. In the solver, an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt; atom
asserts &lt;strong&gt;nothing&lt;/strong&gt;: its value is left completely unconstrained. When the solver
produces a witness, it fills in these unpinned atoms with arbitrary values consistent with
the facts, and the final report explicitly flags those completions as
possibilities rather than observations.&lt;/p&gt;

&lt;p&gt;This gives us three outcomes from a standard SAT/SMT solver. A claim is entailed
if it holds however the unknown atoms are filled in, refuted if it fails in
every completion, and underdetermined if it holds in some completions and fails
in others. This is called
&lt;a href=&quot;https://www.jstor.org/stable/2024549&quot;&gt;&lt;em&gt;supervaluation&lt;/em&gt;&lt;/a&gt;: we consider all
completions of the partial assignment. For a decision over two atoms where one
is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt;, the results are:&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;Expression&lt;/th&gt;
      &lt;th&gt;Known atom&lt;/th&gt;
      &lt;th&gt;Other atom&lt;/th&gt;
      &lt;th&gt;Result&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;A or B&lt;/td&gt;
      &lt;td&gt;A true&lt;/td&gt;
      &lt;td&gt;unknown&lt;/td&gt;
      &lt;td&gt;entailed&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;A and B&lt;/td&gt;
      &lt;td&gt;A true&lt;/td&gt;
      &lt;td&gt;unknown&lt;/td&gt;
      &lt;td&gt;underdetermined&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;A and B&lt;/td&gt;
      &lt;td&gt;A false&lt;/td&gt;
      &lt;td&gt;unknown&lt;/td&gt;
      &lt;td&gt;refuted&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;A implies B&lt;/td&gt;
      &lt;td&gt;A false&lt;/td&gt;
      &lt;td&gt;unknown&lt;/td&gt;
      &lt;td&gt;entailed&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;A or B&lt;/td&gt;
      &lt;td&gt;A false&lt;/td&gt;
      &lt;td&gt;unknown&lt;/td&gt;
      &lt;td&gt;underdetermined&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;As expected, a false conjunct or a true disjunct settles the outcome regardless of
the unknown value, while everything else remains underdetermined.&lt;/p&gt;

&lt;p&gt;This is not Kleene’s three-valued logic. If \(A\) is unknown, Kleene’s logic calls
\(A\vee\neg A\) unknown, while here it is entailed, because it holds however \(A\)
is filled in. Because intuition around three-valued logic is easy to get subtly
wrong, I checked this behavior against truth tables, over every assignment, for each of 150
randomly generated formulas.&lt;/p&gt;

&lt;h2 id=&quot;principle-2-check-satisfiability-before-entailment&quot;&gt;Principle 2: check satisfiability before entailment&lt;/h2&gt;

&lt;p&gt;Using a solver to check whether a conclusion follows from a set of facts comes with a
classic trap: if those facts are contradictory, &lt;em&gt;everything&lt;/em&gt; follows from them
(the principle of explosion). A checker that simply reports “entailed” in that
situation is technically, and uselessly, correct: the entailment is vacuous.&lt;/p&gt;

&lt;p&gt;To avoid this, Unmask first verifies that the underlying base of facts is actually
consistent before checking for entailment. And rather than testing a single combined
base, it builds three separate ones:&lt;/p&gt;

\[B_{\text{proto}}=K\wedge(V\Leftrightarrow D),\qquad
B_{\text{evid}}=F,\qquad
B_{\text{all}}=B_{\text{proto}}\wedge F.\]

&lt;p&gt;Here \(K\) are the protocol’s constraints, \(D\) is the decision expression, \(V\)
is the verdict variable, and \(F\) is the conjunction of the facts from non-UNKNOWN annotations.
Each annotation \(n\in\mathcal{N}\) names an atom, \(\mathrm{atom}(n)\), whose
propositional variable is \(x_{\mathrm{atom}(n)}\), and carries an annotated value,
\(\mathrm{value}(n)\):&lt;/p&gt;

\[F=\bigwedge_{\substack{n\in\mathcal{N}\\ \mathrm{value}(n)\neq\mathrm{UNKNOWN}}}\bigl(x_{\mathrm{atom}(n)}\Leftrightarrow[\mathrm{value}(n)=\mathrm{TRUE}]\bigr).\]

&lt;p&gt;Here \(n\) ranges over annotation entries, so duplicate annotations of one atom are
possible. Each fact is the literal \(x_{\mathrm{atom}(n)}\) or its negation, so a
consistent \(F\) is a partial assignment, and an inconsistent one means an atom
was annotated with both values.&lt;/p&gt;

&lt;p&gt;Why three? Because each one pinpoints a different kind of failure. If
\(B_{\text{proto}}\) is unsatisfiable, the protocol contradicts itself. If
\(B_{\text{evid}}\) is unsatisfiable, the annotations contradict each other (since
each fact is a literal, this happens only when one atom is annotated with two
different values). And if
both are fine on their own, yet \(B_{\text{all}}\) is unsatisfiable, the evidence
conflicts with the protocol. Separating them lets the report distinguish between
“your protocol is broken,” “your annotations are broken,” and “your judge’s claims
violate the protocol.”&lt;/p&gt;

&lt;p&gt;Keeping these bases separate also lets us run the requirement and grounding checks
against \(B_{\text{evid}}\) alone, ensuring that a contradictory protocol can never
poison what the evidence directly shows.&lt;/p&gt;

&lt;h2 id=&quot;principle-3-two-queries-classify-any-claim&quot;&gt;Principle 3: two queries classify any claim&lt;/h2&gt;

&lt;p&gt;Given a satisfiable base \(B\) and a proposition \(\varphi\), two satisfiability
checks classify it:&lt;/p&gt;

\[\mathrm{status}(\varphi)=\begin{cases}
\text{ENTAILED} &amp;amp; B\wedge\neg\varphi\ \text{is unsatisfiable},\\
\text{REFUTED} &amp;amp; B\wedge\varphi\ \text{is unsatisfiable},\\
\text{UNDERDETERMINED} &amp;amp; \text{otherwise.}
\end{cases}\]

&lt;p&gt;A fourth outcome, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SOLVER_UNKNOWN&lt;/code&gt;, handles any &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;unknown&lt;/code&gt; answer from the solver
(a timeout is one reason for it), ensuring the status is never silently guessed. It is
distinct from an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt; annotation, which is an input rather than a solver result.&lt;/p&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNDERDETERMINED&lt;/code&gt; means the facts permit both \(\varphi\) and its negation;
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;REFUTED&lt;/code&gt; means the base entails \(\neg\varphi\). A verdict can fit a rationale
without being &lt;em&gt;forced&lt;/em&gt; by it, and the report keeps those apart.&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-algorithm&quot;&gt;\begin{algorithm}
\caption{Classify a proposition against a base}
\begin{algorithmic}
\PROCEDURE{Classify}{$B, \varphi$}
    \IF{$B$ is unsatisfiable}
        \RETURN inconsistent base
    \ENDIF
    \STATE $t \gets$ \CALL{Check}{$B \wedge \varphi$}
    \STATE $f \gets$ \CALL{Check}{$B \wedge \neg\varphi$}
    \IF{$t$ or $f$ returned unknown}
        \RETURN solver unknown
    \ELIF{$f$ is unsatisfiable}
        \RETURN entailed
    \ELIF{$t$ is unsatisfiable}
        \RETURN refuted
    \ELSE
        \RETURN underdetermined, with a witness for each side
    \ENDIF
\ENDPROCEDURE
\end{algorithmic}
\end{algorithm}
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&quot;principle-4-keep-the-dimensions-apart-and-aggregate-cautiously&quot;&gt;Principle 4: keep the dimensions apart, and aggregate cautiously&lt;/h2&gt;

&lt;p&gt;Because a judge can fail along several independent axes, collapsing everything into a
single score hides what actually went wrong. Instead, Unmask reports four dimensions
separately:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Verdict entailment&lt;/strong&gt;: does the stated rationale force, merely allow, or
contradict the observed verdict?&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Procedural compliance&lt;/strong&gt;: was each requirement actually met?&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Coverage&lt;/strong&gt;: was the requirement observable in the first place? A requirement
that no one can see being satisfied is very different from one that was actively
violated.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Grounding&lt;/strong&gt;: do the judge’s claims match independent observations of the
context?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Of these, I find coverage the most useful in practice. “We couldn’t tell whether the
judge used outside knowledge” and “the judge used outside knowledge” are
different findings, and an audit shouldn’t flatten both into a pass or
both into a failure. In the implementation, an undetermined result stays
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt;. A requirement whose coverage is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;REFUTED&lt;/code&gt;, meaning the evidence
establishes that it was not observable, is reported as a violation
(&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;missing_observable_coverage&lt;/code&gt;) and drives the overall result to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;VIOLATED&lt;/code&gt;, unless an
inconsistent base or any solver &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;unknown&lt;/code&gt; result overrides the aggregation (below). The
rule is that you cannot be credited for a requirement nobody could observe, not
that you are accused of breaking it. The report keeps that violation separate
from a breach of the requirement itself.&lt;/p&gt;

&lt;p&gt;When it comes time to roll these individual checks up into an overall result, Unmask
uses a deliberately conservative rule. Given statuses \(s_1,\dots,s_k\):&lt;/p&gt;

\[\mathrm{agg}(s_1,\dots,s_k)=\begin{cases}
\text{VIOLATED} &amp;amp; \exists i:\ s_i=\text{REFUTED},\\
\text{SATISFIED} &amp;amp; k&amp;gt;0\ \wedge\ \forall i:\ s_i=\text{ENTAILED},\\
\text{UNKNOWN} &amp;amp; \text{otherwise.}
\end{cases}\]

&lt;p&gt;Two further rules apply. First, if a replay supplies
no requirements or no grounding pairs, the overall result can never be &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SATISFIED&lt;/code&gt;: you
don’t get credit for a dimension that was never checked. Second, any base
inconsistency or any solver &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;unknown&lt;/code&gt; result automatically forces the overall result to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt;.
Note that a requirement without a coverage expression reports coverage as
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;NOT_SPECIFIED&lt;/code&gt;, which is not &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ENTAILED&lt;/code&gt;, so it also keeps the overall result from
being &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SATISFIED&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The whole procedure, in one place:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-algorithm&quot;&gt;\begin{algorithm}
\caption{Check protocol adherence of a replay}
\begin{algorithmic}
\REQUIRE Constraints $K$, decision $D$, requirements $Q$ (each with expression $q$ and optional coverage $a$), annotations $\mathcal{N}$, observed verdict $v$, grounding pairs $\Pi$
\ENSURE A status, a result per dimension, and witnesses
\IF{translation status is not valid}
    \RETURN overall UNKNOWN
\ENDIF
\STATE $F \gets \bigwedge_{n \in \mathcal{N},\ \mathrm{value}(n) \neq \mathrm{UNKNOWN}} \bigl(x_{\mathrm{atom}(n)} \Leftrightarrow [\mathrm{value}(n) = \mathrm{TRUE}]\bigr)$
\STATE $B_{\mathrm{proto}} \gets K \wedge (V \Leftrightarrow D)$
\STATE $B_{\mathrm{evid}} \gets F$
\STATE $B_{\mathrm{all}} \gets B_{\mathrm{proto}} \wedge F$
\STATE $\varphi_v \gets V$ if $v = \mathrm{PASS}$, otherwise $\neg V$
\STATE $s_{\mathrm{verdict}} \gets$ \CALL{Classify}{$B_{\mathrm{all}}, \varphi_v$}
\FORALL{$(q, a) \in Q$}
    \STATE $s_q \gets$ \CALL{Classify}{$B_{\mathrm{evid}}, q$}
    \STATE $s_a \gets$ \CALL{Classify}{$B_{\mathrm{evid}}, a$} if $a$ is given, otherwise not specified
\ENDFOR
\FORALL{$(j, c) \in \Pi$}
    \STATE $s_{jc} \gets$ \CALL{Classify}{$B_{\mathrm{evid}}, x_j \Leftrightarrow x_c$}
\ENDFOR
\STATE $S \gets$ all statuses, plus $\mathrm{UNKNOWN}$ if $Q = \emptyset$ or $\Pi = \emptyset$
\STATE $\mathit{overall} \gets \mathrm{agg}(S)$
\IF{$B_{\mathrm{proto}}$, $B_{\mathrm{evid}}$ or $B_{\mathrm{all}}$ is inconsistent, or any query returned unknown}
    \STATE $\mathit{overall} \gets \mathrm{UNKNOWN}$ and report which base failed
\ENDIF
\RETURN statuses, $\mathit{overall}$, and the violated checks with their evidence
\end{algorithmic}
\end{algorithm}
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&quot;two-things-that-never-enter-the-solver&quot;&gt;Two things that never enter the solver&lt;/h2&gt;

&lt;p&gt;Two validation rules apply before the solver even runs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The gold label stays out.&lt;/strong&gt; While the final report notes whether the verdict
matches the gold label, that label never enters the solver. In fact, there is an
explicit test for this: if you flip the gold label, the resulting report must
remain identical apart from the two gold-derived fields. If the gold label
leaked into the solver, an audit could end up rationalizing an answer it had
already been given.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evidence must be a real quote.&lt;/strong&gt; Every non-&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNKNOWN&lt;/code&gt; annotation must point to
an exact Unicode code-point span in the source text, and the quoted string must
match that span verbatim. The validator also rejects spans taken from the wrong
source for a given atom’s role. For instance, you cannot quote the protocol text
as evidence that the judge actually performed a step. A span check
verifies that a &lt;em&gt;quotation&lt;/em&gt; is real, not that it &lt;em&gt;means&lt;/em&gt; what the annotator claims
it means.&lt;/p&gt;

&lt;h2 id=&quot;how-is-this-different-from-verifying-instruction-following&quot;&gt;How is this different from verifying instruction following?&lt;/h2&gt;

&lt;p&gt;A closely related piece of recent work is “Neuro-Symbolic Verification on
Instruction Following of LLMs” (NSVIF, Su et al., &lt;a href=&quot;https://arxiv.org/abs/2601.17789&quot;&gt;arXiv:2601.17789&lt;/a&gt;). Like Unmask, NSVIF
uses Z3, separates logical structure from semantics, and aims to produce
interpretable feedback. It frames instruction following as a
constraint-satisfaction problem: a model extracts constraints from the
instruction and writes a Z3 program to compose them, using model-generated code
to check logical constraints and an LLM to judge semantic ones before Z3
combines everything into a final &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SAT&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UNSAT&lt;/code&gt; result.&lt;/p&gt;

&lt;p&gt;Despite the shared machinery, the two systems ask different
questions:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;NSVIF&lt;/strong&gt; asks: &lt;em&gt;did this output follow the instruction it was given?&lt;/em&gt;&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Unmask&lt;/strong&gt; asks: &lt;em&gt;did this judge follow the protocol it was meant to apply when evaluating something?&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Because the questions differ, the inputs do too. In NSVIF, the subject under
test is a single model output, and the constraints come directly from a prompt.
Here, the subject is an entire evaluation process, and what matters is the
three-way relationship between the judge’s stated claims, an independent reading
of the source material, and the final verdict.&lt;/p&gt;

&lt;p&gt;They also differ in where the semantic judgment lives. NSVIF automates both
constraint extraction and semantic evaluation: a model judges each semantic
constraint, and any violation is explained through that model’s response and
reasoning trace. Unmask, by contrast, turns each semantic judgment into an
explicit, annotated fact backed by an exact evidence span. This trades
automation for inspectability: you can see the exact semantic claim a conclusion
rests on, and the final report makes it clear that its conclusions are
conditional on those annotations.&lt;/p&gt;

&lt;p&gt;Because of this, I see the two approaches as complementary rather than
competing. A natural next step would be to use an automated semantic checker
like NSVIF’s to &lt;em&gt;produce&lt;/em&gt; the annotations that Unmask then reasons over. For
now, though, that remains a hypothesis: the only automation I have built so far
is an experimental, model-assisted proposer for a single fixed protocol, and I
have not yet measured its accuracy.&lt;/p&gt;

&lt;h2 id=&quot;what-this-does-not-do&quot;&gt;What this does not do&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;It does not read prose on its own.&lt;/strong&gt; Both the protocol translation and the
annotations must be supplied upfront. For one fixed protocol (factual claims),
an experimental assistant can use a model to propose annotations, though these
are flagged as unreviewed model output and I have not yet measured their
accuracy. Any other protocol still has to be translated by hand.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;The guarantee is conditional.&lt;/strong&gt; Every conclusion holds only relative to the
supplied formalization and annotations. Z3 decides satisfiability of the structured
encoding rather than the raw text or any hidden reasoning the judge may
have done.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Hand annotation does not scale.&lt;/strong&gt; The fixtures in this post were annotated by
hand. While the experimental assistant helps with one specific protocol, the
broader value of this design depends on getting reliable annotations cheaply,
which remains an open problem.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;There is no empirical evaluation yet.&lt;/strong&gt; While the four test cases show that
the checker behaves as designed, I have not yet measured how well annotated
replays catch real-world judge failures in the wild.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;what-the-two-posts-share&quot;&gt;What the two posts share&lt;/h2&gt;

&lt;p&gt;Both checks follow one pattern. An interpretation is supplied: a propositional
encoding of the policy in the last post, annotated facts here. A solver decides what
follows from it, and the report says its conclusions hold relative to that
interpretation. A language model may help produce the interpretation, but it never
issues the verdict.&lt;/p&gt;

&lt;h2 id=&quot;the-question-i-think-comes-next&quot;&gt;The question I think comes next&lt;/h2&gt;

&lt;p&gt;What I find most interesting about this framing is the separation itself.
Correctness, procedure, coverage, and grounding are distinct properties, yet a
gold-label evaluation only ever sees the first.&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;If we audited our judges the way we audit our code, which of those four
  dimensions would we see failing first?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;My guess is grounding, a failure mode that remains largely invisible to the
metrics we rely on today. That is a hypothesis, not a finding.&lt;/p&gt;
</content>
 </entry>
 
 
 
 <entry>
   <title>How Do You Localize a Fault in a Guardrail?</title>
   <link href="http://huascarsanchez.com/posts/notes/how-do-you-localize-a-fault-in-a-guardrail.html"/>
   <updated>2026-10-16T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/how-do-you-localize-a-fault-in-a-guardrail</id>
   <content type="html">&lt;h2 id=&quot;when-a-guardrail-misbehaves-where-is-the-bug&quot;&gt;When a guardrail misbehaves, where is the bug?&lt;/h2&gt;

&lt;p&gt;A guardrail is a small program written in a language nobody compiles. It might be
five sentences in a system prompt, or a handful of regexes in a config file. It
pretty much says what an agent may and may not do.&lt;/p&gt;

&lt;p&gt;And like any program, it has bugs. The symptoms are familiar:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;It blocks something perfectly harmless.&lt;/li&gt;
  &lt;li&gt;Two rules conflict: they apply to the same request and demand opposite decisions.&lt;/li&gt;
  &lt;li&gt;A rule can be read two ways, and the two readings behave differently.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When code misbehaves, we ask a &lt;em&gt;fault localization&lt;/em&gt; question: which statement is
responsible? That question has decades of tooling behind it. For an agent guardrail, we
mostly have a shrug and a pile of test prompts, friendly and adversarial alike.&lt;/p&gt;

&lt;p&gt;The goal is the same, though: find a defect and point to the rules involved, so
that a person has somewhere to start.&lt;/p&gt;

&lt;p&gt;I’ll argue that this problem is a clean example of something that needs both a
language model and a solver, each doing a different job. This post is about that
division of labor: what each half is good for, what it cannot be trusted with,
and where the handoff between them should sit.&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Let a language model read the policy, but let a solver decide whether it is broken, and make the solver hand back a counterexample.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;People call this shape &lt;em&gt;neuro-symbolic&lt;/em&gt;: a neural component where the problem is
language, and a symbolic component where the problem is proof. The phrase is easy to
say and easy to over-claim, so I’ll be concrete about what it buys here.&lt;/p&gt;

&lt;h2 id=&quot;why-cant-a-language-model-just-tell-me&quot;&gt;Why can’t a language model just tell me?&lt;/h2&gt;

&lt;p&gt;The obvious first attempt is to hand the policy to an LLM and ask, “Is anything
wrong with these rules?”&lt;/p&gt;

&lt;p&gt;Unsurprisingly, the LLM will happily answer. It might say that rule 1 “seems too
broad and could block harmless requests,” or that rules 2 and 3 “may conflict in
edge cases.” Both are plausible, and both are close to unfalsifiable. Which
harmless requests, you might ask? Which edge cases? Do the claims hold for every
phrasing, or just the one the model happened to imagine? Ask again five minutes
later and you may get a different list.&lt;/p&gt;

&lt;p&gt;LLMs are great at reading a policy and raising plausible concerns. They are not
good at being exhaustive, and they can’t hand you a concrete input that shows
the problem is real, a counterexample. Fault localization needs both: some
assurance that nothing was missed, and an input that proves each finding.&lt;/p&gt;

&lt;h2 id=&quot;why-cant-a-solver-just-tell-me&quot;&gt;Why can’t a solver just tell me?&lt;/h2&gt;

&lt;p&gt;The opposite attempt is to use formal methods and reach for a solver. SAT
solvers are excellent at this kind of question: given a set of rules, does some
input make two of them disagree?&lt;/p&gt;

&lt;p&gt;The catch is that a solver needs the rules as logic, and your guardrail is
prose. A solver can’t read “block personalized investment recommendations for
minors and first-time investors.” It doesn’t know what “personalized” means, or
whether “minors and first-time investors” means both groups or either one. A
solver is exact but blind to language, so until someone translates the prose
into logic, there is nothing for it to check.&lt;/p&gt;

&lt;h2 id=&quot;so-what-does-each-half-do&quot;&gt;So what does each half do?&lt;/h2&gt;

&lt;p&gt;Each half is strong exactly where the other is weak, so it’s clear who should do
what. Side by side:&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt; &lt;/th&gt;
      &lt;th&gt;Neural half&lt;/th&gt;
      &lt;th&gt;Symbolic half&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;Works on&lt;/td&gt;
      &lt;td&gt;Prose: rules, intent, examples&lt;/td&gt;
      &lt;td&gt;Logic: each rule as a propositional formula (an AND/OR condition over yes/no predicates)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Job&lt;/td&gt;
      &lt;td&gt;Read and interpret; propose readings&lt;/td&gt;
      &lt;td&gt;Decide satisfiability; return a counterexample&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Strength&lt;/td&gt;
      &lt;td&gt;Handles language: “substantial,” “personalized”&lt;/td&gt;
      &lt;td&gt;Sound and complete with respect to the encoding: every reported fault is real in it, and if a fault exists in it, the solver finds it&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Not trusted for&lt;/td&gt;
      &lt;td&gt;Exactness, or giving the same answer twice&lt;/td&gt;
      &lt;td&gt;Anything outside the facts it was given&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;The one rule that holds the design together is a restriction on the neural half:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The language model proposes an interpretation. It never issues the verdict.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The language model does not say “this rule is over-strict.” It supplies a
propositional encoding of the policy, and a decision procedure it does not control
turns that encoding into a conclusion. For example, that procedure can ask whether any
request \(x\) is both safe and blocked: \(\mathrm{safe}(x) \wedge
\mathrm{blocked}(x)\). The propositional encoding is the handoff, and everything else in
this post is about how to make it narrow enough to inspect.&lt;/p&gt;

&lt;p&gt;Here is what it looks like end to end:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart LR
    accTitle: Neuro-symbolic fault localization
    accDescr: A guardrail is read by a neural component that produces a propositional encoding. That encoding is the handoff. A symbolic component then answers three satisfiability questions and returns findings with counterexamples.
    G[&quot;Guardrail&amp;lt;br/&amp;gt;prose or regex&quot;] --&amp;gt; N
    subgraph NEURAL[&quot;Neural: interpret&quot;]
        N[&quot;Extract predicates,&amp;lt;br/&amp;gt;rules, intent, readings&quot;]
    end
    N --&amp;gt; M[&quot;Propositional encoding&amp;lt;br/&amp;gt;(the handoff)&quot;]
    M --&amp;gt; Z
    subgraph SYMBOLIC[&quot;Symbolic: decide&quot;]
        Z[&quot;Three satisfiability&amp;lt;br/&amp;gt;questions&quot;]
    end
    Z --&amp;gt; F[&quot;Findings with counterexamples&quot;]
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&quot;what-counts-as-a-fault-in-a-guardrail&quot;&gt;What counts as a fault in a guardrail?&lt;/h2&gt;

&lt;p&gt;Before the handoff, let’s look at what the symbolic half is actually being asked
to find. Here is a small financial-information policy, with four rules in plain English:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Block any response containing the words &lt;em&gt;buy&lt;/em&gt;, &lt;em&gt;sell&lt;/em&gt;, or &lt;em&gt;invest&lt;/em&gt;.&lt;/li&gt;
  &lt;li&gt;Allow general financial education that does not recommend a specific transaction.&lt;/li&gt;
  &lt;li&gt;Block instructions explaining how to trade a specific investment.&lt;/li&gt;
  &lt;li&gt;Block personalized investment recommendations for minors and first-time investors.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Three kinds of fault hide in there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Over-strict.&lt;/strong&gt; Rule 1 blocks “Google will invest up to $2 billion.” That is
neutral reporting about someone else’s action, and the policy’s own intent says it
should be allowed. The rule is broader than the policy it serves.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Conflict.&lt;/strong&gt; Rule 2 allows “how to” education and rule 3 blocks instructions
to trade. For a request like “explain how to trade shares,” both apply, and they
demand opposite outcomes. Nothing in the policy says which wins.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ambiguity.&lt;/strong&gt; Rule 4 says “minors and first-time investors.” Does that mean
either group, or only people who are both? The two readings handle a 30-year-old
first-time investor differently.&lt;/p&gt;

&lt;p&gt;These are different problems. An over-strict rule blocks something the policy
was meant to allow. Conflicting rules apply to the same request and demand opposite
decisions, with no way to choose between them. Whether that also makes them
logically inconsistent depends on how their effects are encoded. Ambiguity means
two readings of one rule behave differently. Each problem needs a different check,
and that shapes what the neural half must produce.&lt;/p&gt;

&lt;h2 id=&quot;the-handoff-a-model-a-human-can-audit&quot;&gt;The handoff: a model a human can audit&lt;/h2&gt;

&lt;p&gt;The handoff is the most important design decision, because it is the only place
where the neural half’s output turns into something the symbolic half trusts.&lt;/p&gt;

&lt;p&gt;I want that object to be narrow, and I want a person to be able to read it. So
the language model reads the rules, the system prompt, and any labeled examples,
and returns a small propositional encoding of the policy:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Predicates&lt;/strong&gt;: named yes/no facts about a request, each with a precise
description, such as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;recommends_transaction&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;is_neutral_report&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Rule conditions&lt;/strong&gt;: for each rule, a decision (allow or block) and a condition
over predicates, written in disjunctive normal form. The outer list is OR, each
inner list is AND.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;A safe condition&lt;/strong&gt;: the set of requests that are legitimate, described
&lt;em&gt;independently&lt;/em&gt; of whether any written rule blocks them.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Exclusions&lt;/strong&gt;: groups of mutually exclusive predicates, of which at most one can
be true. These are domain constraints: they rule out combinations that cannot occur.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Ambiguities&lt;/strong&gt;: for each unclear rule, two distinct, plausible formalizations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In symbols, with predicates \(P=\{p_1,\dots,p_n\}\), each rule \(r\) has a decision
\(d_r\in\{\mathrm{allow},\mathrm{block}\}\) and a condition&lt;/p&gt;

\[C_r=\bigvee_{i}\bigwedge_{j}\ell_{ij},\]

&lt;p&gt;where each literal \(\ell_{ij}\) is a predicate or its negation. The safe condition
\(S\) has the same shape. The exclusions form a set \(E\) of groups of predicates,
and they become a global at-most-one constraint: no two distinct predicates \(p,q\) in the same group
\(e\) can both be true,&lt;/p&gt;

\[\Gamma=\bigwedge_{\{p,q\}\subseteq e,\ e\in E}\neg(p\wedge q).\]

&lt;p&gt;That is the whole object, small enough to read in one sitting. Three properties
of it are deliberate.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It is descriptive.&lt;/strong&gt; Every predicate carries a sentence saying what it means. If
the language model mistranslates a rule, the mistake is visible as a predicate you can
disagree with, not buried in a score.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It is shaped by the questions.&lt;/strong&gt; The safe condition and the paired readings are in
the encoding because the symbolic half needs them to ask over-strictness and ambiguity
questions. The neural half is told what the symbolic half will ask. That is the
reverse of the usual “extract everything and see.”&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It is the trust boundary, and I should say so plainly.&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The proofs are about the extracted model, not about the original prose.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If the language model mistranslates a rule, the solver will faithfully prove things
about the wrong model. The design does not remove that risk. It makes the risk
inspectable and keeps it confined to one place.&lt;/p&gt;

&lt;h2 id=&quot;three-faults-three-satisfiability-questions&quot;&gt;Three faults, three satisfiability questions&lt;/h2&gt;

&lt;p&gt;Once the policy is a propositional encoding, each fault becomes a question of the form
&lt;em&gt;does an assignment exist such that…?&lt;/em&gt; Below, \(x\) ranges over assignments of
true or false to the predicates in \(P\), which stand for the possible requests as
the encoding sees them, and \(C(x)\) is the value of a condition \(C\) under \(x\).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Over-strict.&lt;/strong&gt; A blocking rule \(r\) is over-strict if there is some legitimate
request that it blocks. This is the safe-and-blocked question from earlier, asked
about one rule at a time. Equivalently, it is the negation of the property “no
legitimate request is blocked by \(r\)”:&lt;/p&gt;

\[\exists x:\ \Gamma(x)\wedge S(x)\wedge C_r(x).\]

&lt;p&gt;&lt;strong&gt;Conflict.&lt;/strong&gt; Two rules \(r\) and \(r&apos;\) with different decisions conflict
if some request satisfies both conditions:&lt;/p&gt;

\[\exists x:\ \Gamma(x)\wedge C_r(x)\wedge C_{r&apos;}(x)\qquad\text{for } d_r\neq d_{r&apos;}.\]

&lt;p&gt;&lt;strong&gt;Ambiguity.&lt;/strong&gt; Suppose a rule has two candidate formalizations, which I call
readings, \(A\) and \(B\), with conditions \(C_A,C_B\) and decisions \(d_A,d_B\).
The readings diverge on a request when \(\Gamma\wedge\Delta\) is satisfiable, where&lt;/p&gt;

\[\Delta=\begin{cases}
C_A\oplus C_B &amp;amp; \text{if } d_A=d_B,\\
C_A\vee C_B &amp;amp; \text{if } d_A\neq d_B,
\end{cases}\]

&lt;p&gt;Here is why \(\Delta\) splits into two cases. When both readings reach the same
decision, they disagree only when one applies and the other does not (exclusive-or).
Checking satisfiability here tests whether the two formalizations are equivalent
modulo \(\Gamma\). When they reach different decisions, any request covered by
either reading is a disagreement: either both apply and conflict, or only one
applies while the other leaves the request undecided.&lt;/p&gt;

&lt;p&gt;This assumes no default decision, treating an uncovered request as its own
“undecided” outcome. If the policy instead has a default \(d_0\) (say, allow), two
allow readings with different conditions would still produce the same outcome on
every request, making the exclusive-or a false alarm. To handle a default, the
general form defines each reading’s decision function&lt;/p&gt;

\[f_A(x)=\begin{cases}d_A &amp;amp; C_A(x),\\ d_0 &amp;amp; \neg C_A(x),\end{cases}
\qquad
f_B(x)=\begin{cases}d_B &amp;amp; C_B(x),\\ d_0 &amp;amp; \neg C_B(x),\end{cases}\]

&lt;p&gt;and asks whether \(\Gamma\wedge(f_A\neq f_B)\) is satisfiable. This compares the two
readings of the rule in isolation. In a full policy, other rules and the policy’s
combining strategy (XACML specifies several) can mask a difference between them,
so a divergence here is a divergence of the rule, not necessarily of the policy.&lt;/p&gt;

&lt;p&gt;In each case the solver answers SAT or UNSAT. If the formula is UNSAT, no request
can trigger that fault, at least within the extracted encoding. If it is
SAT, the solver returns a satisfying assignment, which we call a &lt;strong&gt;witness&lt;/strong&gt;: a
concrete class of requests that demonstrates the fault. For over-strictness the
witness is a counterexample to the property “no legitimate request is blocked”;
for a conflict it is a witness of overlap. Either way it goes into the finding,
which is why a finding is evidence and not an opinion.&lt;/p&gt;

&lt;p&gt;One caution: the witness is an assignment in the encoding. If the encoding is
missing a constraint, no real request may realize that assignment, the same way an
abstract counterexample in verification can be spurious. Reading the witness as a
sentence, as I do below, may help reveal an unrealizable assignment, but informal
inspection does not establish that a witness is real. &lt;a href=&quot;https://doi.org/10.1007/10722167_15&quot;&gt;Counterexample-guided
abstraction refinement&lt;/a&gt; checks this mechanically and refines the abstraction; this
post’s pipeline does not do that.&lt;/p&gt;

&lt;p&gt;Here is a tiny model, loosely based on rules 1 and 2 above. I wrote it by hand,
the way an extractor would return it, with two rules and one exclusion:&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;Predicate&lt;/th&gt;
      &lt;th&gt;Meaning&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;mentions_invest_word&lt;/code&gt;&lt;/td&gt;
      &lt;td&gt;the response contains buy, sell, or invest&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;recommends_transaction&lt;/code&gt;&lt;/td&gt;
      &lt;td&gt;the response recommends a specific transaction&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;is_neutral_report&lt;/code&gt;&lt;/td&gt;
      &lt;td&gt;the response neutrally reports a third-party action&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;Rule 1 blocks when &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;mentions_invest_word&lt;/code&gt; holds. Rule 2 allows when
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;is_neutral_report&lt;/code&gt; holds and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;recommends_transaction&lt;/code&gt; does not. The exclusion says
a neutral report cannot also be a recommendation. The safe condition is simply
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;is_neutral_report&lt;/code&gt;. Running the analyzer on this encoding, the solver returns the
same witness for both questions (the tool labels the conflict finding
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;contradiction&lt;/code&gt;; I say “conflict” in prose):&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;contradiction (rule:1, rule:2)  {mentions_invest_word: True,
                                 recommends_transaction: False,
                                 is_neutral_report: True}
over_strict   (rule:1)          {mentions_invest_word: True,
                                 recommends_transaction: False,
                                 is_neutral_report: True}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Read the witness as a sentence: &lt;em&gt;a neutral report that happens to mention the word
“invest.”&lt;/em&gt; That is exactly the request rule 1 should not block, and exactly where
rules 1 and 2 collide. You did not have to guess which prompt would trigger the
bug. The solver found it.&lt;/p&gt;

&lt;h2 id=&quot;what-does-the-neural-half-owe-the-symbolic-half&quot;&gt;What does the neural half owe the symbolic half?&lt;/h2&gt;

&lt;p&gt;The three questions are not symmetric in what they demand from the neural half, and
that asymmetry is where the interesting design lives.&lt;/p&gt;

&lt;p&gt;Conflict detection needs only the rules. You can find two conflicting rules without
knowing anything about what the policy is &lt;em&gt;for&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Over-strictness is different. To say a rule blocks something it should not, you
need a notion of “should not” that is separate from the rules themselves. If the
safe set were derived by reading the rules, every rule would trivially agree with
itself, and you would never find an over-strict one.&lt;/p&gt;

&lt;p&gt;So the neural half has to produce the safe condition as its own object: requests that
are semantically legitimate, described independently of whether a written rule
blocks them. The over-strict query then asks whether that independent region overlaps
a blocking rule.&lt;/p&gt;

&lt;p&gt;This is also where extraction quality matters most. A sloppy safe region produces
false alarms or misses real ones. That is the price of separating intent from rules,
and I think it is worth paying, because the alternative cannot find this fault at
all.&lt;/p&gt;

&lt;h2 id=&quot;what-changes-when-the-facts-are-executed-instead-of-inferred&quot;&gt;What changes when the facts are executed instead of inferred?&lt;/h2&gt;

&lt;p&gt;Regex guardrails are an instructive contrast, because nothing needs to be guessed. A
regex either matches a piece of text or it does not.&lt;/p&gt;

&lt;p&gt;For a regex manifest, the same symbolic half runs, but the predicates come from
execution instead of from a language model:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;rule_k_matches_incident&lt;/code&gt; is &lt;strong&gt;pinned&lt;/strong&gt; by actually running rule \(k\)’s pattern
against the text, in an isolated worker process with a match timeout.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;incident_expected_allow&lt;/code&gt; is pinned by the label on the example.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those pinned assignments are added to the solver (I use Z3, an SMT solver, here
purely as a SAT engine, since everything is propositional) as unit constraints,
and then the very same analyzer runs. The same finding types come out, now
carrying exact match spans as evidence.&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt; &lt;/th&gt;
      &lt;th&gt;Natural-language rules&lt;/th&gt;
      &lt;th&gt;Regex rules&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;Where predicates come from&lt;/td&gt;
      &lt;td&gt;Neural extraction&lt;/td&gt;
      &lt;td&gt;Exact regex execution&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;How far to trust them&lt;/td&gt;
      &lt;td&gt;Conditional on faithfulness&lt;/td&gt;
      &lt;td&gt;Exact for the supplied text&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Role of the solver&lt;/td&gt;
      &lt;td&gt;Search for a witness&lt;/td&gt;
      &lt;td&gt;Evaluate pinned facts&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Evidence&lt;/td&gt;
      &lt;td&gt;Rule text and rationale&lt;/td&gt;
      &lt;td&gt;Match spans and patterns&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;This is a useful test of the design. The symbolic half does not care where its
facts came from. Swap inferred facts for executed ones, and nothing about the
questions or the findings changes. The neural half is one possible source of
facts, not a dependency of the reasoning.&lt;/p&gt;

&lt;p&gt;In the regex case, the solver is doing less than it might appear. When every
predicate is pinned, there is nothing left to search for; Z3 is evaluating a
formula, not discovering an assignment. What the shared analyzer buys you is a
single vocabulary of findings, witnesses, and proofs that works whether the
facts were inferred or executed.&lt;/p&gt;

&lt;h2 id=&quot;where-unmask-fits&quot;&gt;Where Unmask fits&lt;/h2&gt;

&lt;p&gt;I built a tool called &lt;a href=&quot;https://github.com/hsanchez/unmask&quot;&gt;Unmask&lt;/a&gt; to try this
out. The findings and JSON below are real output from it. (The tiny model above
is hand-written, but I ran it through the same analyzer.) It is the means here,
not the point; the point is that this division of labor works for the problem.&lt;/p&gt;

&lt;p&gt;On the example policy that ships with Unmask, scanning the regex manifest against six
labeled cases returns three findings, one per fault class:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;block-investment-term&lt;/code&gt; blocks “Google will invest up to $2 billion.” The
pattern matched &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;invest&lt;/code&gt; at characters 15 to 21, while the label says the output
should be allowed.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;allow-how-to-education&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;block-actionable-trading&lt;/code&gt; conflict: they apply to
the same “how to trade” text and demand opposite outcomes.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;first-time-investor-scope&lt;/code&gt; has two supplied readings, “either protected
audience” and “both protected audiences,” that diverge on a first-time-investor
example.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here is the over-strict finding, trimmed from the real output:&lt;/p&gt;

&lt;div class=&quot;language-json highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;type&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;over_strict&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;ruleIds&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;block-investment-term&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;witness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;incident_expected_allow&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;rule_1_matches_incident&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;rule_2_matches_incident&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;rule_3_matches_incident&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;counterexample&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;In the supplied agent output, &apos;Google will invest up to $2 billion.&apos;, the policy label expects allow...&quot;&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;And the whole symbolic half, which is the same for both kinds of guardrail. Here
\(\Gamma&apos;\) is \(\Gamma\) conjoined with the pinned assignment, if there is one, and
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Solve&lt;/code&gt;(\(\varphi\)) returns a satisfying assignment of \(\varphi\), or \(\bot\) if
none exists:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-algorithm&quot;&gt;\begin{algorithm}
\caption{Detect and attribute faults in a propositional encoding of a guardrail}
\begin{algorithmic}
\REQUIRE Extracted by the neural half for prose, or built from the manifest and labels for regex: rules $R$ with decisions $d_r$ and conditions $C_r$; safe condition $S$; exclusion groups $E$; ambiguities $\mathcal{A}$. Optional pinned assignment $\alpha$ (from regex execution)
\ENSURE Findings, each with a witness assignment
\STATE $\Gamma&apos; \gets \bigwedge_{e \in E,\ \{p,q\} \subseteq e} \neg(p \wedge q) \ \wedge\ \alpha$
\STATE $F \gets \emptyset$
\FORALL{$r \in R$ with $d_r = \mathrm{block}$}
    \STATE $w \gets$ \CALL{Solve}{$\Gamma&apos; \wedge S \wedge C_r$}
    \IF{$w \neq \bot$}
        \STATE $F \gets F \cup \{(\mathrm{over\_strict}, r, w)\}$
    \ENDIF
\ENDFOR
\FORALL{unordered pairs $\{r, r&apos;\} \subseteq R$ with $d_r \neq d_{r&apos;}$}
    \STATE $w \gets$ \CALL{Solve}{$\Gamma&apos; \wedge C_r \wedge C_{r&apos;}$}
    \IF{$w \neq \bot$}
        \STATE $F \gets F \cup \{(\mathrm{conflict}, \{r, r&apos;\}, w)\}$
    \ENDIF
\ENDFOR
\FORALL{$a \in \mathcal{A}$ with readings $(C_A, d_A)$ and $(C_B, d_B)$}
    \IF{$d_A = d_B$}
        \STATE $\delta \gets C_A \oplus C_B$
    \ELSE
        \STATE $\delta \gets C_A \vee C_B$
    \ENDIF
    \STATE $w \gets$ \CALL{Solve}{$\Gamma&apos; \wedge \delta$}
    \IF{$w \neq \bot$}
        \STATE $F \gets F \cup \{(\mathrm{ambiguity}, a, w)\}$
    \ENDIF
\ENDFOR
\RETURN $F$ sorted by (type, id)
\end{algorithmic}
\end{algorithm}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Everything interesting is in how the inputs are built, not in the loops. The loops
are three kinds of satisfiability check. The work is making sure those checks are
about the right objects. For prose, that is the neural half’s job and the reason
the handoff has the shape it does.&lt;/p&gt;

&lt;h2 id=&quot;what-do-you-gain-over-either-half-alone&quot;&gt;What do you gain over either half alone?&lt;/h2&gt;

&lt;p&gt;Here is the comparison I would have wanted before starting.&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt; &lt;/th&gt;
      &lt;th&gt;LLM alone&lt;/th&gt;
      &lt;th&gt;Solver alone&lt;/th&gt;
      &lt;th&gt;Neural and symbolic&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;Reads a prose policy&lt;/td&gt;
      &lt;td&gt;Yes&lt;/td&gt;
      &lt;td&gt;No&lt;/td&gt;
      &lt;td&gt;Yes, via the neural half&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Gives a counterexample&lt;/td&gt;
      &lt;td&gt;Not reliably&lt;/td&gt;
      &lt;td&gt;Yes, if the policy is already logic&lt;/td&gt;
      &lt;td&gt;Yes&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Complete over what it considers&lt;/td&gt;
      &lt;td&gt;No&lt;/td&gt;
      &lt;td&gt;Yes&lt;/td&gt;
      &lt;td&gt;Yes, over the extracted predicates (relative to the encoding, not the prose)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Result can be checked&lt;/td&gt;
      &lt;td&gt;Hard&lt;/td&gt;
      &lt;td&gt;Yes&lt;/td&gt;
      &lt;td&gt;Yes; the model is readable&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Typical failure&lt;/td&gt;
      &lt;td&gt;Plausible and wrong&lt;/td&gt;
      &lt;td&gt;Needs a spec you may not have&lt;/td&gt;
      &lt;td&gt;A mistranslation, proved faithfully&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;Combining the halves does not remove the failure modes; it moves them. The
LLM-only failure is plausible and unfalsifiable. The combined failure is a wrong
model that is at least visible and at least confined.&lt;/p&gt;

&lt;p&gt;A counterexample changes what a finding is. An LLM-only report says: &lt;em&gt;“Rule 1 seems too
broad.”&lt;/em&gt; The combined report says: &lt;em&gt;“Rule 1 blocks the request class where
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;is_neutral_report&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;mentions_invest_word&lt;/code&gt; both hold, and your own safe region
contains that class.”&lt;/em&gt; You can inspect the model, disagree with a predicate, or fix
the rule and re-run. The claim is checkable.&lt;/p&gt;

&lt;p&gt;The tool also declines to say some things. For regex guardrails, some analyses
cannot be run on some inputs, and it reports that instead of guessing:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Without an expected outcome or a policy reason for an example, it will not
claim a blocking match is over-strict.&lt;/li&gt;
  &lt;li&gt;Conflict analysis needs both an allow rule and a block rule.&lt;/li&gt;
  &lt;li&gt;Ambiguity needs two explicit policy readings, because regex execution is
deterministic and cannot disagree with itself.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An empty result means every check that could run came back UNSAT, within the
model. It is reported as “no supported defect proven,” which is weaker than “no
defect.”&lt;/p&gt;

&lt;h2 id=&quot;what-this-does-not-do&quot;&gt;What this does not do&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Faithfulness is not proved.&lt;/strong&gt; For natural-language rules, a finding is as
trustworthy as the extraction behind it, and a witness may be spurious if the
encoding omits a constraint. Predicate descriptions and rule
attributions make mistakes visible, but they do not eliminate them.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Completeness is relative to the model.&lt;/strong&gt; The solver can only find faults
expressible in the predicates that were extracted.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Ambiguity needs two formalizable readings.&lt;/strong&gt; If the neural half cannot write
down two materially distinct interpretations, no ambiguity is reported, even if
a human would sense one.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;I have not benchmarked it.&lt;/strong&gt; I can show that the findings are sound relative
to the model, and I can show real examples. I cannot yet tell you detection
rates against injected defects or against an LLM-only baseline.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;where-i-think-the-interesting-part-is&quot;&gt;Where I think the interesting part is&lt;/h2&gt;

&lt;p&gt;None of the individual pieces here are new. Rule sets have been checked for
conflicts for a long time: firewall-policy analysis catalogued anomalies such as
&lt;a href=&quot;https://doi.org/10.1007/978-0-387-35674-7_2&quot;&gt;shadowing, redundancy and
correlation&lt;/a&gt;, access-control
analysis defines rule conflicts, and SMT solvers have been applied to cloud
access policies (&lt;a href=&quot;https://doi.org/10.23919/FMCAD.2018.8602994&quot;&gt;Zelkova at AWS&lt;/a&gt;).
Language models have been used as front ends to logic for a while
(&lt;a href=&quot;https://arxiv.org/abs/2305.12295&quot;&gt;Logic-LM&lt;/a&gt;,
&lt;a href=&quot;https://arxiv.org/abs/2305.09656&quot;&gt;SatLM&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;What I find worth writing about is where the cuts are:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The neural half is restricted to &lt;strong&gt;proposing an interpretation&lt;/strong&gt;, never a
verdict, and the interpretation is a small, described, human-readable model.&lt;/li&gt;
  &lt;li&gt;The model is &lt;strong&gt;shaped by the questions the symbolic half will ask&lt;/strong&gt;, which is why
it includes an independent intent and paired readings.&lt;/li&gt;
  &lt;li&gt;Over-strictness is a query against an &lt;strong&gt;independently modeled intent&lt;/strong&gt;.&lt;/li&gt;
  &lt;li&gt;Ambiguity is &lt;strong&gt;witnessed divergence&lt;/strong&gt; between two formal readings of one rule.&lt;/li&gt;
  &lt;li&gt;The same symbolic half runs over &lt;strong&gt;inferred and executed facts&lt;/strong&gt;, so the reasoning
does not depend on the neural half being there.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Together they turn three fuzzy complaints, “too strict,” “conflicting,” and
“unclear,” into questions with checkable answers.&lt;/p&gt;

&lt;h2 id=&quot;the-question-i-think-comes-next&quot;&gt;The question I think comes next&lt;/h2&gt;

&lt;p&gt;The handoff itself is where the real question lives: what is the smallest, most
inspectable thing a language model can pass to a solver? Here, I used a
propositional encoding with described predicates, though the right choice will
depend on the task.&lt;/p&gt;

&lt;p&gt;There is also a question that sits beside the handoff. A guardrail is a
specification, and something has to apply it. In many systems that something is an
LLM judge, which reads a response, checks it against the rules, and returns a
verdict. Checking the policy says nothing about whether the judge applied it as
written. So the next question is who audits the judge, which is the subject of
&lt;a href=&quot;/posts/notes/who-audits-the-llm-judge.html&quot;&gt;the next post&lt;/a&gt;.&lt;/p&gt;
</content>
 </entry>
 
 
 
 <entry>
   <title>What Do We Mean by LLM Chemistry?</title>
   <link href="http://huascarsanchez.com/posts/notes/what-do-we-mean-by-llm-chemistry.html"/>
   <updated>2026-10-06T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/what-do-we-mean-by-llm-chemistry</id>
   <content type="html">&lt;h2 id=&quot;why-picking-different-models-isnt-the-same-as-picking-models-that-work-well-together&quot;&gt;Why picking different models isn’t the same as picking models that work well together&lt;/h2&gt;

&lt;p&gt;When I started working on LLM Chemistry, I was interested in a fairly practical
question: if you have several LLMs available, from different providers and with
different strengths and weaknesses, how do you know which ones should work
together?&lt;/p&gt;

&lt;p&gt;Picking the strongest models seems reasonable, right? Picking models with
different strengths also seems reasonable. But neither tells you what will
actually happen when those models start collaborating, or whether they will make
each other better. That’s the gap I was trying to capture with LLM Chemistry.&lt;/p&gt;

&lt;p&gt;The easiest way to explain the idea is with three questions:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;How good is a model? That’s capability.&lt;/li&gt;
  &lt;li&gt;How different are two models? That’s diversity.&lt;/li&gt;
  &lt;li&gt;What happens to one model’s contribution when another model joins the team?
That’s chemistry.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last question is subtle. Two models can look very similar and still make a
terrible team. They might reinforce the same errors, contribute redundant
information, or simply fail to combine their strengths. Conversely, two models
that look dramatically different in capability and performance might interact
surprisingly well. The difference between the models matters, but so does what
happens when those differences meet. That’s the idea behind LLM Chemistry.&lt;/p&gt;

&lt;p&gt;So one of the ideas behind LLM Chemistry is:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Diversity tells us how different models are. Chemistry tells us how those
  differences matter when they work together.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Since a former colleague and I published the &lt;a href=&quot;https://arxiv.org/abs/2510.03930&quot;&gt;original LLM
Chemistry work&lt;/a&gt;,&lt;sup id=&quot;fnref:naming&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:naming&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; I’ve come across
other research that makes this distinction especially interesting. One paper,
&lt;a href=&quot;https://arxiv.org/abs/2609.38816&quot;&gt;“You’re Hired: Strategic Model Selection for LLM
Collaboration,”&lt;/a&gt; directly studies how to
choose models for collaborative teams. Another, &lt;a href=&quot;https://arxiv.org/abs/2509.15090&quot;&gt;“Emergent Alignment via
Competition,”&lt;/a&gt; asks what happens when
differently motivated AI systems compete for a user’s decisions.&lt;/p&gt;

&lt;p&gt;They all involve heterogeneous collections of models, but they don’t all ask
the same question, and even when they do, they approach it differently. Looking
at them side by side provides a useful way
to understand what LLM Chemistry is and what it isn’t.&lt;/p&gt;

&lt;h2 id=&quot;chemistry-is-about-interaction&quot;&gt;Chemistry is about interaction&lt;/h2&gt;

&lt;p&gt;Suppose we have three models: A, B, and C.&lt;/p&gt;

&lt;p&gt;A is our strongest individual model. If we’re ranking models by performance, A
is an obvious choice. B has a very different performance profile. Perhaps it’s
better at something A struggles with. A diversity-based strategy might therefore
conclude that A and B are a promising pair.&lt;/p&gt;

&lt;p&gt;But we still don’t know whether B actually helps A. Maybe B catches errors that
A consistently misses. Maybe its critique causes A to reconsider a bad answer.
Or maybe B introduces noise and makes the final result worse.&lt;/p&gt;

&lt;p&gt;Knowing that A and B are different doesn’t tell us which of those things will
happen. LLM Chemistry approaches this by looking at marginal contribution.&lt;/p&gt;

&lt;p&gt;Suppose we have an existing group of models. We can ask how much adding A
changes the performance of that group. In the paper, this is expressed using a
benefit function derived from the change in the cost of answering a user query:&lt;/p&gt;

\[benefit_Q(X,Y)=cost_Q(Y)-cost_Q(X \cup Y).\]

&lt;p&gt;Here, \(Y\) is the existing group and \(X\) is the set being added. The benefit
can even be negative: adding a model can increase the cost and degrade
performance.&lt;/p&gt;

&lt;p&gt;Okay, now add B to the picture. Instead of asking whether A and B are similar,
we ask:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Does the value contributed by A change when B is present?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Dropping \(Q\) and writing \(X\) for the existing group, we compare:&lt;/p&gt;

\[benefit(A,X)\]

&lt;p&gt;with:&lt;/p&gt;

\[benefit(A,X\cup\{B\}).\]

&lt;p&gt;The difference between those quantities gives us what I call an &lt;em&gt;interaction effect&lt;/em&gt;:&lt;/p&gt;

\[\Delta(A,B,X)=|benefit(A,X)-benefit(A,X\cup\{B\})|.\]

&lt;p&gt;The full chemistry definition takes the maximum of this interaction effect over
possible surrounding configurations \(X\), normalized by the cost of the
resulting configuration:&lt;/p&gt;

\[chem_Q(A,B,S)=\max_{X\subseteq S\setminus\{A,B\}}\frac{\Delta(A,B,X)}{cost_Q(X\cup\{A,B\})},\]

&lt;p&gt;where \(S\) is the set of candidate models. This is Equation 2 in the paper.&lt;/p&gt;

&lt;p&gt;The equation matters, but the intuition is simpler:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;B matters to A if A’s contribution changes because B is there.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And that’s the relationship I’m trying to detect.&lt;/p&gt;

&lt;h2 id=&quot;where-diversity-enters-the-picture&quot;&gt;Where diversity enters the picture&lt;/h2&gt;

&lt;p&gt;So far, I’ve used diversity mostly as a contrast: the thing chemistry is &lt;em&gt;not&lt;/em&gt;.
But that doesn’t mean diversity is irrelevant. If chemistry measures how models
interact, where do their differences come in? This is where one of the
theoretical results from the original paper becomes important.&lt;/p&gt;

&lt;p&gt;Theorem 1 says that chemistry emerges when models exhibit heterogeneous
performance profiles. If models have identical—or nearly identical—performance
profiles, the &lt;em&gt;cost-based&lt;/em&gt; selection pressure disappears and there is no
interaction effect for the framework to detect.&lt;/p&gt;

&lt;p&gt;In other words, performance heterogeneity creates the conditions under which
chemistry can emerge. The corollary goes further and connects chemistry to the
amount of performance diversity, while noting that the direction of that
relationship depends on the task. This distinction is important:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Diversity is not irrelevant to chemistry. It is one of the conditions that makes chemistry possible.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;But diversity alone doesn’t tell us what the resulting interaction means for the
team. The same heterogeneous performance profiles can produce useful
complementarity, or they can produce interactions that don’t improve the final
result.&lt;/p&gt;

&lt;p&gt;The experiments reflect the task dependence the corollary predicts.
Chemistry-based selection improved effectiveness in some tasks and group sizes,
was effectively neutral in others, and underperformed in one small-group
summarization setting. So a more precise way to put the relationship is:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Diversity is a necessary condition for chemistry, but it is not sufficient.
  Chemistry is what tells us whether the diversity is useful.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is why I wouldn’t summarize the idea as “choose the most diverse models.”
That would be misleading. The real question is what those differences do once
the models interact.&lt;/p&gt;

&lt;h2 id=&quot;chemistry-is-also-a-selection-signal&quot;&gt;Chemistry is also a selection signal&lt;/h2&gt;

&lt;p&gt;Measuring interaction effects is useful diagnostically, but that wasn’t the
final goal of the work. The practical objective was still the question I started
with:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Which models should work together?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That’s why LLM Chemistry includes two related pieces: the &lt;em&gt;CHEME&lt;/em&gt; and
&lt;em&gt;RECOMMEND&lt;/em&gt; algorithms. CHEME estimates chemistry among candidate models by
measuring interaction effects across configurations. RECOMMEND uses those
estimates to search for model combinations that capture strong chemistry while
accounting for the cost of the resulting team.&lt;/p&gt;

&lt;p&gt;The paper describes the objective as recommending a set of LLMs that maximizes
the benefit of collaboration while minimizing the cost of answering the query.
Conceptually, the pipeline looks like this:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/static/figs/llm-chemistry-pipeline.png&quot; alt=&quot;LLM Chemistry pipeline: performance histories, capture joint LLM impact on query answers, chemistry table generator, recommend LLMs with strong chemistry&quot; style=&quot;max-width:100%;height:auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Read left to right, the figure shows chemistry being estimated from past
performance and then used to choose the team. LLM Chemistry isn’t only a way to
measure how models interact; it turns that interaction into information for
team selection.&lt;/p&gt;

&lt;h2 id=&quot;a-newer-paper-asks-a-familiar-question&quot;&gt;A newer paper asks a familiar question&lt;/h2&gt;

&lt;p&gt;Treating chemistry as a selection signal is what made &lt;a href=&quot;https://arxiv.org/abs/2609.38816&quot;&gt;“You’re Hired: Strategic
Model Selection for LLM Collaboration”&lt;/a&gt; by Cao
et al. particularly interesting to me.&lt;/p&gt;

&lt;p&gt;The paper asks:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;How should we select models to form an effective collaborative team?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Its setup is straightforward. Given a candidate model pool \(M\) and a desired
team size \(n\), a selector chooses a subset \(T\):&lt;/p&gt;

\[\sigma(M,n)=T.\]

&lt;p&gt;At that level, the problem should sound familiar. LLM Chemistry also asks how we
can identify which models should collaborate. In fact, our original paper
explicitly frames its central question as identifying which LLMs “work best
together” and introduces RECOMMEND for selecting model combinations with strong
chemistry.&lt;/p&gt;

&lt;p&gt;The interesting difference is how the two approaches characterize potential
collaborators. The “You’re Hired” paper explores a broad taxonomy of signals:
model descriptions, behavioral idiosyncrasies, Item Response Theory
representations, benchmark performance profiles, combinations of capability and
description information, and LLM-based recruiters. In the performance-profile
approach, for example, models are represented by performance vectors across
benchmarks and compared through the similarity of those vectors. That’s close
to the diversity signal I discussed earlier: useful for telling models apart,
but not on its own a measure of how they’ll interact.&lt;/p&gt;

&lt;p&gt;Put simply, many of these approaches ask:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;What do we know about these models that might tell us who should be on the team?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;LLM Chemistry, on the other hand, comes at the problem from another direction:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;What have we learned from what happens when these models are actually together?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;One reasons from model characteristics toward the team. The other reasons from
observed interaction toward the team. Those aren’t mutually exclusive
approaches. In fact, they may ultimately be complementary.&lt;/p&gt;

&lt;h2 id=&quot;team-composition-cannot-be-decoupled-from-team-interaction&quot;&gt;“Team composition cannot be decoupled from team interaction”&lt;/h2&gt;

&lt;p&gt;One result in the “You’re Hired” paper stood out to me in particular. The
authors found that a selection method that produces a strong team under one
collaboration mechanism can perform poorly under another. They concluded:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;“team composition cannot be decoupled from team interaction.”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;By “team interaction,” the authors mean the collaboration mechanism (how the
team refines, fuses, or merges its outputs) rather than interaction effects in
the sense I’ve used here. But the lesson points in the same direction.&lt;/p&gt;

&lt;p&gt;That result is interesting when viewed through the lens of chemistry. If
collaborative performance were determined primarily by individual model
strength, we could rank the models and select the top \(n\). If it were
determined primarily by diversity, we could maximize some measure of difference.
But if the best team changes with how the models work together, neither
capability nor diversity is enough on its own, and that’s the gap chemistry is
meant to fill.&lt;/p&gt;

&lt;h2 id=&quot;different-doesnt-necessarily-mean-useful&quot;&gt;Different doesn’t necessarily mean useful&lt;/h2&gt;

&lt;p&gt;The “You’re Hired” paper also illustrates why diversity alone can be tricky. As
the authors increase the diversity of the available model pool, additional
distinct models initially help some selectors. But eventually the gains can
saturate or reverse.&lt;/p&gt;

&lt;p&gt;A weak model can also be an unusual model.&lt;/p&gt;

&lt;p&gt;A selection method that rewards difference without sufficiently accounting for
capability can therefore select weak outliers precisely because they look
distinctive. The paper concludes that maximizing distinctness alone isn’t
enough; effective selection has to balance diversity with capability.&lt;/p&gt;

&lt;p&gt;This brings us back to the three questions from the start:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Capability: How good is this model?&lt;/li&gt;
  &lt;li&gt;Diversity: How different are these models?&lt;/li&gt;
  &lt;li&gt;Chemistry: How does one model’s contribution change in the presence of another?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;They’re related questions, but they aren’t interchangeable. And this makes the
complementarity I mentioned earlier more concrete: capability and diversity are
built from per-model signals, so collecting them scales with the size of the
pool rather than the number of possible teams. That makes them natural for
narrowing a large pool, while interaction-aware estimation could help
distinguish among the remaining combinations. I’ll come back to that.&lt;/p&gt;

&lt;h2 id=&quot;what-about-emergent-alignment-via-competition&quot;&gt;What about “Emergent Alignment via Competition”?&lt;/h2&gt;

&lt;p&gt;“You’re Hired” shows that differences in capability can saturate or mislead.
But models can also differ in what they’re trying to achieve, and that’s where a
second paper, &lt;a href=&quot;https://arxiv.org/abs/2509.15090&quot;&gt;“Emergent Alignment via
Competition,”&lt;/a&gt; comes in. At first glance, it
looks related: it also considers heterogeneous AI systems and explicitly
motivates a world in which models come from different providers with potentially
different incentives. But the similarity is mostly at the setting level.&lt;/p&gt;

&lt;p&gt;The paper asks whether a human interacting with multiple differently misaligned
AI agents can nevertheless obtain outcomes comparable to interacting with a
perfectly aligned model.&lt;/p&gt;

&lt;p&gt;Its central assumption is that the user’s utility function can be approximated
by a weighted combination of the AI agents’ utility functions—roughly, that it
lies in their convex hull. The mechanism is then strategic competition.&lt;/p&gt;

&lt;p&gt;So although both papers involve heterogeneous collections of models, they’re
trying to accomplish different things.&lt;/p&gt;

&lt;p&gt;LLM Chemistry asks:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Which models work well together on a collaborative task?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;“Emergent Alignment via Competition” asks:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Can competition among differently aligned models produce outcomes that are good for the user?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The first is fundamentally a collaborator-selection and interaction problem. The
second is fundamentally a strategic alignment and mechanism-design problem.&lt;/p&gt;

&lt;p&gt;That’s an important distinction because it shows just how many different things
we can mean when we talk about “diverse models.”&lt;/p&gt;

&lt;h2 id=&quot;diversity-isnt-one-thing&quot;&gt;Diversity isn’t one thing&lt;/h2&gt;

&lt;p&gt;Putting these papers next to each other helped clarify something for me. We
often talk about model diversity as though it were a single property. It isn’t.&lt;/p&gt;

&lt;p&gt;Models can differ in many ways: in architecture and training, in capability and
observed behavior, in performance profiles, and in their objectives or
incentives. Those differences aren’t interchangeable.&lt;/p&gt;

&lt;p&gt;In “Emergent Alignment via Competition,” the important heterogeneity is
primarily in utility functions and incentives, while in “You’re Hired,” candidate models
are characterized through capability, behavior, descriptions, performance
profiles, and related signals.&lt;/p&gt;

&lt;p&gt;In LLM Chemistry, the theoretical connection is specifically to heterogeneous
performance characteristics—the quality and accuracy profiles used by the
framework—and to the interaction dependencies that emerge from them.&lt;/p&gt;

&lt;p&gt;That suggests that instead of asking:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Should we use diverse models?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;we might ask:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Different in what way?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Then:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Different for what purpose?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And finally:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;What happens when those differences meet?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That last question is where chemistry begins.&lt;/p&gt;

&lt;h2 id=&quot;the-question-i-think-comes-next&quot;&gt;The question I think comes next&lt;/h2&gt;

&lt;p&gt;Earlier, I said I’d come back to how these signals might work together.
Comparing these papers leaves me with a question I find more interesting than
deciding whether capability, diversity, or chemistry is the “right” selection
signal:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;When can capability and diversity predict chemistry—and when do we actually
  need to measure interaction?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Measuring interaction is the expensive part. To learn how models affect one
another, we need evidence from collaborative configurations, and the number of
possible configurations grows combinatorially with the size of the pool.
Capability and diversity signals, by contrast, are collected per model. That
suggests a possible hierarchy:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/static/figs/llm-chemistry-hierarchy.png&quot; alt=&quot;A possible selection hierarchy: a large model pool goes through capability filtering and diversity or behavioral screening to produce a smaller candidate pool, then chemistry estimation and interaction-aware recommendation&quot; style=&quot;max-width:100%;height:auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;In that picture, capability and diversity aren’t alternatives to chemistry.
They could serve as priors for where chemistry is worth measuring.&lt;/p&gt;

&lt;p&gt;That opens several other questions:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Does chemistry remain stable when the collaboration mechanism changes?&lt;/li&gt;
  &lt;li&gt;Can two models have useful chemistry for one task and poor chemistry for
another?&lt;/li&gt;
  &lt;li&gt;How does chemistry change when one of the underlying models is updated?&lt;/li&gt;
  &lt;li&gt;If chemistry-, capability-, and diversity-based selectors were compared on
the same pools, tasks, and collaboration mechanisms, when would each one win?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those are empirical questions. But the conceptual question underneath them is
the same one that motivated LLM Chemistry in the first place. When building a
multi-LLM system, it isn’t enough to ask how good the models are, or how
different they are. We also need to ask:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;What happens to their contributions when they work together?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That’s what I mean by LLM Chemistry.&lt;/p&gt;

&lt;h2 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h2&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:naming&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;In hindsight, we probably should have titled the paper “LLM
Chemistry Estimation for Multi-LLM &lt;em&gt;Team&lt;/em&gt; Recommendation.” That would have
been more accurate, but it’s a mouthful, so we dropped “Team.” That was
probably a mistake: without it, it’s easy to read the work as recommending
individual models, when the point is recommending models that work well
together. &lt;a href=&quot;#fnref:naming&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
 </entry>
 
 
 
 <entry>
   <title>The Anatomy of an AI Harness</title>
   <link href="http://huascarsanchez.com/posts/notes/the-anatomy-of-ai-harness.html"/>
   <updated>2026-10-04T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/the-anatomy-of-ai-harness</id>
   <content type="html">&lt;p&gt;Over the past few months, I have built several AI harnesses to ensure agent
reliability and safety. While the scope of a minimum viable AI harness is still
fresh in my mind, I wanted to write about its anatomy.&lt;/p&gt;

&lt;p&gt;In a nutshell, an AI model generates outputs, while an AI harness turns those
outputs into controlled, stateful execution. That distinction matters.&lt;/p&gt;

&lt;p&gt;A model can answer a question or generate a tool call, but something else must
maintain context, execute the requested action, return the result, decide
whether another invocation is allowed, and determine when the process is complete.
That something is the AI harness.&lt;/p&gt;

&lt;p&gt;At its foundation, the anatomy is surprisingly small:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart LR
    Harness[AI Harness]
    Harness --&amp;gt; Model[Model]
    Harness --&amp;gt; State[State]
    Harness --&amp;gt; Capabilities[Capabilities]
    Harness --&amp;gt; Loop[Execution Loop]
    Harness --&amp;gt; Control[Control]
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Or, more compactly:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;AI harness = state + model + capabilities + execution loop + control.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Everything more sophisticated — e.g., memory, planning, retrieval, subagents,
sandboxing, evaluation — can grow from this foundation.&lt;/p&gt;

&lt;h2 id=&quot;the-core-anatomy&quot;&gt;The Core Anatomy&lt;/h2&gt;

&lt;h3 id=&quot;model&quot;&gt;Model&lt;/h3&gt;

&lt;p&gt;The model serves as the decision-making engine inside the harness. Whether it is
a large language model, a multimodal model, or another system that generates
outputs from inputs, the harness interacts with it to determine next steps.&lt;/p&gt;

&lt;p&gt;From the harness’s perspective, this interface can be remarkably simple:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;response&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;generate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;messages&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;state&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;tools&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;available_tools&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Given the current context and available capabilities, the model produces either
a direct response or a request to perform an action. Crucially, however, the
model does not carry out the action itself:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The model proposes what happens next. The harness makes it executable.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The model is therefore a component of the harness, not the harness itself.&lt;/p&gt;

&lt;h3 id=&quot;state&quot;&gt;State&lt;/h3&gt;

&lt;p&gt;A model invocation only knows what is placed in its context. The harness must
therefore preserve enough information about what has already happened. At
minimum:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;User:
    Find the problem in this program.

Assistant → Tool:
    read_file(&quot;main.py&quot;)

Tool:
    &amp;lt;contents of main.py&amp;gt;

Assistant:
    ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Each action produces new information that can influence the next model invocation.
State gives the execution continuity.&lt;/p&gt;

&lt;p&gt;More sophisticated systems may introduce working memory, task state,
checkpoints, artifacts, or context compression, but these are extensions of the
same basic idea:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;State tells the next model invocation what has happened so far.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3 id=&quot;capabilities&quot;&gt;Capabilities&lt;/h3&gt;

&lt;p&gt;While state provides context and the model decides what happens next, text
generation alone does not allow a model to interact with its environment.&lt;/p&gt;

&lt;p&gt;Capabilities provide the mechanisms to take real-world action. They define the
tools, APIs, and interfaces available to the harness to execute the model’s
intent. They might include operations such as:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;read_file&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;path&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;write_file&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;path&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;content&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;search&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;query&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;run_command&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;command&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;query_database&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sql&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;call_api&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(...)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The harness exposes descriptions of these capabilities to the model. When the
model requests one, the harness interprets the request, validates it, executes
the corresponding operation, and returns the result.&lt;/p&gt;

&lt;p&gt;This creates a fundamental boundary: the model decides what it wants to do while
the harness determines how that action actually happens.&lt;/p&gt;

&lt;p&gt;That boundary becomes especially important once permissions, security,
sandboxing, or human approval enter the picture.&lt;/p&gt;

&lt;h3 id=&quot;execution-loop&quot;&gt;Execution Loop&lt;/h3&gt;

&lt;p&gt;The execution loop is the heart of the harness. A minimal version might look
like this:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;response&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;generate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;state&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tools&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;response&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;is_final&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;response&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;text&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;call&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;response&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tool_calls&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tools&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;execute&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;call&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;state&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;add&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;call&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Without the loop, interaction looks roughly like:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart LR
    Input --&amp;gt; Model --&amp;gt; Output
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;With the harness:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart LR
    input --&amp;gt; model
    model --&amp;gt; action
    action --&amp;gt; observation
    observation --&amp;gt; action
    action --&amp;gt; final
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The model can act, observe the result, reconsider what to do, and act again.&lt;/p&gt;

&lt;p&gt;This seemingly small architectural change is fundamental. A model invocation
becomes an ongoing interaction with an environment.&lt;/p&gt;

&lt;h3 id=&quot;control-who-governs-the-loop&quot;&gt;Control: Who Governs the Loop?&lt;/h3&gt;

&lt;p&gt;Left unchecked, a model will happily loop indefinitely or burn through an API
budget. The harness sets the boundaries.&lt;/p&gt;

&lt;p&gt;At minimum, execution stops when the model emits a final answer. In practice,
the harness also enforces operational guardrails:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Step limits (to catch runaway loops)&lt;/li&gt;
  &lt;li&gt;Execution timeouts and cost budgets&lt;/li&gt;
  &lt;li&gt;Tool permissions and human approval gates&lt;/li&gt;
  &lt;li&gt;User cancellation&lt;/li&gt;
  &lt;li&gt;Retry policies and error limits&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The model proposes what it wants to do, but the harness holds the kill switch.&lt;/p&gt;

&lt;h3 id=&quot;following-one-task-through-the-harness&quot;&gt;Following One Task Through the Harness&lt;/h3&gt;

&lt;p&gt;Consider a user asking:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Find the bug in this repository.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The harness receives the request and initializes state. It then invokes the
model with the current state and available capabilities. E.g., the model decides
that it needs to inspect the repository:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;Model&lt;/span&gt;
  &lt;span class=&quot;err&quot;&gt;↓&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;list_files&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The harness executes the tool:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;Tool&lt;/span&gt;
  &lt;span class=&quot;err&quot;&gt;↓&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;src&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;/&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;main&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;py&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;parser&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;py&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;tests&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;/&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;test_parser&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;py&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;That observation enters the state. After that, runs again and decides to inspect
a file:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;read_file&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;src/parser.py&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The harness executes the request, records the result, and invokes the model again.
Then, the cycle continues:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart TD
    %%{init: {&quot;flowchart&quot;: {&quot;rankSpacing&quot;: 17, &quot;padding&quot;: 7}, &quot;themeVariables&quot;: {&quot;fontSize&quot;: &quot;14px&quot;}}}%%
    request[User request] --&amp;gt; initial_model[Model]
    initial_model --&amp;gt; action[Action]
    action --&amp;gt; execution[Execution]
    execution --&amp;gt; observation[Observation]
    observation --&amp;gt; state[State update]
    state --&amp;gt; next_model[Model]
    next_model --&amp;gt; continuation[...]
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Eventually, the model determines that it has enough information and produces a
final response.&lt;/p&gt;

&lt;p&gt;The harness did not need a predefined debugging workflow. Instead, it provided
something more general:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;A controlled environment in which a model can repeatedly decide what to do
  next based on what has happened previously.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3 id=&quot;the-first-extension-observability&quot;&gt;The First Extension: Observability&lt;/h3&gt;

&lt;p&gt;While these five components are enough to build a working harness, one
addition becomes valuable almost immediately: an event log.&lt;/p&gt;

&lt;p&gt;That is, instead of seeing a run as:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart LR
    Task --&amp;gt; Answer
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;record the execution:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Run
 │
 ├── user_message
 ├── model_request
 ├── model_response
 ├── tool_call
 ├── tool_result
 ├── model_request
 ├── model_response
 └── final
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Now the harness can answer questions such as:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;What did the model see?&lt;/li&gt;
  &lt;li&gt;What action did it choose?&lt;/li&gt;
  &lt;li&gt;What did the tool return?&lt;/li&gt;
  &lt;li&gt;Where did execution fail?&lt;/li&gt;
  &lt;li&gt;How many model calls occurred?&lt;/li&gt;
  &lt;li&gt;How much did the run cost?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This trace becomes the foundation for debugging, replay, monitoring, evaluation,
and performance analysis. A minimal practical harness therefore looks something
like:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart TD
    state[State] --&amp;gt; loop[Execution Loop]
    loop --&amp;gt; model[Model]
    loop --&amp;gt; tools[Tools]
    model --&amp;gt; log[Event Log]
    tools --&amp;gt; log
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&quot;everything-else-builds-on-the-core&quot;&gt;Everything Else Builds on the Core&lt;/h3&gt;

&lt;p&gt;Production systems naturally get much more complex, layering on things like
planning, long-term memory, retrieval, sandboxing, human approvals, and
multi-agent orchestration. But while those capabilities matter, none of them are
required to define a harness. Keeping this distinction clear prevents “AI
harness” from becoming synonymous with an entire agent framework: a harness can
be lean, defined purely by the machinery that turns model decisions into
controlled execution.&lt;/p&gt;

&lt;h3 id=&quot;where-does-the-agent-live&quot;&gt;Where Does the Agent Live?&lt;/h3&gt;

&lt;p&gt;This raises a more interesting question: if the model isn’t the agent, and the
harness isn’t the agent, where does the agent actually live?&lt;/p&gt;

&lt;p&gt;It is tempting to imagine:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart TD
    harness[Harness] --&amp;gt; agent[Agent]
    agent --&amp;gt; model[Model]
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;But agent behavior is better understood as emerging from the interaction of
several components:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart LR
    model[Model] --&amp;gt; combined((+))
    instructions[Instructions] --&amp;gt; combined
    state[State] --&amp;gt; combined
    capabilities[Available capabilities] --&amp;gt; combined
    semantics[Harness execution semantics] --&amp;gt; combined
    combined --&amp;gt; behavior[Agent behavior]
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This distinction has practical consequences. Change the model, the instructions,
or the available tools, and behavior will naturally shift. But change the
harness, and behavior changes just as fundamentally.&lt;/p&gt;

&lt;p&gt;A harness that permits unlimited actions creates a very different effective
agent from one that requires approval for consequential operations. A harness
that retains the complete interaction history produces different behavior from
one that aggressively compresses context. And a harness that retries failed
actions behaves quite differently from one that immediately terminates.&lt;/p&gt;

&lt;p&gt;The harness is not merely plumbing around an otherwise complete agent—its
execution semantics directly shape how the resulting agent behaves.&lt;/p&gt;

&lt;h3 id=&quot;the-anatomy-in-one-picture&quot;&gt;The Anatomy in One Picture&lt;/h3&gt;

&lt;p&gt;Strip away planners, memory systems, retrieval pipelines, subagents, and
orchestration, and the architecture becomes remarkably compact:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-mermaid&quot;&gt;flowchart TD
    S[State] --&amp;gt; M[Model]
    M --&amp;gt;|decides next action| C[Capability]
    C --&amp;gt;|result| S

    CTRL[Control] -.-&amp;gt;|governs invocation| M
    CTRL -.-&amp;gt;|governs execution| C

    S -.-&amp;gt; LOG[Event Log]
    M -.-&amp;gt; LOG
    C -.-&amp;gt; LOG
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The model makes the decisions, state preserves continuity, and capabilities let
it interact with the outside world. The execution loop turns those discrete
steps into an ongoing run, control keeps the process within bounds, and
observability lets you see what actually took place.&lt;/p&gt;

&lt;p&gt;That is the basic anatomy:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;An AI harness is the machinery that turns model inference into controlled, stateful execution.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And that’s it. Everything else is an extension of this foundation.&lt;/p&gt;
</content>
 </entry>
 
 
 
 <entry>
   <title>Gating Your LLM Eval Rubric with Evidence</title>
   <link href="http://huascarsanchez.com/posts/notes/gating-eval-rubric-with-evidence.html"/>
   <updated>2026-05-26T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/gating-eval-rubric-with-evidence</id>
   <content type="html">&lt;p&gt;&lt;em&gt;Part 2: &lt;a href=&quot;/posts/notes/managing-eval-rubric-like-code.html&quot;&gt;Managing Your LLM Eval Rubric Like Code&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Promptory’s staged releases, evidence documents, and release gates let you attach eval results to a rubric candidate and block promotion until that evidence passes. This post shows the workflow step by step. The full example is in &lt;a href=&quot;https://github.com/hsanchez/promptory/blob/main/examples/evals2.py&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;examples/evals2.py&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;the-gate-declaration&quot;&gt;The Gate Declaration&lt;/h2&gt;

&lt;p&gt;Add a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;release_gates&lt;/code&gt; block to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;promptspec.yaml&lt;/code&gt; before releasing any candidates:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;files&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;rubric.yaml&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;required_variables&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;criteria&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;max_file_bytes&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;100000&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;release_gates&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;evidence&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;kind&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;eval&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;eval-run&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;required_status&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;pass&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Any promotion attempt that lacks an evidence document of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;kind: eval&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;name: eval-run&lt;/code&gt;, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;status: pass&lt;/code&gt; raises &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PromptGateError&lt;/code&gt; and leaves &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt; unchanged.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;step-1-stage-a-candidate&quot;&gt;Step 1: Stage a Candidate&lt;/h2&gt;

&lt;p&gt;A staged release writes an immutable snapshot under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;versions/&lt;/code&gt; without updating &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt;. The active version is unaffected while the candidate is evaluated.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory.manager&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptManager&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptManager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;draft&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;prompts_dir&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;drafts&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;rubric.yaml.j2&quot;&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;draft&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;write_text&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;BASIC_RUBRIC&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;v1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;release&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;bump&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;patch&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;variables&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;criteria&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;helpfulness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;staged&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c1&quot;&gt;# v0.0.1 exists in versions/ but current.json is unchanged.
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;step-2-attach-evidence&quot;&gt;Step 2: Attach Evidence&lt;/h2&gt;

&lt;p&gt;Load the staged rubric, run your eval harness, then write the results as an evidence document and attach it to the version.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory.evidence&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;add_evidence&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;store&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;rubric&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;rubric.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;version&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;metrics&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;run_eval_harness&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;rubric&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;evidence_doc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;kind&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;eval&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;name&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;eval-run&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;status&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;fail&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;          &lt;span class=&quot;c1&quot;&gt;# below threshold: 60% accuracy
&lt;/span&gt;  &lt;span class=&quot;s&quot;&gt;&quot;tool&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;eval-harness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;created_at&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;2026-05-28T12:00:00Z&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;summary&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;accuracy=60%&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;metrics&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;metrics&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;evidence_path&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;write_text&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;json&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dumps&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;evidence_doc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;add_evidence&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;spec&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;evidence_path&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Evidence is immutable once attached.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;step-3-check-the-gate&quot;&gt;Step 3: Check the Gate&lt;/h2&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;gate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;gate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c1&quot;&gt;# False — evidence status is fail, required pass
&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;gate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;passed&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;v1&lt;/code&gt; stays staged.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;step-4-iterate&quot;&gt;Step 4: Iterate&lt;/h2&gt;

&lt;p&gt;Revise the rubric draft and stage a new candidate.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;draft&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;write_text&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;STRICT_RUBRIC&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;release&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;bump&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;patch&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;variables&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;criteria&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;helpfulness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;staged&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;rubric_v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;rubric.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;version&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;metrics_v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;run_eval_harness&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;rubric_v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;evidence_doc_v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;kind&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;eval&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;name&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;eval-run&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;status&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;pass&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;          &lt;span class=&quot;c1&quot;&gt;# 100% accuracy clears the threshold
&lt;/span&gt;  &lt;span class=&quot;s&quot;&gt;&quot;tool&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;eval-harness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;created_at&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;2026-05-28T13:00:00Z&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;summary&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;accuracy=100%&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;metrics&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;metrics_v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;add_evidence&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;spec&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;evidence_path_v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;gate_v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;gate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;gate_v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;passed&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;   &lt;span class=&quot;c1&quot;&gt;# True
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;step-5-compare-evidence&quot;&gt;Step 5: Compare Evidence&lt;/h2&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;compare_evidence&lt;/code&gt; diffs the attached evidence between two versions.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory.evidence&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;compare_evidence&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;comparison&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;compare_evidence&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;spec&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;change&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;comparison&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;changes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sa&quot;&gt;f&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;[&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;change&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;kind&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;] &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;change&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;: &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\
&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;    &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;change&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;before_status&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt; -&amp;gt; &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;change&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;after_status&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;metric&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;change&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;metrics&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sa&quot;&gt;f&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;  &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;metric&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;: &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;metric&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;before&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt; -&amp;gt; &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;metric&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;after&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Output:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;[eval] eval-run: fail -&amp;gt; pass
  accuracy: 0.6 -&amp;gt; 1.0
  false_positive_rate: 1.0 -&amp;gt; 0.0
  mean_score: 4.6 -&amp;gt; 3.4
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;step-6-promote&quot;&gt;Step 6: Promote&lt;/h2&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;require_gates=True&lt;/code&gt; makes the gate check part of the promotion call. If evidence is missing or fails, the call raises &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PromptGateError&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt; is not updated.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;promote&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;require_gates&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;current_version&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;())&lt;/span&gt;   &lt;span class=&quot;c1&quot;&gt;# v0.0.2
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;v1&lt;/code&gt; remains staged. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;v2&lt;/code&gt; is now current.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;running-the-example&quot;&gt;Running the Example&lt;/h2&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run python examples/evals2.py
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Expected output:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;[v1 basic] staged v0.0.1
  accuracy=60%  fpr=100%  fnr=0%
  gate: FAIL - candidate stays staged

[v2 strict] staged v0.0.2
  accuracy=100%  fpr=0%  fnr=0%
  gate: PASS

Evidence comparison v0.0.1 -&amp;gt; v0.0.2:
  [eval] eval-run: fail -&amp;gt; pass
    accuracy: 0.6 -&amp;gt; 1.0
    false_positive_rate: 1.0 -&amp;gt; 0.0
    mean_score: 4.6 -&amp;gt; 3.4

Promoted v0.0.2 -&amp;gt; current: v0.0.2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;what-promptory-does-not-do&quot;&gt;What Promptory Does Not Do&lt;/h2&gt;

&lt;p&gt;Promptory stores and validates the evidence document shape. It does not run evals, call LLMs, define metric thresholds, or decide whether a rubric is correct.&lt;/p&gt;
</content>
 </entry>
 
 
 
 <entry>
   <title>Managing Your LLM Eval Rubric Like Code</title>
   <link href="http://huascarsanchez.com/posts/notes/managing-eval-rubric-like-code.html"/>
   <updated>2026-05-19T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/managing-eval-rubric-like-code</id>
   <content type="html">&lt;p&gt;Promptory manages your eval rubric as a versioned prompt artifact: drafts,
immutable releases, template variables, and version history. This post shows
how to draft, release, and load rubric versions using the Python API.&lt;/p&gt;

&lt;p&gt;The full runnable example is in &lt;a href=&quot;https://github.com/hsanchez/promptory/blob/main/examples/evals.py&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;examples/evals.py&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;the-rubric-template&quot;&gt;The Rubric Template&lt;/h2&gt;

&lt;p&gt;A rubric prompt in Promptory is a Jinja template. Declare the file and its
required variables in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;promptspec.yaml&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;files&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;rubric.yaml&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;required_variables&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;criteria&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;max_file_bytes&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;100000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The draft template lives in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompts/drafts/rubric.yaml.j2&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;claude-sonnet-4-6&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;temperature&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;0.0&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;system_prompt&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;pi&quot;&gt;|&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Evaluate the response on {{ criteria }}.&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Be generous; give partial credit for effort.&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Return JSON: {score: int, rationale: str}.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{{ criteria }}&lt;/code&gt; is a template variable. It is rendered at release time, not
at runtime.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;template-variables&quot;&gt;Template Variables&lt;/h2&gt;

&lt;p&gt;Release the same rubric template with different variable values to produce
separate versioned artifacts for each eval dimension.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory.manager&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptManager&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptManager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;v_helpfulness&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;release&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;bump&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;patch&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;variables&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;criteria&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;helpfulness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;v_factual&lt;/span&gt;     &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;release&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;bump&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;patch&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;variables&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;criteria&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;factual accuracy&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Each release is an independent rendered artifact. Loading &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;v_helpfulness&lt;/code&gt; always
returns the rubric with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;criteria=&quot;helpfulness&quot;&lt;/code&gt; regardless of subsequent
releases.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;draft-and-release-separation&quot;&gt;Draft and Release Separation&lt;/h2&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompts/drafts/&lt;/code&gt; holds the working template. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompts/versions/&lt;/code&gt; holds
immutable rendered releases. Editing the draft and cutting a new release does
not modify any previous release.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;BASIC_RUBRIC&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;model: claude-sonnet-4-6&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;temperature: 0.0&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;system_prompt: |&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;  Evaluate the response on {{ criteria }}.&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;  Be generous; give partial credit for effort.&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;  Return JSON: {score: int, rationale: str}.&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;STRICT_RUBRIC&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;model: claude-sonnet-4-6&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;temperature: 0.0&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;system_prompt: |&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;  Evaluate the response on {{ criteria }}.&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;  Penalize vague or incomplete answers.&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;&quot;  Return JSON: {score: int, rationale: str}.&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# v1: initial rubric.
&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;draft&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;write_text&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;BASIC_RUBRIC&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;v1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;release&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;bump&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;patch&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;variables&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;criteria&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;helpfulness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;c1&quot;&gt;# v2: updated rubric. v1 is unchanged.
&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;draft&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;write_text&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;STRICT_RUBRIC&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;manager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;release&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;bump&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;patch&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;variables&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;criteria&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;helpfulness&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The directory &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompts/versions/v0.0.1/&lt;/code&gt; is never touched after creation.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;loading-by-version&quot;&gt;Loading by Version&lt;/h2&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PromptStore.load&lt;/code&gt; accepts an optional &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;version&lt;/code&gt; argument. When supplied, it
reads from that exact release directory regardless of which version is current.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;store&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;rubric_v1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;rubric.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;version&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;rubric_v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;rubric.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;version&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;c1&quot;&gt;# Pass rubric[&quot;system_prompt&quot;] to your eval harness.
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The result is identical whether &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;v1&lt;/code&gt; is current or not.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;version-history&quot;&gt;Version History&lt;/h2&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PromptStore.list_versions&lt;/code&gt; returns all available releases in semantic version
order.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;list_versions&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;())&lt;/span&gt;
&lt;span class=&quot;c1&quot;&gt;# [&apos;v0.0.1&apos;, &apos;v0.0.2&apos;, &apos;v0.0.3&apos;]
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Use this to scan all candidates and compare results across versions.&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;version      criteria               accuracy      fpr      fnr
v0.0.1        helpfulness              60%      100%        0%
v0.0.2        helpfulness             100%        0%        0%
v0.0.3        factual accuracy        100%        0%        0%
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;v0.0.2&lt;/code&gt; has the best metrics for helpfulness. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;v0.0.3&lt;/code&gt; applies the same rubric
instructions to a different eval dimension without re-authoring the template.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;running-the-example&quot;&gt;Running the Example&lt;/h2&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run python examples/evals.py
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The example uses a simulated judge and fixed benchmark data. Replace
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;simulate_judge_call&lt;/code&gt; with a real LLM call to run it against your own eval
suite.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;what-promptory-does-not-do&quot;&gt;What Promptory Does Not Do&lt;/h2&gt;

&lt;p&gt;Promptory manages the rubric prompt. It does not run evals, call LLMs, manage
benchmark datasets, or decide whether one rubric is better than another.&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;&lt;em&gt;Part 2: &lt;a href=&quot;/posts/notes/gating-eval-rubric-with-evidence.html&quot;&gt;Gating Your LLM Eval Rubric with Evidence&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;
</content>
 </entry>
 
 
 
 <entry>
   <title>Promptory, a Git-based prompt versioning system</title>
   <link href="http://huascarsanchez.com/posts/notes/promptory.html"/>
   <updated>2026-05-14T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/promptory</id>
   <content type="html">&lt;p&gt;In many organizations building GenAI solutions, prompts have become production
artifacts. However, despite their importance, many teams still treat them like
ordinary strings. Treating them this way might work for a while, but as the
prompts evolve, as their performance becomes variable across different models,
model versions, safety policies, or use cases, it becomes more complex and
harder to manage them as strings.&lt;/p&gt;

&lt;p&gt;That is the motivation behind
&lt;a href=&quot;https://github.com/hsanchez/promptory&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Promptory&lt;/code&gt;&lt;/a&gt;: a Git-based prompt
versioning system that gives prompt changes the same basic engineering
discipline as code. The core idea is simple:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Author prompts as editable drafts, release them as immutable rendered
artifacts, and make runtime code load only released versions.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;the-prompt-management-problem&quot;&gt;The prompt management problem&lt;/h2&gt;

&lt;p&gt;Prompt changes are deceptively small. Changing a few words can alter:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;how much context the model uses&lt;/li&gt;
  &lt;li&gt;whether it follows a tool policy&lt;/li&gt;
  &lt;li&gt;how it handles uncertain answers&lt;/li&gt;
  &lt;li&gt;whether it refuses unsafe requests&lt;/li&gt;
  &lt;li&gt;which format downstream parsers expect&lt;/li&gt;
  &lt;li&gt;how it behaves under evals&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In traditional software engineering, anything that can change production
behavior is managed deliberately: version control, review, tests, releases, and
rollback. I believe that &lt;em&gt;prompt engineering&lt;/em&gt; should not be exempt from that
discipline just because the artifact is text. On the contrary, it should also be
managed deliberately.&lt;/p&gt;

&lt;p&gt;The hard part is choosing the right amount of discipline. In practice, teams
often land on one of two extremes. The first one is the hardcoded prompt:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;SYSTEM_PROMPT&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;&quot;&quot;
You are a helpful assistant. Answer concisely.
&quot;&quot;&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Hardcoding a prompt like this is fine for a prototype, but it’s brittle for a
system that needs repeatable experiments, reviewable changes, or rollback.&lt;/p&gt;

&lt;p&gt;The second one is &lt;em&gt;overcorrection&lt;/em&gt;. This is the process of adopting a large
hosted prompt platform before the team actually needs one. Those systems can be
useful when non-technical users need a UI, many teams need centralized
governance, or the current prompt metadata no longer fits a particular
repository workflow.&lt;/p&gt;

&lt;p&gt;I strongtly believe fast-moving R&amp;amp;D teams often need something smaller:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;plain files&lt;/li&gt;
  &lt;li&gt;Git review&lt;/li&gt;
  &lt;li&gt;CI-friendly checks&lt;/li&gt;
  &lt;li&gt;immutable releases&lt;/li&gt;
  &lt;li&gt;a clear pointer to the active version&lt;/li&gt;
  &lt;li&gt;runtime loading that never reads work-in-progress drafts&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Promptory is aimed at that middle ground.&lt;/p&gt;

&lt;h2 id=&quot;why-git-alone-is-not-enough&quot;&gt;Why Git alone is not enough&lt;/h2&gt;

&lt;p&gt;Why not just use Git? That’s the obvious pushback I get when socializing
Promptory. Git already handles reviewing and preserving prompt history just
fine, they say. That is true. However, Git doesn’t define a prompt lifecycle.&lt;/p&gt;

&lt;p&gt;In other words, Git can tell you that a prompt file changed, but it cannot tell
your application:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;which rendered prompt version is active&lt;/li&gt;
  &lt;li&gt;whether a prompt has unresolved template variables&lt;/li&gt;
  &lt;li&gt;whether all managed prompt files were released together&lt;/li&gt;
  &lt;li&gt;whether a runtime process is reading drafts or released artifacts&lt;/li&gt;
  &lt;li&gt;how to roll back without rewriting prompt history&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is why Promptory keeps Git as the durable history and adds a small release
model on top of it.&lt;/p&gt;

&lt;h3 id=&quot;why-not-a-central-registry&quot;&gt;Why not a central registry?&lt;/h3&gt;

&lt;p&gt;Promptory’s answer is a deliberate ownership choice: prompts live with the code
that uses them. This builds on the same &lt;em&gt;locality of change&lt;/em&gt; principle from
software engineering — normally applied to code — and extends it to prompts.
Specifically, if a prompt shapes a service’s behavior, it should be reviewed,
released, and rolled back with that service.&lt;/p&gt;

&lt;p&gt;A central registry can be useful for organization-wide discovery, shared
prompts, or non-engineering workflows. But not so much for application-specific
prompts, which should live next to the implementation to make review and
rollback simpler. The prompt diff, code diff, tests, and release artifact all
move through the same Git workflow.&lt;/p&gt;

&lt;p&gt;This ownership choice doesn’t rule out central visibility. In production, teams can still
publish released prompt artifacts to object storage, an internal artifact
registry, or Promptory’s read-only registry service. The repo remains the
source of truth; production systems only aggregate or serve released artifacts
from there.&lt;/p&gt;

&lt;p&gt;In the repo, that looks something like this:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;prompts/
  drafts/
    system.yaml.j2
    input_guardrail.yaml.j2
    output_guardrail.yaml.j2
  versions/
    v0.1.0/
      system.yaml
      input_guardrail.yaml
      output_guardrail.yaml
      metadata.json
  current.json
  promptspec.yaml
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The above layout breaks down into four parts, each with a distinct role:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;drafts/&lt;/code&gt; contains editable &lt;em&gt;Jinja&lt;/em&gt; templates&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;versions/&lt;/code&gt; contains rendered release artifacts&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt; points to the active release&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;promptspec.yaml&lt;/code&gt; declares which prompt files are managed&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two of those four parts carry the real distinction. Developers edit drafts.
Applications load versions. That boundary, simple as it sounds, is the main
design decision in the project.&lt;/p&gt;

&lt;h2 id=&quot;drafts-are-for-authoring&quot;&gt;Drafts are for authoring&lt;/h2&gt;

&lt;p&gt;A draft is where prompt authors and coding agents work. It’s written in plain
YAML. For example:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# prompts/drafts/system.yaml.j2&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;gpt-5.5&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;temperature&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;0.2&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;system_prompt&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;pi&quot;&gt;|&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;You are a helpful assistant.&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Answer concisely and avoid unsupported claims.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Because drafts are Jinja templates rather than static text, they can also
include release-time variables:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# prompts/drafts/message.yaml.j2&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;message&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;pi&quot;&gt;|&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Hello {{ user_name }}.&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Generated at {{ generation_time }}.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Promptory renders these with Jinja’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;StrictUndefined&lt;/code&gt;, on purpose. A missing
variable throws an exception instead of quietly becoming an empty string. Silent
prompt rendering failures are hard to debug because they often look like model
behavior problems later.&lt;/p&gt;

&lt;p&gt;Rendering itself happens at release time, through two commands:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run prompt check
uv run prompt release &lt;span class=&quot;nt&quot;&gt;--patch&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt check&lt;/code&gt; catches problems early: bad drafts, missing variable
declarations. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt release&lt;/code&gt; does the real work. It renders the declared
templates, parses the rendered YAML, and drops the output into a new immutable
version directory before updating &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Variables don’t always come from the CLI, though. When a release needs them
supplied programmatically, the Python API takes them directly:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory.manager&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptManager&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;version&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptManager&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;release&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;variables&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;&quot;user_name&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;Alice&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;&quot;generation_time&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;2026-05-09T12:00:00Z&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The expected variables live in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;promptspec.yaml&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;files&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;system.yaml&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;message.yaml&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;required_variables&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;user_name&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;generation_time&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;max_file_bytes&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;100000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This file is the contract for the prompt directory. It says which rendered YAML
files Promptory manages and which variables must be supplied before release.&lt;/p&gt;

&lt;h2 id=&quot;versions-are-for-runtime&quot;&gt;Versions are for runtime&lt;/h2&gt;

&lt;p&gt;A release turns drafts into rendered YAML artifacts:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run prompt check
uv run prompt release &lt;span class=&quot;nt&quot;&gt;--patch&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;After release, Promptory writes a semantic version directory:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;prompts/
  versions/
    v0.0.1/
      system.yaml
      message.yaml
      metadata.json
  current.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Runtime code should not read from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;drafts/&lt;/code&gt;. It should read from the active
version:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;store&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;system&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;system.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;message&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;message.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PromptStore&lt;/code&gt; reads &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt; and validates the requested prompt name
against &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;promptspec.yaml&lt;/code&gt;. Then it loads rendered YAML from
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;versions/&amp;lt;version&amp;gt;/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The runtime path is deliberately boring as a result: no Jinja rendering at
runtime, no accidental use of draft prompts, and no hidden prompt selection
logic inside the application.&lt;/p&gt;

&lt;p&gt;Applications can also load a specific version:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;store&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;system_v1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;system.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;version&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;v0.0.1&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;all_v2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load_all&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;version&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;v0.0.2&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Pinning to a specific version like this is useful for evals and debugging. If an
output changed between two runs, then the prompt version becomes part of the evidence.&lt;/p&gt;

&lt;h2 id=&quot;the-release-pointer&quot;&gt;The release pointer&lt;/h2&gt;

&lt;p&gt;The small file that makes the runtime story work is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt;. See below:&lt;/p&gt;

&lt;div class=&quot;language-json highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;version&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;v0.0.1&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
  &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;updated_at&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;2026-05-09T22:14:00.000000+00:00&quot;&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This small file is the active release pointer, which makes the runtime story
work. When application code loads the current prompt, it is not asking, “What is
in the drafts directory today?” but rather, “Which release is currently active?”&lt;/p&gt;

&lt;p&gt;Rollback is therefore a pointer change, not a rewrite, accomplished with a
single command:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run prompt rollback v0.0.1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The release artifacts remain immutable. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt; moves back to an existing
version.&lt;/p&gt;

&lt;p&gt;This is closer to how many deployment systems work. You do not edit the old
release in place. You point the system at a known-good artifact.&lt;/p&gt;

&lt;h2 id=&quot;a-concrete-workflow&quot;&gt;A concrete workflow&lt;/h2&gt;

&lt;p&gt;Imagine a research team maintaining a small suite of prompts for a
customer-facing assistant. Rather than one monolithic prompt, they’ve split
responsibilities across three separate files:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;system.yaml&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;input_guardrail.yaml&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;output_guardrail.yaml&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The team captures this list in a spec, along with a few constraints each file
must satisfy:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;files&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;system.yaml&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;input_guardrail.yaml&lt;/span&gt;
  &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;output_guardrail.yaml&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;required_variables&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;pi&quot;&gt;[]&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;max_file_bytes&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;100000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Ultimately, the team doesn’t edit those files directly. They edit
Jinja-templated drafts under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompts/drafts/&lt;/code&gt;, which get rendered into the
finished files the spec expects:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# prompts/drafts/input_guardrail.yaml.j2&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;policy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;pi&quot;&gt;|&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Reject requests for secrets, credentials, or private keys.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# prompts/drafts/output_guardrail.yaml.j2&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;policy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;pi&quot;&gt;|&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;Answer concisely and avoid unsupported claims.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Before releasing, the team checks the prompt set:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run prompt check
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Then it previews the difference between the active release and rendered drafts:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run prompt diff
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Once that looks right, it creates a new release:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run prompt release &lt;span class=&quot;nt&quot;&gt;--patch&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;From there, the application consumes the released files:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;promptory&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;store&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PromptStore&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;prompts&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;system&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;system.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;input_guardrail&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;input_guardrail.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;output_guardrail&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;store&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;load&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;output_guardrail.yaml&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;messages&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;role&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;system&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;content&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;system&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;system_prompt&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]},&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;role&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;developer&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;content&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_guardrail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;policy&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]},&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;role&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;developer&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;content&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output_guardrail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;policy&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]},&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;role&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;user&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;content&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;user_message&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;},&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;That is the whole loop:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;edit draft -&amp;gt; check -&amp;gt; diff -&amp;gt; release -&amp;gt; runtime loads current version
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h2 id=&quot;the-authoringruntime-split&quot;&gt;The authoring/runtime split&lt;/h2&gt;

&lt;p&gt;Promptory is built around a strict boundary.&lt;/p&gt;

&lt;p&gt;Authoring tools can write prompt lifecycle state:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt init&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt check&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt diff&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt release&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt rollback&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prompt draft&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Runtime code reads released prompt state:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;active version from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;rendered YAML from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;versions/&amp;lt;version&amp;gt;/&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;file declarations from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;promptspec.yaml&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This split matters because prompt systems often blur authoring and runtime
concerns. If runtime code renders templates, then a missing variable becomes a
runtime incident. If runtime code reads drafts, then unreviewed edits can affect
production behavior. If applications choose prompt files directly, then the
promptspec contract is bypassed.&lt;/p&gt;

&lt;p&gt;Promptory keeps the runtime path small on purpose.&lt;/p&gt;

&lt;h2 id=&quot;what-the-checks-protect&quot;&gt;What the checks protect&lt;/h2&gt;

&lt;p&gt;The linter is not trying to prove that a prompt is good. It is trying to catch
the kinds of mechanical errors that should not reach a release.&lt;/p&gt;

&lt;p&gt;For example, Promptory checks:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;whether declared draft templates exist&lt;/li&gt;
  &lt;li&gt;whether templates have valid Jinja syntax&lt;/li&gt;
  &lt;li&gt;whether rendered artifacts are valid YAML&lt;/li&gt;
  &lt;li&gt;whether required variables are declared&lt;/li&gt;
  &lt;li&gt;whether undeclared variables are used&lt;/li&gt;
  &lt;li&gt;whether managed files are relative &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.yaml&lt;/code&gt; paths&lt;/li&gt;
  &lt;li&gt;whether file sizes exceed configured limits&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These checks are intentionally mundane, and that’s the point: a prompt release
should not fail because a template variable rendered to an empty string, a YAML
file stopped parsing, or an application loaded a file outside the managed prompt
set.&lt;/p&gt;

&lt;h2 id=&quot;why-immutable-releases-matter&quot;&gt;Why immutable releases matter&lt;/h2&gt;

&lt;p&gt;Immutable prompt releases give three practical benefits.&lt;/p&gt;

&lt;p&gt;First, they make review clearer. A reviewer can inspect both the editable draft
and the rendered artifact that runtime code will load.&lt;/p&gt;

&lt;p&gt;Second, they make debugging easier. If a model behavior changed after a release,
you can load the exact prompt version used by that run.&lt;/p&gt;

&lt;p&gt;Third, they make rollback safer. Rollback does not require editing prompt
content. It only changes the active pointer to a previously released artifact.&lt;/p&gt;

&lt;p&gt;This is the same reason build artifacts, container images, and database
migrations tend to get explicit versioning in mature systems. Prompts are not
identical to those artifacts, but they have the same operational pressure: they
change behavior.&lt;/p&gt;

&lt;h2 id=&quot;serving-prompts-to-non-python-clients&quot;&gt;Serving prompts to non-Python clients&lt;/h2&gt;

&lt;p&gt;The core Promptory model is file-based, but not every consumer is Python.&lt;/p&gt;

&lt;p&gt;For Go, TypeScript, or mixed-language systems, Promptory can expose released
prompts through a small registry service:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;uv run prompt serve &lt;span class=&quot;nt&quot;&gt;--port&lt;/span&gt; 8000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The service exposes endpoints such as:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;GET /versions
GET /versions/current
GET /prompts
GET /prompts/{name}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This does not create a second prompt lifecycle. The service is a read-only layer
over the same &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PromptStore&lt;/code&gt; contract: it reads &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;current.json&lt;/code&gt;, validates prompt
names, and serves rendered release artifacts, existing for consumption, not
authoring.&lt;/p&gt;

&lt;h2 id=&quot;what-promptory-is-not&quot;&gt;What Promptory is not&lt;/h2&gt;

&lt;p&gt;Promptory is deliberately small.&lt;/p&gt;

&lt;p&gt;It is not:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;a hosted prompt CMS&lt;/li&gt;
  &lt;li&gt;an experiment tracking platform&lt;/li&gt;
  &lt;li&gt;an eval runner&lt;/li&gt;
  &lt;li&gt;a deployment system&lt;/li&gt;
  &lt;li&gt;a replacement for Git&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It does not try to answer every question around prompt governance. Instead, it
focuses on one narrow lifecycle:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;turn draft prompt templates into versioned, reviewable, immutable runtime
artifacts.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That narrowness is useful. It keeps the tool understandable and lets teams wire
it into their existing Git, CI, eval, and deployment workflows.&lt;/p&gt;

&lt;h2 id=&quot;closing-thought&quot;&gt;Closing thought&lt;/h2&gt;

&lt;p&gt;Prompt changes are software changes. They may be written in natural language,
but they alter system behavior just as a code change does. Consequently, they
deserve the same basic lifecycle we already trust for code: review, validation,
versioning, release, and rollback. Promptory is my attempt to make that
lifecycle lightweight enough to not get in the way of day-to-day research work,
while still giving production systems the stable, predictable runtime contract
they need.&lt;/p&gt;
</content>
 </entry>
 
 
 
 <entry>
   <title>LLM as an Oracle</title>
   <link href="http://huascarsanchez.com/posts/notes/llm-as-an-oracle.html"/>
   <updated>2026-05-13T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/llm-as-an-oracle</id>
   <content type="html">&lt;p&gt;Task-specific LLM evaluation used to mean reference-based metrics and test
suites, with human raters for anything open-ended. Then LLM-as-a-Judge made
judgment cheap, and the conversation shifted to which judge model is best for a
given task. That’s the wrong first question. The first question should be
whether the task needs &lt;em&gt;judgment&lt;/em&gt; or &lt;em&gt;verification&lt;/em&gt;, two evaluation modes that
rely on different kinds of evidence. Judgment fits tasks that are open-ended,
subjective, or otherwise difficult to reduce to executable checks. It relies on
rubrics, preferences, or expert opinion. Verification fits tasks that are backed
by stronger evidence: ground truth, test cases, or expected behavioral
properties like performance.&lt;/p&gt;

&lt;p&gt;Get this wrong and the evaluation fails in both directions. A rubric can tell
you whether an explanation is clear or persuasive, but it’s a weak substitute
for test cases when you’re evaluating code. Exact-match checks against ground
truth fail the other way: they fall apart the moment the output is qualitative.&lt;/p&gt;

&lt;p&gt;Avoiding that mismatch is what led to
&lt;a href=&quot;https://github.com/hsanchez/llm-as-an-oracle&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;llm-as-an-oracle&lt;/code&gt;&lt;/a&gt;. An Oracle
here isn’t an all-knowing model. It’s
an adaptive decision layer that determines whether a task needs judgment or
verification, then routes it to an &lt;em&gt;LLM-as-a-Judge&lt;/em&gt; or an &lt;em&gt;LLM-as-a-Verifier&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;The central claim is simple:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Evaluation should be routed to the strategy that best matches the structure
of the task.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That sounds obvious, but it’s easy to violate in practice. As soon as a
benchmark, agent workflow, or production pipeline standardizes on a single
evaluator, it treats fundamentally different tasks as though they required the
same kind of evidence. Finding the best judge model doesn’t fix that; it just
decides which evaluator gets applied to everything.&lt;/p&gt;

&lt;p&gt;Here’s what that looks like in practice. Later in this post, three agents fix
the same N+1 query bug. Two of them change the query shape. The third wraps the
buggy call in an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;lru_cache&lt;/code&gt; and looks correct — familiar technique, concrete
code, a plausible performance story. A Judge scoring on presentation alone can
be fooled by it. A Verifier running the test suite cannot. That gap between
looking right and being right is what the Oracle is built to catch.&lt;/p&gt;

&lt;h2 id=&quot;the-evaluation-problem&quot;&gt;The evaluation problem&lt;/h2&gt;

&lt;p&gt;Human evaluation remains the reference point for many LLM systems. Humans can
interpret incomplete instructions, account for context, distinguish severity
from style, and notice when an answer is technically correct but pragmatically
poor. But human evaluation scales badly, which created demand for automated
alternatives. Traditional metrics help, but each covers a narrow slice:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;exact match works when the answer space is constrained&lt;/li&gt;
  &lt;li&gt;unit tests work when executable behavior matters&lt;/li&gt;
  &lt;li&gt;overlap metrics work in narrow summarization settings&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Unfortunately, none of these solves the broader evaluation problem on its own.&lt;/p&gt;

&lt;p&gt;Capable instruction-following models added a new option: LLM-based evaluators.
They’ve produced a family of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-*&lt;/code&gt; patterns, each with its own line of
research:&lt;sup id=&quot;fnref:llm-as-family&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:llm-as-family&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Judge&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Verifier&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Critic&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Ranker&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These patterns are often discussed as alternatives. They’re better understood as
tools with different operating assumptions, especially about what evidence is
available. Critics and rankers produce assessments rather than checks, so they
sit on the judgment side; this post focuses on the Judge and Verifier. So the
useful question isn’t which pattern, or which judge model, is best. It’s the one
from the start:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Does this task need judgment or verification?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;judge-and-verifier-solve-different-problems&quot;&gt;Judge and Verifier solve different problems&lt;/h2&gt;

&lt;p&gt;The Oracle routes between two strategies, one for each evaluation mode.&lt;/p&gt;

&lt;h3 id=&quot;llm-as-a-judge&quot;&gt;LLM-as-a-Judge&lt;/h3&gt;

&lt;p&gt;An &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Judge&lt;/code&gt; performs holistic evaluation. It reads the task, a candidate
trajectory (the full solution attempt, not just the final answer), and the
evaluation criteria, then emits a score or preference.&lt;sup id=&quot;fnref:judge&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:judge&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; It’s the natural
fit for judgment tasks, such as:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Is this answer concise without omitting important details?&lt;/li&gt;
  &lt;li&gt;Does this explanation match the user’s level of expertise?&lt;/li&gt;
  &lt;li&gt;Which recommendation is more useful under vague constraints?&lt;/li&gt;
  &lt;li&gt;Is the reasoning persuasive and coherent?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these collapse into executable checks. &lt;em&gt;They require interpretation.&lt;/em&gt;
That doesn’t make a Judge “ask another model what it thinks”: in
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;llm-as-an-oracle&lt;/code&gt;, it scores against weighted rubric criteria and swaps
pairwise order to reduce positional bias.&lt;/p&gt;

&lt;h3 id=&quot;llm-as-a-verifier&quot;&gt;LLM-as-a-Verifier&lt;/h3&gt;

&lt;p&gt;An &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Verifier&lt;/code&gt; evaluates trajectories against signals closer to
correctness than preference.&lt;sup id=&quot;fnref:verifier&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:verifier&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt; It’s the natural fit for verification
tasks, such as:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;code generation with tests&lt;/li&gt;
  &lt;li&gt;question answering with reference answers&lt;/li&gt;
  &lt;li&gt;tool-use traces with expected outputs&lt;/li&gt;
  &lt;li&gt;structured reasoning tasks with decomposable criteria&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Rather than settle for a single coarse score, the Verifier tries to extract
finer-grained signal: it decomposes criteria, repeats verification, and uses
token log probabilities when the provider exposes them.&lt;/p&gt;

&lt;p&gt;The Judge asks which trajectory seems better under a rubric; the Verifier asks
which survives the strongest evidence-sensitive checks available. Related
questions, but not the same one.&lt;/p&gt;

&lt;h2 id=&quot;what-the-oracle-adds&quot;&gt;What the Oracle adds&lt;/h2&gt;

&lt;p&gt;If Judge and Verifier are both useful, a natural response is to expose both and
let the caller choose. That’s necessary but not sufficient. Many workflows mix
task types:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;an agent may produce code patches, explanations, and planning notes&lt;/li&gt;
  &lt;li&gt;a benchmark may combine factual QA, long-form reasoning, and executable tasks&lt;/li&gt;
  &lt;li&gt;a production system may need to evaluate recommendations, SQL, and tool calls
within the same pipeline&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In those settings, asking the caller to pick an evaluator every time creates
friction and invites inconsistency.&lt;/p&gt;

&lt;p&gt;The Oracle removes that step. It inspects the task and its trajectories, picks
the evaluator that fits better, runs only that one, and returns the result
along with an explanation of the choice. It isn’t a third evaluator; it sits
above the two.&lt;/p&gt;

&lt;h2 id=&quot;anatomy-of-the-oracle-router&quot;&gt;Anatomy of the Oracle router&lt;/h2&gt;

&lt;p&gt;The part of the Oracle that makes this choice is the router. The default router
in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;llm-as-an-oracle&lt;/code&gt; is deterministic: it doesn’t call an LLM to decide which
evaluator to use. Instead, it extracts interpretable signals and applies a
fixed chain of routing policies. That’s deliberate: evaluator selection should
become more legible, not less.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/static/figs/llm-as-an-oracle.jpg&quot; alt=&quot;LLM as an Oracle routing diagram&quot; style=&quot;max-width:100%;height:auto;&quot; /&gt;&lt;/p&gt;

&lt;h3 id=&quot;step-1-extract-routing-signals&quot;&gt;Step 1: extract routing signals&lt;/h3&gt;

&lt;p&gt;The router converts the task and trajectories into a structured set of signals.
The current implementation uses features such as:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;has_ground_truth&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;has_test_cases&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trajectory_count&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;stated_difficulty&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;verifiable_keyword_density&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;judgment_keyword_density&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;problem_length&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;output_available&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prior_hardness&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The goal is not to perfectly infer task type from text. The goal is to make the
selection logic explicit enough to inspect, revise, and extend.&lt;/p&gt;

&lt;h3 id=&quot;step-2-collect-policy-votes&quot;&gt;Step 2: collect policy votes&lt;/h3&gt;

&lt;p&gt;Signals are passed through a chain of policies, roughly one per signal group.
Each policy casts a weighted vote for either &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Judge&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Verifier&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;/p&gt;
&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Ground truth or test cases present?  -&amp;gt; favor Verifier
Execution output available?          -&amp;gt; favor Verifier
Previously observed as hard?         -&amp;gt; favor Verifier
Open-ended wording?                  -&amp;gt; favor Judge
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The implementation is more nuanced than that sketch, but the spirit is the
same: evaluation mode is chosen by accumulating evidence.&lt;/p&gt;

&lt;h3 id=&quot;step-3-aggregate-confidence&quot;&gt;Step 3: aggregate confidence&lt;/h3&gt;

&lt;p&gt;The router aggregates the weighted votes into a final confidence score. The
winning strategy is selected only if its confidence is strong enough; otherwise,
the system falls back to the more general-purpose Judge path.&lt;/p&gt;

&lt;p&gt;This separates a strategy’s power from its fit. A Verifier can be the stronger
tool and still be the wrong one when the evidence for this task doesn’t justify
it. That’s a better design than letting every downstream evaluator silently
assume the task suits its own strengths.&lt;/p&gt;

&lt;h3 id=&quot;step-4-expose-the-routing-trace&quot;&gt;Step 4: expose the routing trace&lt;/h3&gt;

&lt;p&gt;The output of a routing decision includes:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;the selected strategy&lt;/li&gt;
  &lt;li&gt;the final confidence&lt;/li&gt;
  &lt;li&gt;the raw signals&lt;/li&gt;
  &lt;li&gt;every policy vote&lt;/li&gt;
  &lt;li&gt;a human-readable reasoning trace&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I care about this part the most, because evaluation pipelines already accumulate
ambiguity, and a score you can’t trace is hard to debug. The trace makes one
critical source of ambiguity observable: why this evaluator was chosen in the
first place.&lt;/p&gt;

&lt;h2 id=&quot;evaluating-trajectories-not-just-answers&quot;&gt;Evaluating trajectories, not just answers&lt;/h2&gt;

&lt;p&gt;The project uses the term &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trajectory&lt;/code&gt; deliberately.&lt;/p&gt;

&lt;p&gt;A trajectory is a candidate task-solving attempt. It may contain:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;the final answer&lt;/li&gt;
  &lt;li&gt;intermediate reasoning or planning&lt;/li&gt;
  &lt;li&gt;code&lt;/li&gt;
  &lt;li&gt;tool calls&lt;/li&gt;
  &lt;li&gt;execution output&lt;/li&gt;
  &lt;li&gt;an optional reward signal&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is especially relevant for agents. When a coding agent fixes a bug, the
thing we care about is not only the final patch. We may care about:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;whether the patch addresses the requested failure mode&lt;/li&gt;
  &lt;li&gt;whether it satisfies explicit requirements&lt;/li&gt;
  &lt;li&gt;whether execution evidence supports the answer&lt;/li&gt;
  &lt;li&gt;whether two superficially plausible solutions differ materially&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For text-only tasks, evaluating the final answer may be sufficient. For agents,
the evaluation object often needs to be richer.&lt;/p&gt;

&lt;p&gt;The Oracle architecture assumes that richer object from the beginning.&lt;/p&gt;

&lt;h2 id=&quot;a-concrete-example-the-n1-query-bug&quot;&gt;A concrete example: the N+1 query bug&lt;/h2&gt;

&lt;p&gt;One example in the repository asks three agents to fix an N+1 query problem.
The original function loads orders first, then issues one SQL query per order to
load items. That means:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;51 queries for 50 orders&lt;/li&gt;
  &lt;li&gt;5,001 queries for 5,000 orders&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The task asks for a constant-query solution and provides both ground truth and
test cases.&lt;/p&gt;

&lt;p&gt;The original bug has this shape:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;get_orders_with_items&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;user_id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;dict&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;db&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;execute&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;&quot;SELECT * FROM orders WHERE user_id = ?&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;user_id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;order&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;order&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;items&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;db&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;execute&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
            &lt;span class=&quot;s&quot;&gt;&quot;SELECT * FROM items WHERE order_id = ?&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
            &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;order&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;id&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]],&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The loop is the problem. The first query fetches the orders, then each order
triggers another query for its items.&lt;/p&gt;

&lt;p&gt;Three candidate trajectories are evaluated:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;one rewrites the query with a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;JOIN&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;one performs a batched prefetch with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;WHERE IN&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;one adds an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;lru_cache&lt;/code&gt; around the inner item lookup&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first two are legitimate fixes, though they make different tradeoffs. The
third sounds plausible because caching often improves performance. But it does
not solve the stated problem. On a cold cache, query count still grows with the
number of orders.&lt;/p&gt;

&lt;p&gt;The misleading fix looks like this:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;o&quot;&gt;@&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;lru_cache&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;maxsize&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;256&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;fetch_items&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;order_id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;dict&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]:&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;db&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;execute&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;&quot;SELECT * FROM items WHERE order_id = ?&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;order_id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;


&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;get_orders_with_items&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;user_id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;dict&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;db&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;execute&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;&quot;SELECT * FROM orders WHERE user_id = ?&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;user_id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;order&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;order&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;items&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;fetch_items&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;order&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;id&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This may help repeated calls for the same order, but it does not change the
first-run query pattern:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;expected: query_count == O(1)
actual:   query_count == 1 + number_of_orders
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;A real fix changes the query shape. For example, the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;JOIN&lt;/code&gt; trajectory uses one
SQL query and groups the rows afterward:&lt;/p&gt;

&lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;SELECT&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;FROM&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;LEFT&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;JOIN&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;items&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;ON&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;order_id&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;WHERE&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;user_id&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;?&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This is exactly the sort of case where evaluator choice matters.&lt;/p&gt;

&lt;h3 id=&quot;why-judge-alone-is-risky-here&quot;&gt;Why Judge alone is risky here&lt;/h3&gt;

&lt;p&gt;A holistic Judge may recognize that the cache-based answer is weaker. But it is
also possible for that answer to benefit from surface plausibility:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;it uses a familiar optimization technique&lt;/li&gt;
  &lt;li&gt;it contains concrete code&lt;/li&gt;
  &lt;li&gt;it appears to address performance&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the score is driven too much by presentation quality, the wrong candidate can
become competitive.&lt;/p&gt;

&lt;h3 id=&quot;why-verifier-is-the-better-fit&quot;&gt;Why Verifier is the better fit&lt;/h3&gt;

&lt;p&gt;The task has stronger evidence:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;explicit correctness requirements&lt;/li&gt;
  &lt;li&gt;test cases&lt;/li&gt;
  &lt;li&gt;expected behavioral properties&lt;/li&gt;
  &lt;li&gt;a measurable performance invariant: query count must not scale with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;N&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is precisely the situation where the Oracle should route toward the
Verifier. The evaluation problem is not mainly aesthetic. It is evidential.&lt;/p&gt;

&lt;p&gt;The interesting part is not merely that Verifier can help. The more general point
is that the Oracle can identify this task shape before evaluation begins.&lt;/p&gt;

&lt;h2 id=&quot;judge-and-verifier-are-not-rivals&quot;&gt;Judge and Verifier are not rivals&lt;/h2&gt;

&lt;p&gt;It is tempting to treat this as a winner-take-all comparison:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Verifier is more objective&lt;/li&gt;
  &lt;li&gt;Judge is more flexible&lt;/li&gt;
  &lt;li&gt;one must be superior&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I do not think that framing is useful.&lt;/p&gt;

&lt;p&gt;Each strategy fails differently.&lt;/p&gt;

&lt;h3 id=&quot;failure-modes-of-judge&quot;&gt;Failure modes of Judge&lt;/h3&gt;

&lt;p&gt;A Judge can overvalue fluent or confident language, blur correctness and
style, and struggle to separate close technical alternatives without stronger
evidence.&lt;/p&gt;

&lt;h3 id=&quot;failure-modes-of-verifier&quot;&gt;Failure modes of Verifier&lt;/h3&gt;

&lt;p&gt;A Verifier has its own failure modes. It inherits bad ground truth, can
overfit to incomplete criteria, becomes brittle when tests are narrow, and
quietly underperforms on tasks that are fundamentally subjective.&lt;/p&gt;

&lt;p&gt;The Oracle does not eliminate these problems. It tries to reduce one avoidable
problem: choosing the wrong mode of evaluation for the task at hand.&lt;/p&gt;

&lt;h2 id=&quot;when-the-oracle-should-ask-for-help&quot;&gt;When the Oracle should ask for help&lt;/h2&gt;

&lt;p&gt;There is another failure mode worth making explicit: sometimes the task itself
is underspecified.&lt;/p&gt;

&lt;p&gt;Suppose three architecture recommendations are all defensible, but the best one
depends on a missing fact about team size, latency goals, compliance constraints,
or deployment environment. No evaluator should pretend confidence if the
information needed to decide was never supplied.&lt;/p&gt;

&lt;p&gt;The fuller design in this project explores a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Human Oracle&lt;/code&gt; escalation path for
those cases. When evaluation evidence runs out, the system can ask a targeted
clarifying question, incorporate the answer, and re-evaluate.&lt;/p&gt;

&lt;p&gt;The point is not to put a person in the loop by default. It is to avoid
manufactured certainty when a decision depends on missing context.&lt;/p&gt;

&lt;p&gt;That same principle motivates the router itself:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;do not hide assumptions&lt;/li&gt;
  &lt;li&gt;do not oversell confidence&lt;/li&gt;
  &lt;li&gt;make uncertainty inspectable&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;what-this-suggests-about-evaluation-design&quot;&gt;What this suggests about evaluation design&lt;/h2&gt;

&lt;p&gt;The Oracle pattern leads to a broader design lesson.&lt;/p&gt;

&lt;p&gt;Good evaluation systems share a few properties. &lt;strong&gt;Evaluator should match task
structure&lt;/strong&gt;: open-ended and evidence-grounded tasks are different, and treating
them the same introduces avoidable error. &lt;strong&gt;Selection should be explicit&lt;/strong&gt;,
part of the system design rather than a hidden convention buried in notebook
code or benchmark glue. &lt;strong&gt;Inspectability should be preserved&lt;/strong&gt;: routing
traces, criteria, and confidence should be artifacts you can actually look at.
And the system should &lt;strong&gt;admit when evidence is insufficient&lt;/strong&gt;. A strong
evaluator is no substitute for missing context, and escalation can be the
right move.&lt;/p&gt;

&lt;h2 id=&quot;where-this-pattern-is-useful&quot;&gt;Where this pattern is useful&lt;/h2&gt;

&lt;p&gt;I think &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-an-Oracle&lt;/code&gt; is especially relevant for:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;coding agents that generate several candidate patches&lt;/li&gt;
  &lt;li&gt;benchmark pipelines that mix factual, creative, and executable tasks&lt;/li&gt;
  &lt;li&gt;tool-using agents whose outputs include both text and action traces&lt;/li&gt;
  &lt;li&gt;research workflows comparing evaluators across heterogeneous task families&lt;/li&gt;
  &lt;li&gt;production systems that want one evaluation interface without pretending every
task should be judged the same way&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In all of those cases, evaluator selection is part of the problem.&lt;/p&gt;

&lt;p&gt;Treating it as a first-class system component is cleaner than standardizing on a
single evaluation method and compensating later with increasingly elaborate
exceptions.&lt;/p&gt;

&lt;h2 id=&quot;closing-thought&quot;&gt;Closing thought&lt;/h2&gt;

&lt;p&gt;The evaluator should fit the task.&lt;/p&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-an-Oracle&lt;/code&gt; is my attempt to turn that principle into a concrete system:
route between judgment and verification, expose the reasons for the choice, and
leave room for human escalation when neither automated path has enough evidence.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:llm-as-family&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Zheng et al. introduce &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Judge&lt;/code&gt;; the
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM-as-a-Verifier&lt;/code&gt; framework develops evidence-sensitive verification;
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;CritiqueLLM&lt;/code&gt; studies critique generation for evaluation; and pairwise
ranking prompting shows how LLMs can be used directly as rankers.
&lt;a href=&quot;https://arxiv.org/abs/2306.05685&quot;&gt;https://arxiv.org/abs/2306.05685&lt;/a&gt;
&lt;a href=&quot;https://llm-as-a-verifier.notion.site/&quot;&gt;https://llm-as-a-verifier.notion.site/&lt;/a&gt;
&lt;a href=&quot;https://arxiv.org/abs/2311.18702&quot;&gt;https://arxiv.org/abs/2311.18702&lt;/a&gt;
&lt;a href=&quot;https://arxiv.org/abs/2306.17563&quot;&gt;https://arxiv.org/abs/2306.17563&lt;/a&gt; &lt;a href=&quot;#fnref:llm-as-family&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:judge&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot
Arena,” 2023. &lt;a href=&quot;https://arxiv.org/abs/2306.05685&quot;&gt;https://arxiv.org/abs/2306.05685&lt;/a&gt; &lt;a href=&quot;#fnref:judge&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:verifier&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;“LLM-as-a-Verifier: A General-Purpose Verification Framework.”
&lt;a href=&quot;https://llm-as-a-verifier.notion.site/&quot;&gt;https://llm-as-a-verifier.notion.site/&lt;/a&gt; &lt;a href=&quot;#fnref:verifier&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
 </entry>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <entry>
   <title>My book is out!</title>
   <link href="http://huascarsanchez.com/posts/notes/book.html"/>
   <updated>2015-03-05T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/book</id>
   <content type="html">&lt;h2 id=&quot;software-patterns-knowledge-maps-and-domain-analysis&quot;&gt;Software Patterns, Knowledge Maps, and Domain Analysis&lt;/h2&gt;

&lt;p&gt;My book is about domain analysis and software patterns. Briefly, it shows how
to build software systems as systems of patterns, called Knowledge Maps.&lt;/p&gt;

&lt;p&gt;It is now in the global markets. Its publication date was December 2014. Please
check it out:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;At &lt;a href=&quot;http://tinyurl.com/SoftwarePatterns&quot;&gt;&lt;strong&gt;Amazon&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;At &lt;a href=&quot;http://www.crcpress.com/product/isbn/9781466571433&quot;&gt;&lt;strong&gt;CRC Press&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;At &lt;a href=&quot;http://www.taylorandfrancis.com/books/details/9781466571433/&quot;&gt;&lt;strong&gt;Taylor &amp;amp; Francis&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;figure&gt;
    &lt;img style=&quot;border: 1px solid #e5e5e5;&quot; src=&quot;http://www.huascarsanchez.com/static/K16540.jpg&quot; alt=&quot;Software Patterns, Knowledge Maps, and Domain Analysis&quot; height=&quot;684&quot; width=&quot;432&quot; /&gt;
    &lt;figcaption style=&quot;padding-left: 2cm;&quot;&gt;Figure 1. Front cover of my book&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;The citation (in BibTex) of this book is presented as follows:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;@book{hsanchez, 
      author = {M. E. Fayad, H. A. Sanchez, S. G .K. Hegde, 
                A. Basia, and A. Vakil.},
      title  = {Software Patterns, Knowledge Maps, and 
                Domain Analysis},
      publisher = {Auerbach Publications},
      year      = 2014,
      address   = {Boca Raton, Florida},
      isbn      = {978-1466571433}
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
</content>
 </entry>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <entry>
   <title>Cyberbullying detection in social networks</title>
   <link href="http://huascarsanchez.com/posts/notes/twitter-bullying-detection-report.html"/>
   <updated>2012-04-29T00:00:00+00:00</updated>
   <id>http://huascarsanchez.com/posts/notes/twitter-bullying-detection-report</id>
   <content type="html">&lt;p&gt;According to Google Analytics and to other sources, my data mining (ISM 245 class) project titled “Twitter Bullying Detection” is getting some attention. An earlier version of the project report was hosted on the co-author of the project’s &lt;a href=&quot;http://users.soe.ucsc.edu/~shreyask/index.html&quot;&gt;website&lt;/a&gt;. This version
is good in terms of communicating the project’s main idea and motivation. However, I think it is always important to have the latest version available. Therefore, I decided to post this latest version &lt;a href=&quot;http://dl.dropbox.com/u/11902217/huascarsanchez_and_shreyaskumar-twitterbullyingdection-fnl.pdf&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Please feel free to contact me if you have any questions about the content of
that report. Thanks.&lt;/p&gt;
</content>
 </entry>
 
 
 
 
 
 

</feed>
