Huascar Sanchez Posts

What Do We Mean by LLM Chemistry?

2026-10-06T00:00:00+00:00

Why picking different models isn’t the same as picking models that work well together

When I started working on LLM Chemistry, I was interested in a fairly practical question: if you have several LLMs available, from different providers and with different strengths and weaknesses, how do you know which ones should work together?

Picking the strongest models seems reasonable, right? Picking models with different strengths also seems reasonable. But neither tells you what will actually happen when those models start collaborating, or whether they will make each other better. That’s the gap I was trying to capture with LLM Chemistry.

The easiest way to explain the idea is with three questions:

How good is a model? That’s capability.
How different are two models? That’s diversity.
What happens to one model’s contribution when another model joins the team? That’s chemistry.

That last question is subtle. Two models can look very similar and still make a terrible team. They might reinforce the same errors, contribute redundant information, or simply fail to combine their strengths. Conversely, two models that look dramatically different in capability and performance might interact surprisingly well. The difference between the models matters, but so does what happens when those differences meet. That’s the idea behind LLM Chemistry.

So one of the ideas behind LLM Chemistry is:

Diversity tells us how different models are. Chemistry tells us how those differences matter when they work together.

Since a former colleague and I published the original LLM Chemistry work,¹ I’ve come across other research that makes this distinction especially interesting. One paper, “You’re Hired: Strategic Model Selection for LLM Collaboration,” directly studies how to choose models for collaborative teams. Another, “Emergent Alignment via Competition,” asks what happens when differently motivated AI systems compete for a user’s decisions.

They all involve heterogeneous collections of models, but they don’t all ask the same question, and even when they do, they approach it differently. Looking at them side by side provides a useful way to understand what LLM Chemistry is and what it isn’t.

Chemistry is about interaction

Suppose we have three models: A, B, and C.

A is our strongest individual model. If we’re ranking models by performance, A is an obvious choice. B has a very different performance profile. Perhaps it’s better at something A struggles with. A diversity-based strategy might therefore conclude that A and B are a promising pair.

But we still don’t know whether B actually helps A. Maybe B catches errors that A consistently misses. Maybe its critique causes A to reconsider a bad answer. Or maybe B introduces noise and makes the final result worse.

Knowing that A and B are different doesn’t tell us which of those things will happen. LLM Chemistry approaches this by looking at marginal contribution.

Suppose we have an existing group of models. We can ask how much adding A changes the performance of that group. In the paper, this is expressed using a benefit function derived from the change in the cost of answering a user query:

\[benefit_Q(X,Y)=cost_Q(Y)-cost_Q(X \cup Y).\]

Here, \(Y\) is the existing group and \(X\) is the set being added. The benefit can even be negative: adding a model can increase the cost and degrade performance.

Okay, now add B to the picture. Instead of asking whether A and B are similar, we ask:

Does the value contributed by A change when B is present?

Dropping \(Q\) and writing \(X\) for the existing group, we compare:

\[benefit(A,X)\]

with:

\[benefit(A,X\cup\{B\}).\]

The difference between those quantities gives us what I call an interaction effect:

\[\Delta(A,B,X)=|benefit(A,X)-benefit(A,X\cup\{B\})|.\]

The full chemistry definition takes the maximum of this interaction effect over possible surrounding configurations \(X\), normalized by the cost of the resulting configuration:

\[chem_Q(A,B,S)=\max_{X\subseteq S\setminus\{A,B\}}\frac{\Delta(A,B,X)}{cost_Q(X\cup\{A,B\})},\]

where \(S\) is the set of candidate models. This is Equation 2 in the paper.

The equation matters, but the intuition is simpler:

B matters to A if A’s contribution changes because B is there.

And that’s the relationship I’m trying to detect.

Where diversity enters the picture

So far, I’ve used diversity mostly as a contrast: the thing chemistry is not. But that doesn’t mean diversity is irrelevant. If chemistry measures how models interact, where do their differences come in? This is where one of the theoretical results from the original paper becomes important.

Theorem 1 says that chemistry emerges when models exhibit heterogeneous performance profiles. If models have identical—or nearly identical—performance profiles, the cost-based selection pressure disappears and there is no interaction effect for the framework to detect.

In other words, performance heterogeneity creates the conditions under which chemistry can emerge. The corollary goes further and connects chemistry to the amount of performance diversity, while noting that the direction of that relationship depends on the task. This distinction is important:

Diversity is not irrelevant to chemistry. It is one of the conditions that makes chemistry possible.

But diversity alone doesn’t tell us what the resulting interaction means for the team. The same heterogeneous performance profiles can produce useful complementarity, or they can produce interactions that don’t improve the final result.

The experiments reflect the task dependence the corollary predicts. Chemistry-based selection improved effectiveness in some tasks and group sizes, was effectively neutral in others, and underperformed in one small-group summarization setting. So a more precise way to put the relationship is:

Diversity is a necessary condition for chemistry, but it is not sufficient. Chemistry is what tells us whether the diversity is useful.

This is why I wouldn’t summarize the idea as “choose the most diverse models.” That would be misleading. The real question is what those differences do once the models interact.

Chemistry is also a selection signal

Measuring interaction effects is useful diagnostically, but that wasn’t the final goal of the work. The practical objective was still the question I started with:

Which models should work together?

That’s why LLM Chemistry includes two related pieces: the CHEME and RECOMMEND algorithms. CHEME estimates chemistry among candidate models by measuring interaction effects across configurations. RECOMMEND uses those estimates to search for model combinations that capture strong chemistry while accounting for the cost of the resulting team.

The paper describes the objective as recommending a set of LLMs that maximizes the benefit of collaboration while minimizing the cost of answering the query. Conceptually, the pipeline looks like this:

Read left to right, the figure shows chemistry being estimated from past performance and then used to choose the team. LLM Chemistry isn’t only a way to measure how models interact; it turns that interaction into information for team selection.

A newer paper asks a familiar question

Treating chemistry as a selection signal is what made “You’re Hired: Strategic Model Selection for LLM Collaboration” by Cao et al. particularly interesting to me.

The paper asks:

How should we select models to form an effective collaborative team?

Its setup is straightforward. Given a candidate model pool \(M\) and a desired team size \(n\), a selector chooses a subset \(T\):

\[\sigma(M,n)=T.\]

At that level, the problem should sound familiar. LLM Chemistry also asks how we can identify which models should collaborate. In fact, our original paper explicitly frames its central question as identifying which LLMs “work best together” and introduces RECOMMEND for selecting model combinations with strong chemistry.

The interesting difference is how the two approaches characterize potential collaborators. The “You’re Hired” paper explores a broad taxonomy of signals: model descriptions, behavioral idiosyncrasies, Item Response Theory representations, benchmark performance profiles, combinations of capability and description information, and LLM-based recruiters. In the performance-profile approach, for example, models are represented by performance vectors across benchmarks and compared through the similarity of those vectors. That’s close to the diversity signal I discussed earlier: useful for telling models apart, but not on its own a measure of how they’ll interact.

Put simply, many of these approaches ask:

What do we know about these models that might tell us who should be on the team?

LLM Chemistry, on the other hand, comes at the problem from another direction:

What have we learned from what happens when these models are actually together?

One reasons from model characteristics toward the team. The other reasons from observed interaction toward the team. Those aren’t mutually exclusive approaches. In fact, they may ultimately be complementary.

“Team composition cannot be decoupled from team interaction”

One result in the “You’re Hired” paper stood out to me in particular. The authors found that a selection method that produces a strong team under one collaboration mechanism can perform poorly under another. They concluded:

“team composition cannot be decoupled from team interaction.”

By “team interaction,” the authors mean the collaboration mechanism (how the team refines, fuses, or merges its outputs) rather than interaction effects in the sense I’ve used here. But the lesson points in the same direction.

That result is interesting when viewed through the lens of chemistry. If collaborative performance were determined primarily by individual model strength, we could rank the models and select the top \(n\). If it were determined primarily by diversity, we could maximize some measure of difference. But if the best team changes with how the models work together, neither capability nor diversity is enough on its own, and that’s the gap chemistry is meant to fill.

Different doesn’t necessarily mean useful

The “You’re Hired” paper also illustrates why diversity alone can be tricky. As the authors increase the diversity of the available model pool, additional distinct models initially help some selectors. But eventually the gains can saturate or reverse.

A weak model can also be an unusual model.

A selection method that rewards difference without sufficiently accounting for capability can therefore select weak outliers precisely because they look distinctive. The paper concludes that maximizing distinctness alone isn’t enough; effective selection has to balance diversity with capability.

This brings us back to the three questions from the start:

Capability: How good is this model?
Diversity: How different are these models?
Chemistry: How does one model’s contribution change in the presence of another?

They’re related questions, but they aren’t interchangeable. And this makes the complementarity I mentioned earlier more concrete: capability and diversity are built from per-model signals, so collecting them scales with the size of the pool rather than the number of possible teams. That makes them natural for narrowing a large pool, while interaction-aware estimation could help distinguish among the remaining combinations. I’ll come back to that.

What about “Emergent Alignment via Competition”?

“You’re Hired” shows that differences in capability can saturate or mislead. But models can also differ in what they’re trying to achieve, and that’s where a second paper, “Emergent Alignment via Competition,” comes in. At first glance, it looks related: it also considers heterogeneous AI systems and explicitly motivates a world in which models come from different providers with potentially different incentives. But the similarity is mostly at the setting level.

The paper asks whether a human interacting with multiple differently misaligned AI agents can nevertheless obtain outcomes comparable to interacting with a perfectly aligned model.

Its central assumption is that the user’s utility function can be approximated by a weighted combination of the AI agents’ utility functions—roughly, that it lies in their convex hull. The mechanism is then strategic competition.

So although both papers involve heterogeneous collections of models, they’re trying to accomplish different things.

LLM Chemistry asks:

Which models work well together on a collaborative task?

“Emergent Alignment via Competition” asks:

Can competition among differently aligned models produce outcomes that are good for the user?

The first is fundamentally a collaborator-selection and interaction problem. The second is fundamentally a strategic alignment and mechanism-design problem.

That’s an important distinction because it shows just how many different things we can mean when we talk about “diverse models.”

Diversity isn’t one thing

Putting these papers next to each other helped clarify something for me. We often talk about model diversity as though it were a single property. It isn’t.

Models can differ in many ways: in architecture and training, in capability and observed behavior, in performance profiles, and in their objectives or incentives. Those differences aren’t interchangeable.

In “Emergent Alignment via Competition,” the important heterogeneity is primarily in utility functions and incentives, while in “You’re Hired,” candidate models are characterized through capability, behavior, descriptions, performance profiles, and related signals.

In LLM Chemistry, the theoretical connection is specifically to heterogeneous performance characteristics—the quality and accuracy profiles used by the framework—and to the interaction dependencies that emerge from them.

That suggests that instead of asking:

Should we use diverse models?

we might ask:

Different in what way?

Then:

Different for what purpose?

And finally:

What happens when those differences meet?

That last question is where chemistry begins.

The question I think comes next

Earlier, I said I’d come back to how these signals might work together. Comparing these papers leaves me with a question I find more interesting than deciding whether capability, diversity, or chemistry is the “right” selection signal:

When can capability and diversity predict chemistry—and when do we actually need to measure interaction?

Measuring interaction is the expensive part. To learn how models affect one another, we need evidence from collaborative configurations, and the number of possible configurations grows combinatorially with the size of the pool. Capability and diversity signals, by contrast, are collected per model. That suggests a possible hierarchy:

In that picture, capability and diversity aren’t alternatives to chemistry. They could serve as priors for where chemistry is worth measuring.

That opens several other questions:

Does chemistry remain stable when the collaboration mechanism changes?
Can two models have useful chemistry for one task and poor chemistry for another?
How does chemistry change when one of the underlying models is updated?
If chemistry-, capability-, and diversity-based selectors were compared on the same pools, tasks, and collaboration mechanisms, when would each one win?

Those are empirical questions. But the conceptual question underneath them is the same one that motivated LLM Chemistry in the first place. When building a multi-LLM system, it isn’t enough to ask how good the models are, or how different they are. We also need to ask:

What happens to their contributions when they work together?

That’s what I mean by LLM Chemistry.

Footnotes

In hindsight, we probably should have titled the paper “LLM Chemistry Estimation for Multi-LLM Team Recommendation.” That would have been more accurate, but it’s a mouthful, so we dropped “Team.” That was probably a mistake: without it, it’s easy to read the work as recommending individual models, when the point is recommending models that work well together. ↩

The Anatomy of an AI Harness

2026-10-04T00:00:00+00:00

Over the past few months, I have built several AI harnesses to ensure agent reliability and safety. While the scope of a minimum viable AI harness is still fresh in my mind, I wanted to write about its anatomy.

In a nutshell, an AI model generates outputs, while an AI harness turns those outputs into controlled, stateful execution. That distinction matters.

A model can answer a question or generate a tool call, but something else must maintain context, execute the requested action, return the result, decide whether another invocation is allowed, and determine when the process is complete. That something is the AI harness.

At its foundation, the anatomy is surprisingly small:

flowchart LR
    Harness[AI Harness]
    Harness --> Model[Model]
    Harness --> State[State]
    Harness --> Capabilities[Capabilities]
    Harness --> Loop[Execution Loop]
    Harness --> Control[Control]

Or, more compactly:

AI harness = state + model + capabilities + execution loop + control.

Everything more sophisticated — e.g., memory, planning, retrieval, subagents, sandboxing, evaluation — can grow from this foundation.

The Core Anatomy

Model

The model serves as the decision-making engine inside the harness. Whether it is a large language model, a multimodal model, or another system that generates outputs from inputs, the harness interacts with it to determine next steps.

From the harness’s perspective, this interface can be remarkably simple:

response = model.generate(
    messages=state,
    tools=available_tools
)

Given the current context and available capabilities, the model produces either a direct response or a request to perform an action. Crucially, however, the model does not carry out the action itself:

The model proposes what happens next. The harness makes it executable.

The model is therefore a component of the harness, not the harness itself.

State

A model invocation only knows what is placed in its context. The harness must therefore preserve enough information about what has already happened. At minimum:

User:
    Find the problem in this program.

Assistant → Tool:
    read_file("main.py")

Tool:
    <contents of main.py>

Assistant:
    ...

Each action produces new information that can influence the next model invocation. State gives the execution continuity.

More sophisticated systems may introduce working memory, task state, checkpoints, artifacts, or context compression, but these are extensions of the same basic idea:

State tells the next model invocation what has happened so far.

Capabilities

While state provides context and the model decides what happens next, text generation alone does not allow a model to interact with its environment.

Capabilities provide the mechanisms to take real-world action. They define the tools, APIs, and interfaces available to the harness to execute the model’s intent. They might include operations such as:

read_file(path)
write_file(path, content)
search(query)
run_command(command)
query_database(sql)
call_api(...)

The harness exposes descriptions of these capabilities to the model. When the model requests one, the harness interprets the request, validates it, executes the corresponding operation, and returns the result.

This creates a fundamental boundary: the model decides what it wants to do while the harness determines how that action actually happens.

That boundary becomes especially important once permissions, security, sandboxing, or human approval enter the picture.

Execution Loop

The execution loop is the heart of the harness. A minimal version might look like this:

while True:
    response = model.generate(state, tools)

    if response.is_final:
        return response.text

    for call in response.tool_calls:
        result = tools.execute(call)
        state.add(call, result)

Without the loop, interaction looks roughly like:

flowchart LR
    Input --> Model --> Output

With the harness:

flowchart LR
    input --> model
    model --> action
    action --> observation
    observation --> action
    action --> final

The model can act, observe the result, reconsider what to do, and act again.

This seemingly small architectural change is fundamental. A model invocation becomes an ongoing interaction with an environment.

Control: Who Governs the Loop?

Left unchecked, a model will happily loop indefinitely or burn through an API budget. The harness sets the boundaries.

At minimum, execution stops when the model emits a final answer. In practice, the harness also enforces operational guardrails:

Step limits (to catch runaway loops)
Execution timeouts and cost budgets
Tool permissions and human approval gates
User cancellation
Retry policies and error limits

The model proposes what it wants to do, but the harness holds the kill switch.

Following One Task Through the Harness

Consider a user asking:

Find the bug in this repository.

The harness receives the request and initializes state. It then invokes the model with the current state and available capabilities. E.g., the model decides that it needs to inspect the repository:

Model
  ↓
list_files()

The harness executes the tool:

Tool
  ↓
src/
  main.py
  parser.py
tests/
  test_parser.py

That observation enters the state. After that, runs again and decides to inspect a file:

read_file("src/parser.py")

The harness executes the request, records the result, and invokes the model again. Then, the cycle continues:

flowchart TD
    %%{init: {"flowchart": {"rankSpacing": 17, "padding": 7}, "themeVariables": {"fontSize": "14px"}}}%%
    request[User request] --> initial_model[Model]
    initial_model --> action[Action]
    action --> execution[Execution]
    execution --> observation[Observation]
    observation --> state[State update]
    state --> next_model[Model]
    next_model --> continuation[...]

Eventually, the model determines that it has enough information and produces a final response.

The harness did not need a predefined debugging workflow. Instead, it provided something more general:

A controlled environment in which a model can repeatedly decide what to do next based on what has happened previously.

The First Extension: Observability

While these five components are enough to build a working harness, one addition becomes valuable almost immediately: an event log.

That is, instead of seeing a run as:

flowchart LR
    Task --> Answer

record the execution:

Run
 │
 ├── user_message
 ├── model_request
 ├── model_response
 ├── tool_call
 ├── tool_result
 ├── model_request
 ├── model_response
 └── final

Now the harness can answer questions such as:

What did the model see?
What action did it choose?
What did the tool return?
Where did execution fail?
How many model calls occurred?
How much did the run cost?

This trace becomes the foundation for debugging, replay, monitoring, evaluation, and performance analysis. A minimal practical harness therefore looks something like:

flowchart TD
    state[State] --> loop[Execution Loop]
    loop --> model[Model]
    loop --> tools[Tools]
    model --> log[Event Log]
    tools --> log

Everything Else Builds on the Core

Production systems naturally get much more complex, layering on things like planning, long-term memory, retrieval, sandboxing, human approvals, and multi-agent orchestration. But while those capabilities matter, none of them are required to define a harness. Keeping this distinction clear prevents “AI harness” from becoming synonymous with an entire agent framework: a harness can be lean, defined purely by the machinery that turns model decisions into controlled execution.

Where Does the Agent Live?

This raises a more interesting question: if the model isn’t the agent, and the harness isn’t the agent, where does the agent actually live?

It is tempting to imagine:

flowchart TD
    harness[Harness] --> agent[Agent]
    agent --> model[Model]

But agent behavior is better understood as emerging from the interaction of several components:

flowchart LR
    model[Model] --> combined((+))
    instructions[Instructions] --> combined
    state[State] --> combined
    capabilities[Available capabilities] --> combined
    semantics[Harness execution semantics] --> combined
    combined --> behavior[Agent behavior]

This distinction has practical consequences. Change the model, the instructions, or the available tools, and behavior will naturally shift. But change the harness, and behavior changes just as fundamentally.

A harness that permits unlimited actions creates a very different effective agent from one that requires approval for consequential operations. A harness that retains the complete interaction history produces different behavior from one that aggressively compresses context. And a harness that retries failed actions behaves quite differently from one that immediately terminates.

The harness is not merely plumbing around an otherwise complete agent—its execution semantics directly shape how the resulting agent behaves.

The Anatomy in One Picture

Strip away planners, memory systems, retrieval pipelines, subagents, and orchestration, and the architecture becomes remarkably compact:

flowchart TD
    S[State] --> M[Model]
    M -->|decides next action| C[Capability]
    C -->|result| S

    CTRL[Control] -.->|governs invocation| M
    CTRL -.->|governs execution| C

    S -.-> LOG[Event Log]
    M -.-> LOG
    C -.-> LOG

The model makes the decisions, state preserves continuity, and capabilities let it interact with the outside world. The execution loop turns those discrete steps into an ongoing run, control keeps the process within bounds, and observability lets you see what actually took place.

That is the basic anatomy:

An AI harness is the machinery that turns model inference into controlled, stateful execution.

And that’s it. Everything else is an extension of this foundation.

Gating Your LLM Eval Rubric with Evidence

2026-05-26T00:00:00+00:00

Part 2: Managing Your LLM Eval Rubric Like Code

Promptory’s staged releases, evidence documents, and release gates let you attach eval results to a rubric candidate and block promotion until that evidence passes. This post shows the workflow step by step. The full example is in examples/evals2.py.

The Gate Declaration

Add a release_gates block to promptspec.yaml before releasing any candidates:

files:
  - rubric.yaml
required_variables:
  - criteria
max_file_bytes: 100000
release_gates:
  evidence:
    - kind: eval
      name: eval-run
      required_status: pass

Any promotion attempt that lacks an evidence document of kind: eval, name: eval-run, and status: pass raises PromptGateError and leaves current.json unchanged.

Step 1: Stage a Candidate

A staged release writes an immutable snapshot under versions/ without updating current.json. The active version is unaffected while the candidate is evaluated.

from promptory.manager import PromptManager

manager = PromptManager("prompts")

draft = manager.prompts_dir / "drafts" / "rubric.yaml.j2"
draft.write_text(BASIC_RUBRIC)

v1 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
  staged=True,
)
# v0.0.1 exists in versions/ but current.json is unchanged.

Step 2: Attach Evidence

Load the staged rubric, run your eval harness, then write the results as an evidence document and attach it to the version.

from promptory import PromptStore
from promptory.evidence import add_evidence

store = PromptStore("prompts")
rubric = store.load("rubric.yaml", version=v1)

metrics = run_eval_harness(rubric)

evidence_doc = {
  "kind": "eval",
  "name": "eval-run",
  "status": "fail",          # below threshold: 60% accuracy
  "tool": "eval-harness",
  "created_at": "2026-05-28T12:00:00Z",
  "summary": "accuracy=60%",
  "metrics": metrics,
}
evidence_path.write_text(json.dumps(evidence_doc))
add_evidence(manager.spec(), v1, evidence_path)

Evidence is immutable once attached.

Step 3: Check the Gate

gate = manager.gate(v1)
# False — evidence status is fail, required pass
print(gate.passed)

v1 stays staged.

Step 4: Iterate

Revise the rubric draft and stage a new candidate.

draft.write_text(STRICT_RUBRIC)

v2 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
  staged=True,
)

rubric_v2 = store.load("rubric.yaml", version=v2)
metrics_v2 = run_eval_harness(rubric_v2)

evidence_doc_v2 = {
  "kind": "eval",
  "name": "eval-run",
  "status": "pass",          # 100% accuracy clears the threshold
  "tool": "eval-harness",
  "created_at": "2026-05-28T13:00:00Z",
  "summary": "accuracy=100%",
  "metrics": metrics_v2,
}
add_evidence(manager.spec(), v2, evidence_path_v2)

gate_v2 = manager.gate(v2)
print(gate_v2.passed)   # True

Step 5: Compare Evidence

compare_evidence diffs the attached evidence between two versions.

from promptory.evidence import compare_evidence

comparison = compare_evidence(manager.spec(), v1, v2)

for change in comparison.changes:
  print(f"[{change.kind}] {change.name}: \
    {change.before_status} -> {change.after_status}")
  for metric in change.metrics:
    print(f"  {metric.name}: {metric.before} -> {metric.after}")

Output:

[eval] eval-run: fail -> pass
  accuracy: 0.6 -> 1.0
  false_positive_rate: 1.0 -> 0.0
  mean_score: 4.6 -> 3.4

Step 6: Promote

require_gates=True makes the gate check part of the promotion call. If evidence is missing or fails, the call raises PromptGateError and current.json is not updated.

manager.promote(v2, require_gates=True)
print(store.current_version())   # v0.0.2

v1 remains staged. v2 is now current.

Running the Example

uv run python examples/evals2.py

Expected output:

[v1 basic] staged v0.0.1
  accuracy=60%  fpr=100%  fnr=0%
  gate: FAIL - candidate stays staged

[v2 strict] staged v0.0.2
  accuracy=100%  fpr=0%  fnr=0%
  gate: PASS

Evidence comparison v0.0.1 -> v0.0.2:
  [eval] eval-run: fail -> pass
    accuracy: 0.6 -> 1.0
    false_positive_rate: 1.0 -> 0.0
    mean_score: 4.6 -> 3.4

Promoted v0.0.2 -> current: v0.0.2

What Promptory Does Not Do

Promptory stores and validates the evidence document shape. It does not run evals, call LLMs, define metric thresholds, or decide whether a rubric is correct.

Managing Your LLM Eval Rubric Like Code

2026-05-19T00:00:00+00:00

Promptory manages your eval rubric as a versioned prompt artifact: drafts, immutable releases, template variables, and version history. This post shows how to draft, release, and load rubric versions using the Python API.

The full runnable example is in examples/evals.py.

The Rubric Template

A rubric prompt in Promptory is a Jinja template. Declare the file and its required variables in promptspec.yaml:

files:
  - rubric.yaml
required_variables:
  - criteria
max_file_bytes: 100000

The draft template lives in prompts/drafts/rubric.yaml.j2:

model: claude-sonnet-4-6
temperature: 0.0
system_prompt: |
  Evaluate the response on {{ criteria }}.
  Be generous; give partial credit for effort.
  Return JSON: {score: int, rationale: str}.

{{ criteria }} is a template variable. It is rendered at release time, not at runtime.

Template Variables

Release the same rubric template with different variable values to produce separate versioned artifacts for each eval dimension.

from promptory.manager import PromptManager

manager = PromptManager("prompts")

v_helpfulness = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
)
v_factual     = manager.release(
  bump="patch",
  variables={"criteria": "factual accuracy"},
)

Each release is an independent rendered artifact. Loading v_helpfulness always returns the rubric with criteria="helpfulness" regardless of subsequent releases.

Draft and Release Separation

prompts/drafts/ holds the working template. prompts/versions/ holds immutable rendered releases. Editing the draft and cutting a new release does not modify any previous release.

BASIC_RUBRIC = (
  "model: claude-sonnet-4-6\n"
  "temperature: 0.0\n"
  "system_prompt: |\n"
  "  Evaluate the response on {{ criteria }}.\n"
  "  Be generous; give partial credit for effort.\n"
  "  Return JSON: {score: int, rationale: str}.\n"
)

STRICT_RUBRIC = (
  "model: claude-sonnet-4-6\n"
  "temperature: 0.0\n"
  "system_prompt: |\n"
  "  Evaluate the response on {{ criteria }}.\n"
  "  Penalize vague or incomplete answers.\n"
  "  Return JSON: {score: int, rationale: str}.\n"
)

# v1: initial rubric.
draft.write_text(BASIC_RUBRIC)
v1 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
)

# v2: updated rubric. v1 is unchanged.
draft.write_text(STRICT_RUBRIC)
v2 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
)

The directory prompts/versions/v0.0.1/ is never touched after creation.

Loading by Version

PromptStore.load accepts an optional version argument. When supplied, it reads from that exact release directory regardless of which version is current.

from promptory import PromptStore

store = PromptStore("prompts")

rubric_v1 = store.load("rubric.yaml", version=v1)
rubric_v2 = store.load("rubric.yaml", version=v2)

# Pass rubric["system_prompt"] to your eval harness.

The result is identical whether v1 is current or not.

Version History

PromptStore.list_versions returns all available releases in semantic version order.

print(store.list_versions())
# ['v0.0.1', 'v0.0.2', 'v0.0.3']

Use this to scan all candidates and compare results across versions.

version      criteria               accuracy      fpr      fnr
v0.0.1        helpfulness              60%      100%        0%
v0.0.2        helpfulness             100%        0%        0%
v0.0.3        factual accuracy        100%        0%        0%

v0.0.2 has the best metrics for helpfulness. v0.0.3 applies the same rubric instructions to a different eval dimension without re-authoring the template.

Running the Example

uv run python examples/evals.py

The example uses a simulated judge and fixed benchmark data. Replace simulate_judge_call with a real LLM call to run it against your own eval suite.

What Promptory Does Not Do

Promptory manages the rubric prompt. It does not run evals, call LLMs, manage benchmark datasets, or decide whether one rubric is better than another.

Part 2: Gating Your LLM Eval Rubric with Evidence

Promptory, a Git-based prompt versioning system

2026-05-14T00:00:00+00:00

In many organizations building GenAI solutions, prompts have become production artifacts. However, despite their importance, many teams still treat them like ordinary strings. Treating them this way might work for a while, but as the prompts evolve, as their performance becomes variable across different models, model versions, safety policies, or use cases, it becomes more complex and harder to manage them as strings.

That is the motivation behind Promptory: a Git-based prompt versioning system that gives prompt changes the same basic engineering discipline as code. The core idea is simple:

Author prompts as editable drafts, release them as immutable rendered artifacts, and make runtime code load only released versions.

The prompt management problem

Prompt changes are deceptively small. Changing a few words can alter:

how much context the model uses
whether it follows a tool policy
how it handles uncertain answers
whether it refuses unsafe requests
which format downstream parsers expect
how it behaves under evals

In traditional software engineering, anything that can change production behavior is managed deliberately: version control, review, tests, releases, and rollback. I believe that prompt engineering should not be exempt from that discipline just because the artifact is text. On the contrary, it should also be managed deliberately.

The hard part is choosing the right amount of discipline. In practice, teams often land on one of two extremes. The first one is the hardcoded prompt:

SYSTEM_PROMPT = """
You are a helpful assistant. Answer concisely.
"""

Hardcoding a prompt like this is fine for a prototype, but it’s brittle for a system that needs repeatable experiments, reviewable changes, or rollback.

The second one is overcorrection. This is the process of adopting a large hosted prompt platform before the team actually needs one. Those systems can be useful when non-technical users need a UI, many teams need centralized governance, or the current prompt metadata no longer fits a particular repository workflow.

I strongtly believe fast-moving R&D teams often need something smaller:

plain files
Git review
CI-friendly checks
immutable releases
a clear pointer to the active version
runtime loading that never reads work-in-progress drafts

Promptory is aimed at that middle ground.

Why Git alone is not enough

Why not just use Git? That’s the obvious pushback I get when socializing Promptory. Git already handles reviewing and preserving prompt history just fine, they say. That is true. However, Git doesn’t define a prompt lifecycle.

In other words, Git can tell you that a prompt file changed, but it cannot tell your application:

which rendered prompt version is active
whether a prompt has unresolved template variables
whether all managed prompt files were released together
whether a runtime process is reading drafts or released artifacts
how to roll back without rewriting prompt history

That is why Promptory keeps Git as the durable history and adds a small release model on top of it.

Why not a central registry?

Promptory’s answer is a deliberate ownership choice: prompts live with the code that uses them. This builds on the same locality of change principle from software engineering — normally applied to code — and extends it to prompts. Specifically, if a prompt shapes a service’s behavior, it should be reviewed, released, and rolled back with that service.

A central registry can be useful for organization-wide discovery, shared prompts, or non-engineering workflows. But not so much for application-specific prompts, which should live next to the implementation to make review and rollback simpler. The prompt diff, code diff, tests, and release artifact all move through the same Git workflow.

This ownership choice doesn’t rule out central visibility. In production, teams can still publish released prompt artifacts to object storage, an internal artifact registry, or Promptory’s read-only registry service. The repo remains the source of truth; production systems only aggregate or serve released artifacts from there.

In the repo, that looks something like this:

prompts/
  drafts/
    system.yaml.j2
    input_guardrail.yaml.j2
    output_guardrail.yaml.j2
  versions/
    v0.1.0/
      system.yaml
      input_guardrail.yaml
      output_guardrail.yaml
      metadata.json
  current.json
  promptspec.yaml

The above layout breaks down into four parts, each with a distinct role:

drafts/ contains editable Jinja templates
versions/ contains rendered release artifacts
current.json points to the active release
promptspec.yaml declares which prompt files are managed

Two of those four parts carry the real distinction. Developers edit drafts. Applications load versions. That boundary, simple as it sounds, is the main design decision in the project.

Drafts are for authoring

A draft is where prompt authors and coding agents work. It’s written in plain YAML. For example:

# prompts/drafts/system.yaml.j2
model: gpt-5.5
temperature: 0.2
system_prompt: |
  You are a helpful assistant.
  Answer concisely and avoid unsupported claims.

Because drafts are Jinja templates rather than static text, they can also include release-time variables:

# prompts/drafts/message.yaml.j2
message: |
  Hello {{ user_name }}.
  Generated at {{ generation_time }}.

Promptory renders these with Jinja’s StrictUndefined, on purpose. A missing variable throws an exception instead of quietly becoming an empty string. Silent prompt rendering failures are hard to debug because they often look like model behavior problems later.

Rendering itself happens at release time, through two commands:

uv run prompt check
uv run prompt release --patch

prompt check catches problems early: bad drafts, missing variable declarations. prompt release does the real work. It renders the declared templates, parses the rendered YAML, and drops the output into a new immutable version directory before updating current.json.

Variables don’t always come from the CLI, though. When a release needs them supplied programmatically, the Python API takes them directly:

from promptory.manager import PromptManager

version = PromptManager("prompts").release(
  variables={
    "user_name": "Alice",
    "generation_time": "2026-05-09T12:00:00Z",
  }
)

The expected variables live in promptspec.yaml:

files:
  - system.yaml
  - message.yaml
required_variables:
  - user_name
  - generation_time
max_file_bytes: 100000

This file is the contract for the prompt directory. It says which rendered YAML files Promptory manages and which variables must be supplied before release.

Versions are for runtime

A release turns drafts into rendered YAML artifacts:

uv run prompt check
uv run prompt release --patch

After release, Promptory writes a semantic version directory:

prompts/
  versions/
    v0.0.1/
      system.yaml
      message.yaml
      metadata.json
  current.json

Runtime code should not read from drafts/. It should read from the active version:

from promptory import PromptStore

store = PromptStore("prompts")

system = store.load("system.yaml")
message = store.load("message.yaml")

PromptStore reads current.json and validates the requested prompt name against promptspec.yaml. Then it loads rendered YAML from versions/<version>/.

The runtime path is deliberately boring as a result: no Jinja rendering at runtime, no accidental use of draft prompts, and no hidden prompt selection logic inside the application.

Applications can also load a specific version:

store = PromptStore("prompts")

system_v1 = store.load("system.yaml", version="v0.0.1")
all_v2 = store.load_all(version="v0.0.2")

Pinning to a specific version like this is useful for evals and debugging. If an output changed between two runs, then the prompt version becomes part of the evidence.

The release pointer

The small file that makes the runtime story work is current.json. See below:

{
  "version": "v0.0.1",
  "updated_at": "2026-05-09T22:14:00.000000+00:00"
}

This small file is the active release pointer, which makes the runtime story work. When application code loads the current prompt, it is not asking, “What is in the drafts directory today?” but rather, “Which release is currently active?”

Rollback is therefore a pointer change, not a rewrite, accomplished with a single command:

uv run prompt rollback v0.0.1

The release artifacts remain immutable. current.json moves back to an existing version.

This is closer to how many deployment systems work. You do not edit the old release in place. You point the system at a known-good artifact.

A concrete workflow

Imagine a research team maintaining a small suite of prompts for a customer-facing assistant. Rather than one monolithic prompt, they’ve split responsibilities across three separate files:

system.yaml
input_guardrail.yaml
output_guardrail.yaml

The team captures this list in a spec, along with a few constraints each file must satisfy:

files:
  - system.yaml
  - input_guardrail.yaml
  - output_guardrail.yaml
required_variables: []
max_file_bytes: 100000

Ultimately, the team doesn’t edit those files directly. They edit Jinja-templated drafts under prompts/drafts/, which get rendered into the finished files the spec expects:

# prompts/drafts/input_guardrail.yaml.j2
policy: |
  Reject requests for secrets, credentials, or private keys.

# prompts/drafts/output_guardrail.yaml.j2
policy: |
  Answer concisely and avoid unsupported claims.

Before releasing, the team checks the prompt set:

uv run prompt check

Then it previews the difference between the active release and rendered drafts:

uv run prompt diff

Once that looks right, it creates a new release:

uv run prompt release --patch

From there, the application consumes the released files:

from promptory import PromptStore

store = PromptStore("prompts")

system = store.load("system.yaml")
input_guardrail = store.load("input_guardrail.yaml")
output_guardrail = store.load("output_guardrail.yaml")

messages = [
  {"role": "system", "content": system["system_prompt"]},
  {"role": "developer", "content": input_guardrail["policy"]},
  {"role": "developer", "content": output_guardrail["policy"]},
  {"role": "user", "content": user_message},
]

That is the whole loop:

edit draft -> check -> diff -> release -> runtime loads current version

The authoring/runtime split

Promptory is built around a strict boundary.

Authoring tools can write prompt lifecycle state:

prompt init
prompt check
prompt diff
prompt release
prompt rollback
prompt draft

Runtime code reads released prompt state:

active version from current.json
rendered YAML from versions/<version>/
file declarations from promptspec.yaml

This split matters because prompt systems often blur authoring and runtime concerns. If runtime code renders templates, then a missing variable becomes a runtime incident. If runtime code reads drafts, then unreviewed edits can affect production behavior. If applications choose prompt files directly, then the promptspec contract is bypassed.

Promptory keeps the runtime path small on purpose.

What the checks protect

The linter is not trying to prove that a prompt is good. It is trying to catch the kinds of mechanical errors that should not reach a release.

For example, Promptory checks:

whether declared draft templates exist
whether templates have valid Jinja syntax
whether rendered artifacts are valid YAML
whether required variables are declared
whether undeclared variables are used
whether managed files are relative .yaml paths
whether file sizes exceed configured limits

These checks are intentionally mundane, and that’s the point: a prompt release should not fail because a template variable rendered to an empty string, a YAML file stopped parsing, or an application loaded a file outside the managed prompt set.

Why immutable releases matter

Immutable prompt releases give three practical benefits.

First, they make review clearer. A reviewer can inspect both the editable draft and the rendered artifact that runtime code will load.

Second, they make debugging easier. If a model behavior changed after a release, you can load the exact prompt version used by that run.

Third, they make rollback safer. Rollback does not require editing prompt content. It only changes the active pointer to a previously released artifact.

This is the same reason build artifacts, container images, and database migrations tend to get explicit versioning in mature systems. Prompts are not identical to those artifacts, but they have the same operational pressure: they change behavior.

Serving prompts to non-Python clients

The core Promptory model is file-based, but not every consumer is Python.

For Go, TypeScript, or mixed-language systems, Promptory can expose released prompts through a small registry service:

uv run prompt serve --port 8000

The service exposes endpoints such as:

GET /versions
GET /versions/current
GET /prompts
GET /prompts/{name}

This does not create a second prompt lifecycle. The service is a read-only layer over the same PromptStore contract: it reads current.json, validates prompt names, and serves rendered release artifacts, existing for consumption, not authoring.

What Promptory is not

Promptory is deliberately small.

It is not:

a hosted prompt CMS
an experiment tracking platform
an eval runner
a deployment system
a replacement for Git

It does not try to answer every question around prompt governance. Instead, it focuses on one narrow lifecycle:

turn draft prompt templates into versioned, reviewable, immutable runtime artifacts.

That narrowness is useful. It keeps the tool understandable and lets teams wire it into their existing Git, CI, eval, and deployment workflows.

Closing thought

Prompt changes are software changes. They may be written in natural language, but they alter system behavior just as a code change does. Consequently, they deserve the same basic lifecycle we already trust for code: review, validation, versioning, release, and rollback. Promptory is my attempt to make that lifecycle lightweight enough to not get in the way of day-to-day research work, while still giving production systems the stable, predictable runtime contract they need.

LLM as an Oracle

2026-05-13T00:00:00+00:00

Task-specific LLM evaluation used to mean reference-based metrics and test suites, with human raters for anything open-ended. Then LLM-as-a-Judge made judgment cheap, and the conversation shifted to which judge model is best for a given task. That’s the wrong first question. The first question should be whether the task needs judgment or verification, two evaluation modes that rely on different kinds of evidence. Judgment fits tasks that are open-ended, subjective, or otherwise difficult to reduce to executable checks. It relies on rubrics, preferences, or expert opinion. Verification fits tasks that are backed by stronger evidence: ground truth, test cases, or expected behavioral properties like performance.

Get this wrong and the evaluation fails in both directions. A rubric can tell you whether an explanation is clear or persuasive, but it’s a weak substitute for test cases when you’re evaluating code. Exact-match checks against ground truth fail the other way: they fall apart the moment the output is qualitative.

Avoiding that mismatch is what led to llm-as-an-oracle. An Oracle here isn’t an all-knowing model. It’s an adaptive decision layer that determines whether a task needs judgment or verification, then routes it to an LLM-as-a-Judge or an LLM-as-a-Verifier.

The central claim is simple:

Evaluation should be routed to the strategy that best matches the structure of the task.

That sounds obvious, but it’s easy to violate in practice. As soon as a benchmark, agent workflow, or production pipeline standardizes on a single evaluator, it treats fundamentally different tasks as though they required the same kind of evidence. Finding the best judge model doesn’t fix that; it just decides which evaluator gets applied to everything.

Here’s what that looks like in practice. Later in this post, three agents fix the same N+1 query bug. Two of them change the query shape. The third wraps the buggy call in an lru_cache and looks correct — familiar technique, concrete code, a plausible performance story. A Judge scoring on presentation alone can be fooled by it. A Verifier running the test suite cannot. That gap between looking right and being right is what the Oracle is built to catch.

The evaluation problem

Human evaluation remains the reference point for many LLM systems. Humans can interpret incomplete instructions, account for context, distinguish severity from style, and notice when an answer is technically correct but pragmatically poor. But human evaluation scales badly, which created demand for automated alternatives. Traditional metrics help, but each covers a narrow slice:

exact match works when the answer space is constrained
unit tests work when executable behavior matters
overlap metrics work in narrow summarization settings

Unfortunately, none of these solves the broader evaluation problem on its own.

Capable instruction-following models added a new option: LLM-based evaluators. They’ve produced a family of LLM-as-* patterns, each with its own line of research:¹

LLM-as-a-Judge
LLM-as-a-Verifier
LLM-as-a-Critic
LLM-as-a-Ranker

These patterns are often discussed as alternatives. They’re better understood as tools with different operating assumptions, especially about what evidence is available. Critics and rankers produce assessments rather than checks, so they sit on the judgment side; this post focuses on the Judge and Verifier. So the useful question isn’t which pattern, or which judge model, is best. It’s the one from the start:

Does this task need judgment or verification?

Judge and Verifier solve different problems

The Oracle routes between two strategies, one for each evaluation mode.

LLM-as-a-Judge

An LLM-as-a-Judge performs holistic evaluation. It reads the task, a candidate trajectory (the full solution attempt, not just the final answer), and the evaluation criteria, then emits a score or preference.² It’s the natural fit for judgment tasks, such as:

Is this answer concise without omitting important details?
Does this explanation match the user’s level of expertise?
Which recommendation is more useful under vague constraints?
Is the reasoning persuasive and coherent?

None of these collapse into executable checks. They require interpretation. That doesn’t make a Judge “ask another model what it thinks”: in llm-as-an-oracle, it scores against weighted rubric criteria and swaps pairwise order to reduce positional bias.

LLM-as-a-Verifier

An LLM-as-a-Verifier evaluates trajectories against signals closer to correctness than preference.³ It’s the natural fit for verification tasks, such as:

code generation with tests
question answering with reference answers
tool-use traces with expected outputs
structured reasoning tasks with decomposable criteria

Rather than settle for a single coarse score, the Verifier tries to extract finer-grained signal: it decomposes criteria, repeats verification, and uses token log probabilities when the provider exposes them.

The Judge asks which trajectory seems better under a rubric; the Verifier asks which survives the strongest evidence-sensitive checks available. Related questions, but not the same one.

What the Oracle adds

If Judge and Verifier are both useful, a natural response is to expose both and let the caller choose. That’s necessary but not sufficient. Many workflows mix task types:

an agent may produce code patches, explanations, and planning notes
a benchmark may combine factual QA, long-form reasoning, and executable tasks
a production system may need to evaluate recommendations, SQL, and tool calls within the same pipeline

In those settings, asking the caller to pick an evaluator every time creates friction and invites inconsistency.

The Oracle removes that step. It inspects the task and its trajectories, picks the evaluator that fits better, runs only that one, and returns the result along with an explanation of the choice. It isn’t a third evaluator; it sits above the two.

Anatomy of the Oracle router

The part of the Oracle that makes this choice is the router. The default router in llm-as-an-oracle is deterministic: it doesn’t call an LLM to decide which evaluator to use. Instead, it extracts interpretable signals and applies a fixed chain of routing policies. That’s deliberate: evaluator selection should become more legible, not less.

Step 1: extract routing signals

The router converts the task and trajectories into a structured set of signals. The current implementation uses features such as:

has_ground_truth
has_test_cases
trajectory_count
stated_difficulty
verifiable_keyword_density
judgment_keyword_density
problem_length
output_available
prior_hardness

The goal is not to perfectly infer task type from text. The goal is to make the selection logic explicit enough to inspect, revise, and extend.

Step 2: collect policy votes

Signals are passed through a chain of policies, roughly one per signal group. Each policy casts a weighted vote for either Judge or Verifier.

Conceptually:

Ground truth or test cases present?  -> favor Verifier
Execution output available?          -> favor Verifier
Previously observed as hard?         -> favor Verifier
Open-ended wording?                  -> favor Judge

The implementation is more nuanced than that sketch, but the spirit is the same: evaluation mode is chosen by accumulating evidence.

Step 3: aggregate confidence

The router aggregates the weighted votes into a final confidence score. The winning strategy is selected only if its confidence is strong enough; otherwise, the system falls back to the more general-purpose Judge path.

This separates a strategy’s power from its fit. A Verifier can be the stronger tool and still be the wrong one when the evidence for this task doesn’t justify it. That’s a better design than letting every downstream evaluator silently assume the task suits its own strengths.

Step 4: expose the routing trace

The output of a routing decision includes:

the selected strategy
the final confidence
the raw signals
every policy vote
a human-readable reasoning trace

I care about this part the most, because evaluation pipelines already accumulate ambiguity, and a score you can’t trace is hard to debug. The trace makes one critical source of ambiguity observable: why this evaluator was chosen in the first place.

Evaluating trajectories, not just answers

The project uses the term trajectory deliberately.

A trajectory is a candidate task-solving attempt. It may contain:

the final answer
intermediate reasoning or planning
code
tool calls
execution output
an optional reward signal

This is especially relevant for agents. When a coding agent fixes a bug, the thing we care about is not only the final patch. We may care about:

whether the patch addresses the requested failure mode
whether it satisfies explicit requirements
whether execution evidence supports the answer
whether two superficially plausible solutions differ materially

For text-only tasks, evaluating the final answer may be sufficient. For agents, the evaluation object often needs to be richer.

The Oracle architecture assumes that richer object from the beginning.

A concrete example: the N+1 query bug

One example in the repository asks three agents to fix an N+1 query problem. The original function loads orders first, then issues one SQL query per order to load items. That means:

51 queries for 50 orders
5,001 queries for 5,000 orders

The task asks for a constant-query solution and provides both ground truth and test cases.

The original bug has this shape:

def get_orders_with_items(user_id: int) -> list[dict]:
    orders = db.execute(
        "SELECT * FROM orders WHERE user_id = ?",
        [user_id],
    )

    for order in orders:
        order["items"] = db.execute(
            "SELECT * FROM items WHERE order_id = ?",
            [order["id"]],
        )

    return orders

The loop is the problem. The first query fetches the orders, then each order triggers another query for its items.

Three candidate trajectories are evaluated:

one rewrites the query with a JOIN
one performs a batched prefetch with WHERE IN
one adds an lru_cache around the inner item lookup

The first two are legitimate fixes, though they make different tradeoffs. The third sounds plausible because caching often improves performance. But it does not solve the stated problem. On a cold cache, query count still grows with the number of orders.

The misleading fix looks like this:

@lru_cache(maxsize=256)
def fetch_items(order_id: int) -> list[dict]:
    return db.execute(
        "SELECT * FROM items WHERE order_id = ?",
        [order_id],
    )


def get_orders_with_items(user_id: int) -> list[dict]:
    orders = db.execute(
        "SELECT * FROM orders WHERE user_id = ?",
        [user_id],
    )

    for order in orders:
        order["items"] = fetch_items(order["id"])

    return orders

This may help repeated calls for the same order, but it does not change the first-run query pattern:

expected: query_count == O(1)
actual:   query_count == 1 + number_of_orders

A real fix changes the query shape. For example, the JOIN trajectory uses one SQL query and groups the rows afterward:

SELECT o.*, i.*
FROM orders o
LEFT JOIN items i ON i.order_id = o.id
WHERE o.user_id = ?;

This is exactly the sort of case where evaluator choice matters.

Why Judge alone is risky here

A holistic Judge may recognize that the cache-based answer is weaker. But it is also possible for that answer to benefit from surface plausibility:

it uses a familiar optimization technique
it contains concrete code
it appears to address performance

If the score is driven too much by presentation quality, the wrong candidate can become competitive.

Why Verifier is the better fit

The task has stronger evidence:

explicit correctness requirements
test cases
expected behavioral properties
a measurable performance invariant: query count must not scale with N

That is precisely the situation where the Oracle should route toward the Verifier. The evaluation problem is not mainly aesthetic. It is evidential.

The interesting part is not merely that Verifier can help. The more general point is that the Oracle can identify this task shape before evaluation begins.

Judge and Verifier are not rivals

It is tempting to treat this as a winner-take-all comparison:

Verifier is more objective
Judge is more flexible
one must be superior

I do not think that framing is useful.

Each strategy fails differently.

Failure modes of Judge

A Judge can overvalue fluent or confident language, blur correctness and style, and struggle to separate close technical alternatives without stronger evidence.

Failure modes of Verifier

A Verifier has its own failure modes. It inherits bad ground truth, can overfit to incomplete criteria, becomes brittle when tests are narrow, and quietly underperforms on tasks that are fundamentally subjective.

The Oracle does not eliminate these problems. It tries to reduce one avoidable problem: choosing the wrong mode of evaluation for the task at hand.

When the Oracle should ask for help

There is another failure mode worth making explicit: sometimes the task itself is underspecified.

Suppose three architecture recommendations are all defensible, but the best one depends on a missing fact about team size, latency goals, compliance constraints, or deployment environment. No evaluator should pretend confidence if the information needed to decide was never supplied.

The fuller design in this project explores a Human Oracle escalation path for those cases. When evaluation evidence runs out, the system can ask a targeted clarifying question, incorporate the answer, and re-evaluate.

The point is not to put a person in the loop by default. It is to avoid manufactured certainty when a decision depends on missing context.

That same principle motivates the router itself:

do not hide assumptions
do not oversell confidence
make uncertainty inspectable

What this suggests about evaluation design

The Oracle pattern leads to a broader design lesson.

Good evaluation systems share a few properties. Evaluator should match task structure: open-ended and evidence-grounded tasks are different, and treating them the same introduces avoidable error. Selection should be explicit, part of the system design rather than a hidden convention buried in notebook code or benchmark glue. Inspectability should be preserved: routing traces, criteria, and confidence should be artifacts you can actually look at. And the system should admit when evidence is insufficient. A strong evaluator is no substitute for missing context, and escalation can be the right move.

Where this pattern is useful

I think LLM-as-an-Oracle is especially relevant for:

coding agents that generate several candidate patches
benchmark pipelines that mix factual, creative, and executable tasks
tool-using agents whose outputs include both text and action traces
research workflows comparing evaluators across heterogeneous task families
production systems that want one evaluation interface without pretending every task should be judged the same way

In all of those cases, evaluator selection is part of the problem.

Treating it as a first-class system component is cleaner than standardizing on a single evaluation method and compensating later with increasingly elaborate exceptions.

Closing thought

The evaluator should fit the task.

LLM-as-an-Oracle is my attempt to turn that principle into a concrete system: route between judgment and verification, expose the reasons for the choice, and leave room for human escalation when neither automated path has enough evidence.

Zheng et al. introduce LLM-as-a-Judge; the LLM-as-a-Verifier framework develops evidence-sensitive verification; CritiqueLLM studies critique generation for evaluation; and pairwise ranking prompting shows how LLMs can be used directly as rankers. https://arxiv.org/abs/2306.05685 https://llm-as-a-verifier.notion.site/ https://arxiv.org/abs/2311.18702 https://arxiv.org/abs/2306.17563 ↩
Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” 2023. https://arxiv.org/abs/2306.05685 ↩
“LLM-as-a-Verifier: A General-Purpose Verification Framework.” https://llm-as-a-verifier.notion.site/ ↩

My book is out!

2015-03-05T00:00:00+00:00

Software Patterns, Knowledge Maps, and Domain Analysis

My book is about domain analysis and software patterns. Briefly, it shows how to build software systems as systems of patterns, called Knowledge Maps.

It is now in the global markets. Its publication date was December 2014. Please check it out:

Figure 1. Front cover of my book

The citation (in BibTex) of this book is presented as follows:

@book{hsanchez, 
      author = {M. E. Fayad, H. A. Sanchez, S. G .K. Hegde, 
                A. Basia, and A. Vakil.},
      title  = {Software Patterns, Knowledge Maps, and 
                Domain Analysis},
      publisher = {Auerbach Publications},
      year      = 2014,
      address   = {Boca Raton, Florida},
      isbn      = {978-1466571433}
}

Cyberbullying detection in social networks

2012-04-29T00:00:00+00:00

According to Google Analytics and to other sources, my data mining (ISM 245 class) project titled “Twitter Bullying Detection” is getting some attention. An earlier version of the project report was hosted on the co-author of the project’s website. This version is good in terms of communicating the project’s main idea and motivation. However, I think it is always important to have the latest version available. Therefore, I decided to post this latest version here.

Please feel free to contact me if you have any questions about the content of that report. Thanks.