Huascar Sanchez Posts

Gating Your LLM Eval Rubric with Evidence

2026-05-26T00:00:00+00:00

Part 2: Managing Your LLM Eval Rubric Like Code

Promptory’s staged releases, evidence documents, and release gates let you attach eval results to a rubric candidate and block promotion until that evidence passes. This post shows the workflow step by step. The full example is in examples/evals2.py.

The Gate Declaration

Add a release_gates block to promptspec.yaml before releasing any candidates:

files:
  - rubric.yaml
required_variables:
  - criteria
max_file_bytes: 100000
release_gates:
  evidence:
    - kind: eval
      name: eval-run
      required_status: pass

Any promotion attempt that lacks an evidence document of kind: eval, name: eval-run, and status: pass raises PromptGateError and leaves current.json unchanged.

Step 1: Stage a Candidate

A staged release writes an immutable snapshot under versions/ without updating current.json. The active version is unaffected while the candidate is evaluated.

from promptory.manager import PromptManager

manager = PromptManager("prompts")

draft = manager.prompts_dir / "drafts" / "rubric.yaml.j2"
draft.write_text(BASIC_RUBRIC)

v1 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
  staged=True,
)
# v0.0.1 exists in versions/ but current.json is unchanged.

Step 2: Attach Evidence

Load the staged rubric, run your eval harness, then write the results as an evidence document and attach it to the version.

from promptory import PromptStore
from promptory.evidence import add_evidence

store = PromptStore("prompts")
rubric = store.load("rubric.yaml", version=v1)

metrics = run_eval_harness(rubric)

evidence_doc = {
  "kind": "eval",
  "name": "eval-run",
  "status": "fail",          # below threshold: 60% accuracy
  "tool": "eval-harness",
  "created_at": "2026-05-28T12:00:00Z",
  "summary": "accuracy=60%",
  "metrics": metrics,
}
evidence_path.write_text(json.dumps(evidence_doc))
add_evidence(manager.spec(), v1, evidence_path)

Evidence is immutable once attached.

Step 3: Check the Gate

gate = manager.gate(v1)
# False — evidence status is fail, required pass
print(gate.passed)

v1 stays staged.

Step 4: Iterate

Revise the rubric draft and stage a new candidate.

draft.write_text(STRICT_RUBRIC)

v2 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
  staged=True,
)

rubric_v2 = store.load("rubric.yaml", version=v2)
metrics_v2 = run_eval_harness(rubric_v2)

evidence_doc_v2 = {
  "kind": "eval",
  "name": "eval-run",
  "status": "pass",          # 100% accuracy clears the threshold
  "tool": "eval-harness",
  "created_at": "2026-05-28T13:00:00Z",
  "summary": "accuracy=100%",
  "metrics": metrics_v2,
}
add_evidence(manager.spec(), v2, evidence_path_v2)

gate_v2 = manager.gate(v2)
print(gate_v2.passed)   # True

Step 5: Compare Evidence

compare_evidence diffs the attached evidence between two versions.

from promptory.evidence import compare_evidence

comparison = compare_evidence(manager.spec(), v1, v2)

for change in comparison.changes:
  print(f"[{change.kind}] {change.name}: \
    {change.before_status} -> {change.after_status}")
  for metric in change.metrics:
    print(f"  {metric.name}: {metric.before} -> {metric.after}")

Output:

[eval] eval-run: fail -> pass
  accuracy: 0.6 -> 1.0
  false_positive_rate: 1.0 -> 0.0
  mean_score: 4.6 -> 3.4

Step 6: Promote

require_gates=True makes the gate check part of the promotion call. If evidence is missing or fails, the call raises PromptGateError and current.json is not updated.

manager.promote(v2, require_gates=True)
print(store.current_version())   # v0.0.2

v1 remains staged. v2 is now current.

Running the Example

uv run python examples/evals2.py

Expected output:

[v1 basic] staged v0.0.1
  accuracy=60%  fpr=100%  fnr=0%
  gate: FAIL - candidate stays staged

[v2 strict] staged v0.0.2
  accuracy=100%  fpr=0%  fnr=0%
  gate: PASS

Evidence comparison v0.0.1 -> v0.0.2:
  [eval] eval-run: fail -> pass
    accuracy: 0.6 -> 1.0
    false_positive_rate: 1.0 -> 0.0
    mean_score: 4.6 -> 3.4

Promoted v0.0.2 -> current: v0.0.2

What Promptory Does Not Do

Promptory stores and validates the evidence document shape. It does not run evals, call LLMs, define metric thresholds, or decide whether a rubric is correct.

Managing Your LLM Eval Rubric Like Code

2026-05-19T00:00:00+00:00

Promptory manages your eval rubric as a versioned prompt artifact: drafts, immutable releases, template variables, and version history. This post shows how to draft, release, and load rubric versions using the Python API.

The full runnable example is in examples/evals.py.

The Rubric Template

A rubric prompt in Promptory is a Jinja template. Declare the file and its required variables in promptspec.yaml:

files:
  - rubric.yaml
required_variables:
  - criteria
max_file_bytes: 100000

The draft template lives in prompts/drafts/rubric.yaml.j2:

model: claude-sonnet-4-6
temperature: 0.0
system_prompt: |
  Evaluate the response on {{ criteria }}.
  Be generous; give partial credit for effort.
  Return JSON: {score: int, rationale: str}.

{{ criteria }} is a template variable. It is rendered at release time, not at runtime.

Template Variables

Release the same rubric template with different variable values to produce separate versioned artifacts for each eval dimension.

from promptory.manager import PromptManager

manager = PromptManager("prompts")

v_helpfulness = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
)
v_factual     = manager.release(
  bump="patch",
  variables={"criteria": "factual accuracy"},
)

Each release is an independent rendered artifact. Loading v_helpfulness always returns the rubric with criteria="helpfulness" regardless of subsequent releases.

Draft and Release Separation

prompts/drafts/ holds the working template. prompts/versions/ holds immutable rendered releases. Editing the draft and cutting a new release does not modify any previous release.

BASIC_RUBRIC = (
  "model: claude-sonnet-4-6\n"
  "temperature: 0.0\n"
  "system_prompt: |\n"
  "  Evaluate the response on {{ criteria }}.\n"
  "  Be generous; give partial credit for effort.\n"
  "  Return JSON: {score: int, rationale: str}.\n"
)

STRICT_RUBRIC = (
  "model: claude-sonnet-4-6\n"
  "temperature: 0.0\n"
  "system_prompt: |\n"
  "  Evaluate the response on {{ criteria }}.\n"
  "  Penalize vague or incomplete answers.\n"
  "  Return JSON: {score: int, rationale: str}.\n"
)

# v1: initial rubric.
draft.write_text(BASIC_RUBRIC)
v1 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
)

# v2: updated rubric. v1 is unchanged.
draft.write_text(STRICT_RUBRIC)
v2 = manager.release(
  bump="patch",
  variables={"criteria": "helpfulness"},
)

The directory prompts/versions/v0.0.1/ is never touched after creation.

Loading by Version

PromptStore.load accepts an optional version argument. When supplied, it reads from that exact release directory regardless of which version is current.

from promptory import PromptStore

store = PromptStore("prompts")

rubric_v1 = store.load("rubric.yaml", version=v1)
rubric_v2 = store.load("rubric.yaml", version=v2)

# Pass rubric["system_prompt"] to your eval harness.

The result is identical whether v1 is current or not.

Version History

PromptStore.list_versions returns all available releases in semantic version order.

print(store.list_versions())
# ['v0.0.1', 'v0.0.2', 'v0.0.3']

Use this to scan all candidates and compare results across versions.

version      criteria               accuracy      fpr      fnr
v0.0.1        helpfulness              60%      100%        0%
v0.0.2        helpfulness             100%        0%        0%
v0.0.3        factual accuracy        100%        0%        0%

v0.0.2 has the best metrics for helpfulness. v0.0.3 applies the same rubric instructions to a different eval dimension without re-authoring the template.

Running the Example

uv run python examples/evals.py

The example uses a simulated judge and fixed benchmark data. Replace simulate_judge_call with a real LLM call to run it against your own eval suite.

What Promptory Does Not Do

Promptory manages the rubric prompt. It does not run evals, call LLMs, manage benchmark datasets, or decide whether one rubric is better than another.

Part 2: Gating Your LLM Eval Rubric with Evidence

Promptory, a Git-based prompt versioning system

2026-05-14T00:00:00+00:00

In many organizations building GenAI solutions, prompts have become production artifacts. However, despite their importance, many teams still treat them like ordinary strings. Treating them this way might work for a while, but as the prompts evolve, as their performance becomes variable across different models, model versions, safety policies, or use cases, it becomes more complex and harder to manage them as strings.

That is the motivation behind Promptory: a Git-based prompt versioning system that gives prompt changes the same basic engineering discipline as code. The core idea is simple:

Author prompts as editable drafts, release them as immutable rendered artifacts, and make runtime code load only released versions.

The prompt management problem

Prompt changes are deceptively small. Changing a few words can alter:

how much context the model uses
whether it follows a tool policy
how it handles uncertain answers
whether it refuses unsafe requests
which format downstream parsers expect
how it behaves under evals

In traditional software engineering, anything that can change production behavior is managed deliberately: version control, review, tests, releases, and rollback. I believe that prompt engineering should not be exempt from that discipline just because the artifact is text. On the contrary, it should also be managed deliberately.

The hard part is choosing the right amount of discipline. In practice, teams often land on one of two extremes. The first one is the hardcoded prompt:

SYSTEM_PROMPT = """
You are a helpful assistant. Answer concisely.
"""

Hardcoding a prompt like this is fine for a prototype, but it’s brittle for a system that needs repeatable experiments, reviewable changes, or rollback.

The second one is overcorrection. This is the process of adopting a large hosted prompt platform before the team actually needs one. Those systems can be useful when non-technical users need a UI, many teams need centralized governance, or the current prompt metadata no longer fits a particular repository workflow.

I strongtly believe fast-moving R&D teams often need something smaller:

plain files
Git review
CI-friendly checks
immutable releases
a clear pointer to the active version
runtime loading that never reads work-in-progress drafts

Promptory is aimed at that middle ground.

Why Git alone is not enough

Why not just use Git? That’s the obvious pushback I get when socializing Promptory. Git already handles reviewing and preserving prompt history just fine, they say. That is true. However, Git doesn’t define a prompt lifecycle.

In other words, Git can tell you that a prompt file changed, but it cannot tell your application:

which rendered prompt version is active
whether a prompt has unresolved template variables
whether all managed prompt files were released together
whether a runtime process is reading drafts or released artifacts
how to roll back without rewriting prompt history

That is why Promptory keeps Git as the durable history and adds a small release model on top of it.

Why not a central registry?

Promptory’s answer is a deliberate ownership choice: prompts live with the code that uses them. This builds on the same locality of change principle from software engineering — normally applied to code — and extends it to prompts. Specifically, if a prompt shapes a service’s behavior, it should be reviewed, released, and rolled back with that service.

A central registry can be useful for organization-wide discovery, shared prompts, or non-engineering workflows. But not so much for application-specific prompts, which should live next to the implementation to make review and rollback simpler. The prompt diff, code diff, tests, and release artifact all move through the same Git workflow.

This ownership choice doesn’t rule out central visibility. In production, teams can still publish released prompt artifacts to object storage, an internal artifact registry, or Promptory’s read-only registry service. The repo remains the source of truth; production systems only aggregate or serve released artifacts from there.

In the repo, that looks something like this:

prompts/
  drafts/
    system.yaml.j2
    input_guardrail.yaml.j2
    output_guardrail.yaml.j2
  versions/
    v0.1.0/
      system.yaml
      input_guardrail.yaml
      output_guardrail.yaml
      metadata.json
  current.json
  promptspec.yaml

The above layout breaks down into four parts, each with a distinct role:

drafts/ contains editable Jinja templates
versions/ contains rendered release artifacts
current.json points to the active release
promptspec.yaml declares which prompt files are managed

Two of those four parts carry the real distinction. Developers edit drafts. Applications load versions. That boundary, simple as it sounds, is the main design decision in the project.

Drafts are for authoring

A draft is where prompt authors and coding agents work. It’s written in plain YAML. For example:

# prompts/drafts/system.yaml.j2
model: gpt-5.5
temperature: 0.2
system_prompt: |
  You are a helpful assistant.
  Answer concisely and avoid unsupported claims.

Because drafts are Jinja templates rather than static text, they can also include release-time variables:

# prompts/drafts/message.yaml.j2
message: |
  Hello {{ user_name }}.
  Generated at {{ generation_time }}.

Promptory renders these with Jinja’s StrictUndefined, on purpose. A missing variable throws an exception instead of quietly becoming an empty string. Silent prompt rendering failures are hard to debug because they often look like model behavior problems later.

Rendering itself happens at release time, through two commands:

uv run prompt check
uv run prompt release --patch

prompt check catches problems early: bad drafts, missing variable declarations. prompt release does the real work. It renders the declared templates, parses the rendered YAML, and drops the output into a new immutable version directory before updating current.json.

Variables don’t always come from the CLI, though. When a release needs them supplied programmatically, the Python API takes them directly:

from promptory.manager import PromptManager

version = PromptManager("prompts").release(
  variables={
    "user_name": "Alice",
    "generation_time": "2026-05-09T12:00:00Z",
  }
)

The expected variables live in promptspec.yaml:

files:
  - system.yaml
  - message.yaml
required_variables:
  - user_name
  - generation_time
max_file_bytes: 100000

This file is the contract for the prompt directory. It says which rendered YAML files Promptory manages and which variables must be supplied before release.

Versions are for runtime

A release turns drafts into rendered YAML artifacts:

uv run prompt check
uv run prompt release --patch

After release, Promptory writes a semantic version directory:

prompts/
  versions/
    v0.0.1/
      system.yaml
      message.yaml
      metadata.json
  current.json

Runtime code should not read from drafts/. It should read from the active version:

from promptory import PromptStore

store = PromptStore("prompts")

system = store.load("system.yaml")
message = store.load("message.yaml")

PromptStore reads current.json and validates the requested prompt name against promptspec.yaml. Then it loads rendered YAML from versions/<version>/.

The runtime path is deliberately boring as a result: no Jinja rendering at runtime, no accidental use of draft prompts, and no hidden prompt selection logic inside the application.

Applications can also load a specific version:

store = PromptStore("prompts")

system_v1 = store.load("system.yaml", version="v0.0.1")
all_v2 = store.load_all(version="v0.0.2")

Pinning to a specific version like this is useful for evals and debugging. If an output changed between two runs, then the prompt version becomes part of the evidence.

The release pointer

The small file that makes the runtime story work is current.json. See below:

{
  "version": "v0.0.1",
  "updated_at": "2026-05-09T22:14:00.000000+00:00"
}

This small file is the active release pointer, which makes the runtime story work. When application code loads the current prompt, it is not asking, “What is in the drafts directory today?” but rather, “Which release is currently active?”

Rollback is therefore a pointer change, not a rewrite, accomplished with a single command:

uv run prompt rollback v0.0.1

The release artifacts remain immutable. current.json moves back to an existing version.

This is closer to how many deployment systems work. You do not edit the old release in place. You point the system at a known-good artifact.

A concrete workflow

Imagine a research team maintaining a small suite of prompts for a customer-facing assistant. Rather than one monolithic prompt, they’ve split responsibilities across three separate files:

system.yaml
input_guardrail.yaml
output_guardrail.yaml

The team captures this list in a spec, along with a few constraints each file must satisfy:

files:
  - system.yaml
  - input_guardrail.yaml
  - output_guardrail.yaml
required_variables: []
max_file_bytes: 100000

Ultimately, the team doesn’t edit those files directly. They edit Jinja-templated drafts under prompts/drafts/, which get rendered into the finished files the spec expects:

# prompts/drafts/input_guardrail.yaml.j2
policy: |
  Reject requests for secrets, credentials, or private keys.

# prompts/drafts/output_guardrail.yaml.j2
policy: |
  Answer concisely and avoid unsupported claims.

Before releasing, the team checks the prompt set:

uv run prompt check

Then it previews the difference between the active release and rendered drafts:

uv run prompt diff

Once that looks right, it creates a new release:

uv run prompt release --patch

From there, the application consumes the released files:

from promptory import PromptStore

store = PromptStore("prompts")

system = store.load("system.yaml")
input_guardrail = store.load("input_guardrail.yaml")
output_guardrail = store.load("output_guardrail.yaml")

messages = [
  {"role": "system", "content": system["system_prompt"]},
  {"role": "developer", "content": input_guardrail["policy"]},
  {"role": "developer", "content": output_guardrail["policy"]},
  {"role": "user", "content": user_message},
]

That is the whole loop:

edit draft -> check -> diff -> release -> runtime loads current version

The authoring/runtime split

Promptory is built around a strict boundary.

Authoring tools can write prompt lifecycle state:

prompt init
prompt check
prompt diff
prompt release
prompt rollback
prompt draft

Runtime code reads released prompt state:

active version from current.json
rendered YAML from versions/<version>/
file declarations from promptspec.yaml

This split matters because prompt systems often blur authoring and runtime concerns. If runtime code renders templates, then a missing variable becomes a runtime incident. If runtime code reads drafts, then unreviewed edits can affect production behavior. If applications choose prompt files directly, then the promptspec contract is bypassed.

Promptory keeps the runtime path small on purpose.

What the checks protect

The linter is not trying to prove that a prompt is good. It is trying to catch the kinds of mechanical errors that should not reach a release.

For example, Promptory checks:

whether declared draft templates exist
whether templates have valid Jinja syntax
whether rendered artifacts are valid YAML
whether required variables are declared
whether undeclared variables are used
whether managed files are relative .yaml paths
whether file sizes exceed configured limits

These checks are intentionally mundane, and that’s the point: a prompt release should not fail because a template variable rendered to an empty string, a YAML file stopped parsing, or an application loaded a file outside the managed prompt set.

Why immutable releases matter

Immutable prompt releases give three practical benefits.

First, they make review clearer. A reviewer can inspect both the editable draft and the rendered artifact that runtime code will load.

Second, they make debugging easier. If a model behavior changed after a release, you can load the exact prompt version used by that run.

Third, they make rollback safer. Rollback does not require editing prompt content. It only changes the active pointer to a previously released artifact.

This is the same reason build artifacts, container images, and database migrations tend to get explicit versioning in mature systems. Prompts are not identical to those artifacts, but they have the same operational pressure: they change behavior.

Serving prompts to non-Python clients

The core Promptory model is file-based, but not every consumer is Python.

For Go, TypeScript, or mixed-language systems, Promptory can expose released prompts through a small registry service:

uv run prompt serve --port 8000

The service exposes endpoints such as:

GET /versions
GET /versions/current
GET /prompts
GET /prompts/{name}

This does not create a second prompt lifecycle. The service is a read-only layer over the same PromptStore contract: it reads current.json, validates prompt names, and serves rendered release artifacts, existing for consumption, not authoring.

What Promptory is not

Promptory is deliberately small.

It is not:

a hosted prompt CMS
an experiment tracking platform
an eval runner
a deployment system
a replacement for Git

It does not try to answer every question around prompt governance. Instead, it focuses on one narrow lifecycle:

turn draft prompt templates into versioned, reviewable, immutable runtime artifacts.

That narrowness is useful. It keeps the tool understandable and lets teams wire it into their existing Git, CI, eval, and deployment workflows.

Closing thought

Prompt changes are software changes. They may be written in natural language, but they alter system behavior just as a code change does. Consequently, they deserve the same basic lifecycle we already trust for code: review, validation, versioning, release, and rollback. Promptory is my attempt to make that lifecycle lightweight enough to not get in the way of day-to-day research work, while still giving production systems the stable, predictable runtime contract they need.

LLM as an Oracle

2026-05-13T00:00:00+00:00

Most discussions of LLM evaluation ask which evaluator is best. Wrong first question. What actually matters is whether the task needs judgment or verification. A rubric is great for deciding if an explanation is clear or persuasive, but it’s a weak substitute for test cases when you’re evaluating code. Exact-match checks have the opposite problem: they fall apart the moment the output is qualitative.

That split is what led to llm-as-an-oracle. An Oracle here isn’t an all-knowing model. It’s an adaptive layer that routes each task to either an LLM-as-a-Judge or an LLM-as-a-Verifier, depending on which kind of evaluation the task actually needs.

The central claim is simple:

Evaluation should be routed to the strategy that best matches the structure of the task.

Here is why that matters in practice. Later in this post, three agents fix the same N+1 query bug. Two of them change the query shape. The third wraps the buggy call in an lru_cache and looks correct — familiar technique, concrete code, a plausible performance story. A Judge scoring on presentation alone can be fooled by it. A Verifier running the test suite cannot. That gap between looking right and being right is the reason this router exists.

That sounds obvious once stated plainly, but it is easy to violate in practice. As soon as a benchmark, agent workflow, or production evaluation pipeline standardizes on a single evaluator, it begins to treat fundamentally different tasks as though they required the same kind of evidence.

The evaluation problem

Human evaluation remains the reference point for many LLM systems. It is often the most flexible form of assessment because humans can interpret incomplete instructions, account for context, distinguish severity from style, and notice when a candidate answer is technically correct but pragmatically poor. It also scales badly, which created demand for automated evaluation. Traditional metrics can be useful, but they are narrow:

exact match is valuable when the answer space is constrained
unit tests are valuable when executable behavior matters
overlap metrics can be useful in narrow summarization settings
preference labels can summarize subjective quality

None of these solves the broader evaluation problem on its own.

The rise of capable instruction-following models created a new option: LLM-based evaluators. This has produced a family of LLM-as-* patterns:

LLM-as-a-Judge
LLM-as-a-Verifier
LLM-as-a-Critic
LLM-as-a-Ranker

These ideas have appeared across several lines of work on model-based judging, verification, critique generation, and ranking. ¹

These patterns are often discussed as alternatives. I think they are better understood as evaluation modes with different operating assumptions.

The question is not merely whether LLM evaluators are useful. The more precise question is:

Which evaluator is appropriate for this task, given the evidence available?

Judge and Verifier solve different problems

The easiest way to understand the Oracle idea is to first separate the two evaluation strategies it routes between.

LLM-as-a-Judge

An LLM-as-a-Judge performs holistic evaluation. It reads the task, candidate trajectory, and evaluation criteria, then emits a score or preference. This is the natural fit when the target quality is open-ended, subjective, or otherwise difficult to reduce to executable checks. ²

Typical Judge-friendly questions include:

Is this answer concise without omitting important details?
Does this explanation match the user’s level of expertise?
Which recommendation is more useful under vague constraints?
Is the reasoning persuasive and coherent?

The Judge pattern is valuable because many real tasks do not collapse cleanly into executable checks. They require interpretation.

In llm-as-an-oracle, the Judge strategy supports:

rubric-driven scoring
pointwise trajectory scoring
pairwise comparisons
order-swapped pairwise evaluation to reduce positional bias
aggregation across multiple criteria

These details matter because a Judge is not merely “ask another model what it thinks.” A useful Judge has structure around how scores are produced and how comparisons are stabilized.

For example, a Judge can score each trajectory against several weighted criteria, then use pairwise comparisons only when two candidates are close. If the pairwise order is swapped and averaged, the system can reduce simple positional bias without pretending that the evaluator has become objective. ²

LLM-as-a-Verifier

An LLM-as-a-Verifier is better suited to tasks where stronger evidence exists. It is appropriate when candidate trajectories can be evaluated against signals that are closer to correctness than preference. ³

Typical Verifier-friendly tasks include:

code generation with tests
question answering with reference answers
tool-use traces with expected outputs
structured reasoning tasks with decomposable criteria
tasks where execution evidence is available

The Verifier strategy in this project is designed around:

finer-grained score extraction
repeated verification
criteria decomposition
pairwise tournament-style ranking
support for logprob-aware scoring when the provider exposes token probabilities

That last point is important. A Verifier tries to squeeze more discriminative signal out of the evaluator than a single coarse score can provide.

The Judge is asking which answer seems better under a rubric; the Verifier is asking which trajectory survives the strongest evidence-sensitive checks available. Related questions, but not the same one.

What the Oracle adds

If Judge and Verifier are both useful, a natural response is to expose both and let the caller choose. That’s necessary but not sufficient. Many workflows mix task types:

an agent may produce code patches, explanations, and planning notes
a benchmark may combine factual QA, long-form reasoning, and executable tasks
a production system may need to evaluate recommendations, SQL, and tool calls within the same pipeline

In those settings, asking the caller to manually select an evaluator every time creates friction and invites inconsistency.

The Oracle layer addresses that problem.

Its job is to:

inspect the task and trajectories
extract signals about the task structure
decide which evaluator is the better fit
execute only that strategy
return both the result and the routing explanation

The Oracle is therefore not a third evaluator. It is a decision layer above the two evaluators.

Anatomy of the Oracle router

The default router in llm-as-an-oracle is deterministic. It does not call an LLM to decide which evaluator to use. Instead, it extracts interpretable signals and applies a fixed chain of routing policies.

That design choice is intentional. The system should make evaluator selection more legible, not less.

Step 1: extract routing signals

The router converts the task and trajectories into a structured set of signals. The current implementation uses features such as:

has_ground_truth
has_test_cases
trajectory_count
stated_difficulty
verifiable_keyword_density
judgment_keyword_density
problem_length
output_available
prior_hardness

These features encode simple but meaningful intuitions.

For example:

ground truth and test cases usually favor verification
execution output usually favors verification
open-ended language often favors judgment
a previously observed hard task may deserve a stronger verification path

The goal is not to perfectly infer task type from text. The goal is to make the selection logic explicit enough to inspect, revise, and extend.

Step 2: collect policy votes

Signals are passed through a chain of policies. The default router uses policies that reason about:

prior hardness
available ground truth
keyword/domain cues
task difficulty
output availability
trajectory count

Each policy casts a weighted vote for either Judge or Verifier.

Conceptually:

Ground truth present?        -> favor Verifier
Execution output available?  -> favor Verifier
Open-ended wording?          -> favor Judge
Very low routing confidence? -> fall back to Judge

The implementation is more nuanced than that sketch, but the spirit is the same. Evaluation mode is chosen by accumulating evidence.

Step 3: aggregate confidence

The router aggregates weighted policy votes into a final confidence score. The winning strategy is selected only if its confidence is strong enough. Otherwise, the system falls back to the more general-purpose Judge path.

This creates an important separation:

a strategy can be powerful
the router can still decide that the available evidence does not justify using it for this particular task

That is a better design than letting every downstream evaluator silently assume the task is well suited to its own strengths.

Step 4: expose the routing trace

The output of a routing decision includes:

the selected strategy
the final confidence
the raw signals
every policy vote
a human-readable reasoning trace

I care about this part the most, because evaluation pipelines already accumulate ambiguity. A score without a path to understanding how it was obtained is hard to debug. The Oracle makes one critical source of ambiguity observable: why this evaluator was chosen in the first place.

Evaluating trajectories, not just answers

The project uses the term trajectory deliberately.

A trajectory is a candidate task-solving attempt. It may contain:

the final answer
intermediate reasoning or planning
code
tool calls
execution output
an optional reward signal

This is especially relevant for agents. When a coding agent fixes a bug, the thing we care about is not only the final patch. We may care about:

whether the patch addresses the requested failure mode
whether it satisfies explicit requirements
whether execution evidence supports the answer
whether two superficially plausible solutions differ materially

For text-only tasks, evaluating the final answer may be sufficient. For agents, the evaluation object often needs to be richer.

The Oracle architecture assumes that richer object from the beginning.

A concrete example: the N+1 query bug

One example in the repository asks three agents to fix an N+1 query problem. The original function loads orders first, then issues one SQL query per order to load items. That means:

51 queries for 50 orders
5,001 queries for 5,000 orders

The task asks for a constant-query solution and provides both ground truth and test cases.

The original bug has this shape:

def get_orders_with_items(user_id: int) -> list[dict]:
    orders = db.execute(
        "SELECT * FROM orders WHERE user_id = ?",
        [user_id],
    )

    for order in orders:
        order["items"] = db.execute(
            "SELECT * FROM items WHERE order_id = ?",
            [order["id"]],
        )

    return orders

The loop is the problem. The first query fetches the orders, then each order triggers another query for its items.

Three candidate trajectories are evaluated:

one rewrites the query with a JOIN
one performs a batched prefetch with WHERE IN
one adds an lru_cache around the inner item lookup

The first two are legitimate fixes, though they make different tradeoffs. The third sounds plausible because caching often improves performance. But it does not solve the stated problem. On a cold cache, query count still grows with the number of orders.

The misleading fix looks like this:

@lru_cache(maxsize=256)
def fetch_items(order_id: int) -> list[dict]:
    return db.execute(
        "SELECT * FROM items WHERE order_id = ?",
        [order_id],
    )


def get_orders_with_items(user_id: int) -> list[dict]:
    orders = db.execute(
        "SELECT * FROM orders WHERE user_id = ?",
        [user_id],
    )

    for order in orders:
        order["items"] = fetch_items(order["id"])

    return orders

This may help repeated calls for the same order, but it does not change the first-run query pattern:

expected: query_count == O(1)
actual:   query_count == 1 + number_of_orders

A real fix changes the query shape. For example, the JOIN trajectory uses one SQL query and groups the rows afterward:

SELECT o.*, i.*
FROM orders o
LEFT JOIN items i ON i.order_id = o.id
WHERE o.user_id = ?;

This is exactly the sort of case where evaluator choice matters.

Why Judge alone is risky here

A holistic Judge may recognize that the cache-based answer is weaker. But it is also possible for that answer to benefit from surface plausibility:

it uses a familiar optimization technique
it contains concrete code
it appears to address performance

If the score is driven too much by presentation quality, the wrong candidate can become competitive.

Why Verifier is the better fit

The task has stronger evidence:

explicit correctness requirements
test cases
expected behavioral properties
a measurable performance invariant: query count must not scale with N

That is precisely the situation where the Oracle should route toward the Verifier. The evaluation problem is not mainly aesthetic. It is evidential.

The interesting part is not merely that Verifier can help. The more general point is that the Oracle can identify this task shape before evaluation begins.

Judge and Verifier are not rivals

It is tempting to treat this as a winner-take-all comparison:

Verifier is more objective
Judge is more flexible
one must be superior

I do not think that framing is useful.

Each strategy fails differently.

Failure modes of Judge

A Judge can overvalue fluent or confident language, blur correctness and style, and struggle to separate close technical alternatives without stronger evidence.

Failure modes of Verifier

A Verifier has its own failure modes. It inherits bad ground truth, can overfit to incomplete criteria, becomes brittle when tests are narrow, and quietly underperforms on tasks that are fundamentally subjective.

The Oracle does not eliminate these problems. It tries to reduce one avoidable problem: choosing the wrong mode of evaluation for the task at hand.

When the Oracle should ask for help

There is another failure mode worth making explicit: sometimes the task itself is underspecified.

Suppose three architecture recommendations are all defensible, but the best one depends on a missing fact about team size, latency goals, compliance constraints, or deployment environment. No evaluator should pretend confidence if the information needed to decide was never supplied.

The fuller design in this project explores a Human Oracle escalation path for those cases. When evaluation evidence runs out, the system can ask a targeted clarifying question, incorporate the answer, and re-evaluate.

The point is not to put a person in the loop by default. It is to avoid manufactured certainty when a decision depends on missing context.

That same principle motivates the router itself:

do not hide assumptions
do not oversell confidence
make uncertainty inspectable

What this suggests about evaluation design

The Oracle pattern leads to a broader design lesson.

Good evaluation systems share a few properties. Evaluator should match task structure: open-ended and evidence-grounded tasks are different, and treating them the same introduces avoidable error. Selection should be explicit, part of the system design rather than a hidden convention buried in notebook code or benchmark glue. Inspectability should be preserved: routing traces, criteria, and confidence should be artifacts you can actually look at. And the system should admit when evidence is insufficient. A strong evaluator is no substitute for missing context, and escalation can be the right move.

Where this pattern is useful

I think LLM-as-an-Oracle is especially relevant for:

coding agents that generate several candidate patches
benchmark pipelines that mix factual, creative, and executable tasks
tool-using agents whose outputs include both text and action traces
research workflows comparing evaluators across heterogeneous task families
production systems that want one evaluation interface without pretending every task should be judged the same way

In all of those cases, evaluator selection is part of the problem.

Treating it as a first-class system component is cleaner than standardizing on a single evaluation method and compensating later with increasingly elaborate exceptions.

Closing thought

The evaluator should fit the task.

LLM-as-an-Oracle is my attempt to turn that principle into a concrete system: route between judgment and verification, expose the reasons for the choice, and leave room for human escalation when neither automated path has enough evidence.

Zheng et al. introduce LLM-as-a-Judge; the LLM-as-a-Verifier framework develops evidence-sensitive verification; CritiqueLLM studies critique generation for evaluation; and pairwise ranking prompting shows how LLMs can be used directly as rankers. https://arxiv.org/abs/2306.05685 https://llm-as-a-verifier.notion.site/ https://arxiv.org/abs/2311.18702 https://arxiv.org/abs/2306.17563 ↩
Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” 2023. https://arxiv.org/abs/2306.05685 ↩ ↩²
“LLM-as-a-Verifier: A General-Purpose Verification Framework.” https://llm-as-a-verifier.notion.site/ ↩

My book is out!

2015-03-05T00:00:00+00:00

Software Patterns, Knowledge Maps, and Domain Analysis

My book is about domain analysis and software patterns. Briefly, it shows how to build software systems as systems of patterns, called Knowledge Maps.

It is now in the global markets. Its publication date was December 2014. Please check it out:

Figure 1. Front cover of my book

The citation (in BibTex) of this book is presented as follows:

@book{hsanchez, 
      author = {M. E. Fayad, H. A. Sanchez, S. G .K. Hegde, 
                A. Basia, and A. Vakil.},
      title  = {Software Patterns, Knowledge Maps, and 
                Domain Analysis},
      publisher = {Auerbach Publications},
      year      = 2014,
      address   = {Boca Raton, Florida},
      isbn      = {978-1466571433}
}

Cyberbullying detection in social networks

2012-04-29T00:00:00+00:00

According to Google Analytics and to other sources, my data mining (ISM 245 class) project titled “Twitter Bullying Detection” is getting some attention. An earlier version of the project report was hosted on the co-author of the project’s website. This version is good in terms of communicating the project’s main idea and motivation. However, I think it is always important to have the latest version available. Therefore, I decided to post this latest version here.

Please feel free to contact me if you have any questions about the content of that report. Thanks.