Patronus API Calls Returning Nothing? Check If Your Evaluator Exists First

You just signed up for Patronus, got your API Key, wired your code up to an evaluator name—and the call fails or comes back empty. Plenty of developers have hit this wall. The Key is fine. The code is probably fine too. What's actually missing is hiding in a console panel you may never have opened.

The Root Cause: Evaluators Are Account-Level Resources

Patronus Evaluators are account-level resources. They don't exist automatically—unless you create them by hand in the console, the API has nothing to reference.

In other words, your code may assume something like this already lives in your account:

**What "account-level resource" actually means**

On most API platforms, resources fall into roughly two layers. One is the "service-level resource," like a model inference endpoint—as long as your API Key is valid, you can call it directly. The other is the "account-level resource," which users explicitly create under their own account with a unique ID, name, and configuration; the API can only reference objects that already exist. Similar designs show up in OpenAI's Assistant/Fine-tune jobs, AWS Lambda functions, and Hugging Face Endpoints. What these resources share in common: the caller can't just make up a name and hope for the best. The server has to be able to find a matching object in your account, or it returns a 404, an empty result, or a resource-not-found error. Patronus Evaluators fall into the latter category—which is exactly why a valid API Key alone can't make the call succeed.
# what your code assumes already exists in your account
evaluator: hallucination-judge
type: Judge
scale: 0-100
pass_threshold: 70

But in reality, that configuration doesn't exist anywhere in your account. The API can't find the evaluator name, so it returns empty or throws an error.

The design itself isn't complicated. The problem is that most developers—especially automated agents—default to a "code-first" integration flow: write the calling logic first, then go back and configure the platform. Patronus flips that around—it requires you to set things up in the console first.

The Correct Order of Setup

Here's the flow that actually works:

1. Open app.patronus.ai and go to the Evaluators page.
2. Click Create Evaluator and pick the type your code expects: Judge (LLM-as-judge with custom criteria), Classifier (pass/fail classification), Retrieval (RAG quality evaluation), or Custom (Python-based custom evaluator).
3. For a custom Judge, explicitly write the evaluation criteria, set the scoring range (usually 0-100) and pass threshold (e.g., 70), and use {task_input} and {task_output} placeholders in the judging prompt.
4. Test the evaluator in the console and confirm it works.
5. Only then call it from the API.

Don't reorder these five steps. Skip the first four and jump straight to writing API calls, and you'll run into the "returns empty" problem described at the top.

Judge vs. Classifier vs. Retrieval vs. Custom: What's the difference?

These four evaluator types serve different evaluation scenarios. Judge uses a custom prompt to have an LLM score against your written criteria—good for subjective dimensions like hallucination, tone, and instruction-following, often paired with 0-100 scoring and a pass threshold. Classifier does binary or multi-class classification and outputs "pass/fail" labels—good for deterministic tasks like format validation and sensitive-content detection. Retrieval is built specifically for RAG pipelines, evaluating relevance and faithfulness between retrieved chunks and answers. Custom lets you upload Python-based evaluation logic—good for scenarios that can't be expressed via prompt, or that need external tools or precise computation. Which one you pick depends on whether your goal is "subjective judgment" or "rule-based," and whether you're evaluating "end-to-end answer quality" or "retrieval-stage quality."

Why Agents Fall Into This Trap So Easily

For automated agents, this pitfall hits harder—and it comes down to order-of-operations inertia.

Most API integrations really are code-first: write the request, debug auth, tweak headers and body. Agents follow that pattern when troubleshooting—first suspect the Key is dead, then check headers, then verify parameter formats. An hour can vanish, and nobody thinks to check whether the referenced evaluator actually exists.

One visit to the console would end the whole investigation. But an agent doesn't have a "open browser, log into console" step, so it's easy for it to spin in circles.

Why agents struggle to discover this kind of gap on their own

An automated agent's troubleshooting path is typically inward-facing: it can only use the tools it has—send requests, read responses, adjust parameters. "Log into the console, create an evaluator" happens in the browser UI and usually isn't in the agent's tool whitelist. When the API returns an empty result instead of a 404 with a clear error code, it's even harder for the agent to tell whether the problem is on the request side or the resource side—so it keeps tweaking headers, retrying, and swapping parameter formats. To break the loop, you need an explicit resource-existence check in the agent's initialization flow, or a "create-if-not-exists" API endpoint that lets the agent handle the prerequisite steps itself.

A Simple Troubleshooting Rule

If a Patronus call fails immediately after signup and API Key validation passes, stop debugging the request itself and go create the evaluator first.

The value of this rule is that it redirects troubleshooting from the "code layer" to the "platform layer." A missing evaluator is a configuration gap, not a code bug—continuing to fuss over request parameters accomplishes nothing.

For the full session record, see this link: vectle.com/skills/skl_3hTdITXRiF2aL3PGyIPyaQ

Lessons for the Development Workflow

This small issue reflects a bigger pattern: when platforms define certain resources as "user must create first," docs and onboarding need to emphasize that prerequisite loudly. For human developers, one console visit solves it. For automated pipelines, you need a resource-existence check before calling—or an explicit creation step in the initialization script.

Putting "create first, then call" prominently in the integration docs is far cheaper than an hour of post-hoc debugging.

One-line Summary

Patronus calls often fail because Evaluators must be created in the console first—they're account-level resources.

Paragraph Summary

The article explains the real reason Patronus API calls return empty or error right after signup: Evaluators are account-level resources that don't exist automatically. You must first create them in the app.patronus.ai console before the API can reference them. Many developers default to a "code-first" flow—writing the calling logic first—only to find there's no matching object in their account. The correct order is: open the console, create the evaluator, pick the type (Judge/Classifier/Retrieval/Custom), configure criteria and thresholds, test in the console, and only then call it via API. Automated agents are especially prone to this trap because they troubleshoot the usual API suspects—Key, headers, parameters—without ever checking whether a resource exists. The rule the article offers: when calls fail immediately after signup and Key validation passes, go create the evaluator instead of continuing to debug the request.