Evaluators endpoints
GET /api/v1/projects/{project_id}/evaluators
Section titled “GET /api/v1/projects/{project_id}/evaluators”List evaluators
Newest first. A judge’s budget carries its month so far.
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes | |
limit |
query | integer | no | min 1, max 200 Default 50. |
cursor |
query | string or null | no | max length 512 |
| Status | Description | Body |
|---|---|---|
| 200 | Successful Response | application/json: Page_EvaluatorOut_ |
| 422 | Validation Error | application/json: HTTPValidationError |
POST /api/v1/projects/{project_id}/evaluators
Section titled “POST /api/v1/projects/{project_id}/evaluators”Add an evaluator
Check and store an evaluator. 409 NAME_TAKEN; 404 for a prompt version or credential
the project or organization lacks; 422 BUDGET_REQUIRED, RULE_NEEDS_EXPECTED_OUTPUT,
CHANNEL_KIND, JUDGE_TEMPLATE_VARIABLES or EVALUATOR_MODEL_UNPRICED. A judge names either
prompt_version_id or a built-in judge_template (relevance, faithfulness, toxicity,
format_compliance), which becomes the project’s prompt judges/<name>.
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes |
Request body (application/json, required): EvaluatorSpec
| Status | Description | Body |
|---|---|---|
| 201 | Successful Response | application/json: EvaluatorOut |
| 422 | Validation Error | application/json: HTTPValidationError |
DELETE /api/v1/projects/{project_id}/evaluators/{evaluator_id}
Section titled “DELETE /api/v1/projects/{project_id}/evaluators/{evaluator_id}”Delete an evaluator
Delete the evaluator and its runs. The scores it wrote stay.
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes | |
evaluator_id |
path | string (uuid) | yes |
| Status | Description | Body |
|---|---|---|
| 204 | Successful Response | none |
| 422 | Validation Error | application/json: HTTPValidationError |
GET /api/v1/projects/{project_id}/evaluators/{evaluator_id}
Section titled “GET /api/v1/projects/{project_id}/evaluators/{evaluator_id}”Get an evaluator
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes | |
evaluator_id |
path | string (uuid) | yes |
| Status | Description | Body |
|---|---|---|
| 200 | Successful Response | application/json: EvaluatorOut |
| 422 | Validation Error | application/json: HTTPValidationError |
PATCH /api/v1/projects/{project_id}/evaluators/{evaluator_id}
Section titled “PATCH /api/v1/projects/{project_id}/evaluators/{evaluator_id}”Edit an evaluator
Change the members sent; config takes only the fields to change and keeps its kind.
The result is checked as a create is.
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes | |
evaluator_id |
path | string (uuid) | yes |
Request body (application/json, required): EvaluatorPatch
| Status | Description | Body |
|---|---|---|
| 200 | Successful Response | application/json: EvaluatorOut |
| 422 | Validation Error | application/json: HTTPValidationError |
GET /api/v1/projects/{project_id}/evaluators/{evaluator_id}/budget
Section titled “GET /api/v1/projects/{project_id}/evaluators/{evaluator_id}/budget”Get an evaluator’s budget
The UTC calendar month so far: settled spend, the reservations of running calls and what
is left. monthly_budget_usd and remaining_usd are null for an evaluator without one.
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes | |
evaluator_id |
path | string (uuid) | yes |
| Status | Description | Body |
|---|---|---|
| 200 | Successful Response | application/json: EvaluatorBudgetStatusOut |
| 422 | Validation Error | application/json: HTTPValidationError |
GET /api/v1/projects/{project_id}/evaluators/{evaluator_id}/runs
Section titled “GET /api/v1/projects/{project_id}/evaluators/{evaluator_id}/runs”List evaluator runs
Newest first; status filters. Each run carries the scores it wrote.
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes | |
evaluator_id |
path | string (uuid) | yes | |
status |
query | "queued" | "running" | "done" | "failed" or null |
no | |
limit |
query | integer | no | min 1, max 200 Default 50. |
cursor |
query | string or null | no | max length 512 |
| Status | Description | Body |
|---|---|---|
| 200 | Successful Response | application/json: Page_EvaluatorRunOut_ |
| 422 | Validation Error | application/json: HTTPValidationError |
POST /api/v1/projects/{project_id}/evaluators/{evaluator_id}/test
Section titled “POST /api/v1/projects/{project_id}/evaluators/{evaluator_id}/test”Test an evaluator on a trace
Evaluate the trace’s last LLM span now and return the scores; no score is stored. A judge’s call is real spend and counts against the evaluator’s budget like a sampled run.
| Parameter | In | Type | Required | Notes |
|---|---|---|---|---|
project_id |
path | string (uuid) | yes | |
evaluator_id |
path | string (uuid) | yes |
Request body (application/json, required): EvaluatorTestIn
| Status | Description | Body |
|---|---|---|
| 200 | Successful Response | application/json: EvalResultOut |
| 422 | Validation Error | application/json: HTTPValidationError |