{"id":"com.completionkit/evals","slug":"com-completionkit-evals","title":"CompletionKit","description":"Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.","version":"1.0.0","publisher":{"kind":"domain","id":"completionkit.com","label":"completionkit.com","url":"https://completionkit.com"},"website":"https://completionkit.com","repository":"https://github.com/homemade-software-inc/completion-kit","remotes":[{"type":"streamable-http","url":"https://completionkit.com/mcp","headers":[{"description":"Your CompletionKit API token as: Bearer <token>. Create one under Settings > API tokens.","isRequired":true,"isSecret":true,"name":"Authorization"}]}],"packages":[],"published":"2026-07-18","updated":"2026-07-18","status":"online","auth":"","latency_ms":1014,"checked":"2026-11-19","uptime":100,"checks":3,"tools":[{"name":"prompts_list","title":"","description":"List all prompts","read_only":false},{"name":"prompts_get","title":"","description":"Get a prompt by ID","read_only":false},{"name":"prompts_create","title":"","description":"Create a prompt","read_only":false},{"name":"prompts_update","title":"","description":"Update a prompt. Changing the template or model of the published version, or editing any version that already has runs, creates a new DRAFT version (current=false) instead of changing it, so an agent's edits don't go live without a gate. Run and compare the draft, then promote it with prompts_publish. An unpublished draft with no runs, or a name, description or tag change on a published version wi","read_only":false},{"name":"prompts_delete","title":"","description":"Delete a prompt","read_only":false},{"name":"prompts_publish","title":"","description":"Publish a prompt version, making it the current version","read_only":false},{"name":"prompts_suggest_improvement","title":"","description":"Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).","read_only":false},{"name":"runs_list","title":"","description":"List all runs","read_only":false},{"name":"runs_get","title":"","description":"Get a run by ID, including \"metric_averages\": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.","read_only":false},{"name":"runs_create","title":"","description":"Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.","read_only":false},{"name":"runs_update","title":"","description":"Update a run","read_only":false},{"name":"runs_delete","title":"","description":"Delete a run","read_only":false},{"name":"runs_generate","title":"","description":"Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.","read_only":false},{"name":"runs_regrade","title":"","description":"Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.","read_only":false},{"name":"runs_rerun","title":"","description":"Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.","read_only":false},{"name":"runs_retry_failures","title":"","description":"Re-run only the failed responses of a run, optionally limited to specific response ids via \"only\".","read_only":false},{"name":"responses_list","title":"","description":"List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use \"fields\" to drop the bodies, \"min_score\"/\"max_score\" to isolate low scorers, and sort \"score_asc\" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.","read_only":false},{"name":"responses_get","title":"","description":"Get a specific response","read_only":false},{"name":"datasets_list","title":"","description":"List all datasets","read_only":false},{"name":"datasets_get","title":"","description":"Get a dataset by ID","read_only":false},{"name":"datasets_create","title":"","description":"Create a dataset with CSV data. First row is the header. Two column names are recognized specially: \"expected_output\" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and \"actual_output\" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also avail","read_only":false},{"name":"datasets_update","title":"","description":"Update a dataset","read_only":false},{"name":"datasets_delete","title":"","description":"Delete a dataset","read_only":false},{"name":"datasets_create_from_url","title":"","description":"Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the \"expected_output\" (answer key) and \"actual_output\" (pre-made output) columns are ","read_only":false},{"name":"metrics_list","title":"","description":"List all metrics","read_only":false},{"name":"metrics_get","title":"","description":"Get a metric by ID","read_only":false},{"name":"metrics_create","title":"","description":"Create a metric with evaluation criteria. For a deterministic check set metric_type:\"check\" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path s","read_only":false},{"name":"metrics_update","title":"","description":"Update a metric. For a deterministic check set metric_type:\"check\" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets t","read_only":false},{"name":"metrics_delete","title":"","description":"Delete a metric","read_only":false},{"name":"metrics_suggest_variants","title":"","description":"Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source=\"suggestion\". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.","read_only":false},{"name":"metric_groups_list","title":"","description":"List all metric groups","read_only":false},{"name":"metric_groups_get","title":"","description":"Get a metric group by ID","read_only":false},{"name":"metric_groups_create","title":"","description":"Create a metric group","read_only":false},{"name":"metric_groups_update","title":"","description":"Update a metric group","read_only":false},{"name":"metric_groups_delete","title":"","description":"Delete a metric group","read_only":false},{"name":"metric_versions_list","title":"","description":"List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.","read_only":false},{"name":"metric_versions_publish","title":"","description":"Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.","read_only":false},{"name":"metric_versions_dismiss","title":"","description":"Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.","read_only":false},{"name":"provider_credentials_list","title":"","description":"List all provider credentials (API keys are not exposed)","read_only":false},{"name":"provider_credentials_get","title":"","description":"Get a provider credential by ID (API key is not exposed)","read_only":false},{"name":"provider_credentials_create","title":"","description":"Create a provider credential","read_only":false},{"name":"provider_credentials_update","title":"","description":"Update a provider credential. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint.","read_only":false},{"name":"provider_credentials_delete","title":"","description":"Delete a provider credential","read_only":false},{"name":"tags_list","title":"","description":"List all tags","read_only":false},{"name":"tags_get","title":"","description":"Get a tag by ID","read_only":false},{"name":"tags_create","title":"","description":"Create a tag. Color is auto-assigned.","read_only":false},{"name":"tags_update","title":"","description":"Rename a tag.","read_only":false},{"name":"tags_delete","title":"","description":"Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.","read_only":false},{"name":"agreements_list","title":"","description":"List agreements. Filter by run_id, response_id, metric_id, or created_by.","read_only":false},{"name":"agreements_create","title":"","description":"Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.","read_only":false},{"name":"judges_replay","title":"","description":"Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.","read_only":false},{"name":"judges_compare","title":"","description":"Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.","read_only":false},{"name":"promptfoo_import","title":"","description":"Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.","read_only":false},{"name":"usage_get","title":"","description":"Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.","read_only":false}],"server_info":{"name":"CompletionKit","version":"0.29.3"},"category":"analytics","tags":["data","news","health"],"kind":"remote","source":"registry","submitted":{},"first_party":true,"domain_rank":null,"score":100,"indexable":true,"first_seen":"2026-07-18","stars":0,"badges":[],"sponsor":null,"page":"https://agentsindex.org/agents/com-completionkit-evals/"}