Agents Index

CompletionKit

Online

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

Visit publisherConnectJSON

What it can do: 54 tools

  • prompts_listList all prompts
  • prompts_getGet a prompt by ID
  • prompts_createCreate a prompt
  • prompts_updateUpdate a prompt. Changing the template or model of the published version, or editing any version that already has runs, creates a new DRAFT version (current=false) instead of changing it, so an agent's edits don't go liv
  • prompts_deleteDelete a prompt
  • prompts_publishPublish a prompt version, making it the current version
  • prompts_suggest_improvementSuggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}})
  • runs_listList all runs
  • runs_getGet a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a pr
  • runs_createCreate a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.
  • runs_updateUpdate a run
  • runs_deleteDelete a run
  • runs_generateStart a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.
  • runs_regradeRe-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.
  • runs_rerunCreate and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.
  • runs_retry_failuresRe-run only the failed responses of a run, optionally limited to specific response ids via "only".
  • responses_listList responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low sc
  • responses_getGet a specific response
  • datasets_listList all datasets
  • datasets_getGet a dataset by ID
  • datasets_createCreate a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row'
  • datasets_updateUpdate a dataset
  • datasets_deleteDelete a dataset
  • datasets_create_from_urlCreate a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call a
  • metrics_listList all metrics
  • metrics_getGet a metric by ID
  • metrics_createCreate a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), jso
  • metrics_updateUpdate a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_pat
  • metrics_deleteDelete a metric
  • metrics_suggest_variantsAsk the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-me
  • metric_groups_listList all metric groups
  • metric_groups_getGet a metric group by ID
  • metric_groups_createCreate a metric group
  • metric_groups_updateUpdate a metric group
  • metric_groups_deleteDelete a metric group
  • metric_versions_listList every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.
  • metric_versions_publishPublish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's i
  • metric_versions_dismissDestroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.
  • provider_credentials_listList all provider credentials (API keys are not exposed)
  • provider_credentials_getGet a provider credential by ID (API key is not exposed)

…and 14 more.

Connect it

Claude Code
claude mcp add --transport http completionkit https://completionkit.com/mcp
Claude.ai / ChatGPT
Settings → Connectors → Add custom connector
URL: https://completionkit.com/mcp
Cursor (~/.cursor/mcp.json)
{
  "mcpServers": {
    "completionkit": {
      "url": "https://completionkit.com/mcp"
    }
  }
}
VS Code (.vscode/mcp.json)
{
  "servers": {
    "completionkit": {
      "type": "http",
      "url": "https://completionkit.com/mcp"
    }
  }
}

Questions

What does CompletionKit do?

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge. It exposes 54 tools to AI assistants.

What tools does CompletionKit provide?

54 tools, including prompts_list, prompts_get, prompts_create, prompts_update, prompts_delete, prompts_publish, prompts_suggest_improvement, runs_list, runs_get, runs_create, runs_update, runs_delete….

How do I connect CompletionKit to Claude, ChatGPT or Cursor?

Add the remote MCP endpoint https://completionkit.com/mcp as a custom connector or MCP server. Copy-paste setup is on this page.

Does CompletionKit require an account or API key?

No sign-in was needed to list its tools. Individual tools may still need an account or API key.

Is CompletionKit online?

As of 2026-11-19 our probe found it online. It answered 100% of our last 3 checks.

Who publishes CompletionKit?

completionkit.com. The official MCP Registry verified that the publisher controls this domain before accepting the listing.

Similar in Analytics & BI