CompletionKit
OnlinePrompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
What it can do: 54 tools
prompts_listList all promptsprompts_getGet a prompt by IDprompts_createCreate a promptprompts_updateUpdate a prompt. Changing the template or model of the published version, or editing any version that already has runs, creates a new DRAFT version (current=false) instead of changing it, so an agent's edits don't go livprompts_deleteDelete a promptprompts_publishPublish a prompt version, making it the current versionprompts_suggest_improvementSuggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}})runs_listList all runsruns_getGet a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prruns_createCreate a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.runs_updateUpdate a runruns_deleteDelete a runruns_generateStart a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.runs_regradeRe-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.runs_rerunCreate and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.runs_retry_failuresRe-run only the failed responses of a run, optionally limited to specific response ids via "only".responses_listList responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scresponses_getGet a specific responsedatasets_listList all datasetsdatasets_getGet a dataset by IDdatasets_createCreate a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row'datasets_updateUpdate a datasetdatasets_deleteDelete a datasetdatasets_create_from_urlCreate a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call ametrics_listList all metricsmetrics_getGet a metric by IDmetrics_createCreate a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), jsometrics_updateUpdate a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_patmetrics_deleteDelete a metricmetrics_suggest_variantsAsk the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-memetric_groups_listList all metric groupsmetric_groups_getGet a metric group by IDmetric_groups_createCreate a metric groupmetric_groups_updateUpdate a metric groupmetric_groups_deleteDelete a metric groupmetric_versions_listList every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.metric_versions_publishPublish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's imetric_versions_dismissDestroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.provider_credentials_listList all provider credentials (API keys are not exposed)provider_credentials_getGet a provider credential by ID (API key is not exposed)
…and 14 more.
Connect it
claude mcp add --transport http completionkit https://completionkit.com/mcpSettings → Connectors → Add custom connector
URL: https://completionkit.com/mcp{
"mcpServers": {
"completionkit": {
"url": "https://completionkit.com/mcp"
}
}
}{
"servers": {
"completionkit": {
"type": "http",
"url": "https://completionkit.com/mcp"
}
}
}Questions
What does CompletionKit do?
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge. It exposes 54 tools to AI assistants.
What tools does CompletionKit provide?
54 tools, including prompts_list, prompts_get, prompts_create, prompts_update, prompts_delete, prompts_publish, prompts_suggest_improvement, runs_list, runs_get, runs_create, runs_update, runs_delete….
How do I connect CompletionKit to Claude, ChatGPT or Cursor?
Add the remote MCP endpoint https://completionkit.com/mcp as a custom connector or MCP server. Copy-paste setup is on this page.
Does CompletionKit require an account or API key?
No sign-in was needed to list its tools. Individual tools may still need an account or API key.
Is CompletionKit online?
As of 2026-11-19 our probe found it online. It answered 100% of our last 3 checks.
Who publishes CompletionKit?
completionkit.com. The official MCP Registry verified that the publisher controls this domain before accepting the listing.
Similar in Analytics & BI
-
Google MonitoringOnlineOfficialPopular
Gain visibility into the performance, availability, and health of your apps and infrastructure.
-
beehiivOnline · sign-in requiredOfficialPopular
Manage beehiiv newsletters, posts, podcasts, community, subscribers, and analytics.
-
Kong KonnectOnline · sign-in requiredOfficialPopular
Manage Kong Konnect AI and API gateways, routes, plugins, and analytics from your agent.
-
UserpilotOnline · sign-in requiredOfficialPopular
Product analytics, user and company insights, surveys and NPS, reports, dashboards, and more.
-
PaddleOnline · sign-in requiredOfficial
Query and manage Paddle Billing: pricing, subscriptions, payments, revenue reports and webhooks.
-
Avalara E-Invoicing & Live ReportingOnline · sign-in requiredOfficial
Provides access to Avalara E-Invoicing and Live Reporting APIs for document statuses and compliance