Official MCP RegistryListed
OpenMark
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
First seen 18 Oct 2026. Evidence as of 25 Oct 2026.
9
Tools
From an anonymous probe
1
Source listings
Each with its own history
15
Recorded changes
Since first seen
Tools
| Tool | Description | Behaviour |
|---|---|---|
| estimate_cost | Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running. | Not declared |
| get_account | The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits. | Not declared |
| get_results | Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others. | Not declared |
| get_task_format | How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name. | Not declared |
| list_models | Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort. | Not declared |
| prepare_attachments | Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark. | Not declared |
| run_benchmark | Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast). | Not declared |
| validate_task | Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look. | Not declared |
| watch_task | Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate. | Not declared |
Change history
- validate_task: tool added
- get_task_format: input schema changed (+topic)
- get_task_format: description changed (+"all 18" +"modes," +"Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name.")
- server instructions changed (+"check it with validate_task (free; fix what it reports),")
- watch_task: input schema changed (+model)
- watch_task: description changed (+"when" +"ship, OpenMark re-tests up to 4 of them next to the user's pick" +"it. Each check uses")
- run_benchmark: input schema changed (+custom_timeout, +fail_fast, +max_credits, +max_tokens, +preset, +temp_preference, +temperature_discovery, +timeout_profile)
- run_benchmark: description changed (+"job_id and the expected cost;" +"credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast)." -"job_id;")
- prepare_attachments: description changed (+"a one-hour" +"link (no API key needed) for" +"documents, with the exact curl command.")
- list_models: description changed (+"tokens), as text and as a structured list." -"tokens).")
- get_results: input schema changed (+show_misses)
- get_results: description changed (+"latency), the tests each model missed with its answer," -"latency) plus")
- estimate_cost: tool added
- server instructions changed (+"check the price with estimate_cost (or pass max_credits to run_benchmark; both take the app's run settings such as max_tokens, preset and temperature_discovery)," +"get_results ranks by accuracy by default (highest score; ties go to the cheaper, then the faster model) and also lists the best pick for cost, speed and consistency." +"if a much cheaper")
- Listed (registry)
| Source | Listing | First seen | Last seen | Versions |
|---|---|---|---|---|
| Official MCP Registry | ai.openmark/openmark | 18 Oct 2026 | 25 Oct 2026 | 1 |