Official MCP RegistryListed
IA-QA Eval & RAG — 38 tools
LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.
First seen 26 Oct 2026. Evidence as of 5 Nov 2026.
38
Tools
From an anonymous probe
1
Source listings
Each with its own history
50
Recorded changes
Since first seen
Tools
| Tool | Description | Behaviour |
|---|---|---|
| ab_test_report | Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant. | Read-only |
| analyze_responses | Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). For a 2-way head-to-head with structural JSON diff use compare_responses instead. | Read-only |
| bm25_score | Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores. | Read-only |
| build_rag_prompt | Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution. | Read-only |
| compare_models | Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose. | Read-only |
| compare_responses | Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs. | Read-only |
| consistency_check | Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing. | Read-only |
| context_window_check | Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk. | Read-only |
| conversation_analyze | Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA. | Read-only |
| cot_analyzer | Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation. | Read-only |
| count_tokens | Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count. | Read-only |
| embedding_similarity | Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication. | Read-only |
| estimate_llm_cost | Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models. | Read-only |
| extract_json_from_text | Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON. | Read-only |
| find_tool | Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web. | Read-only |
| generate_ci_workflow | Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract. | Read-only |
| generate_eval_yaml | Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type. | Read-only |
| get_testing_guidelines | Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic. | Read-only |
| hallucination_check | Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". For entailment use run_semantic_tests (NLI/embedding) or a calibrated judge. | Read-only |
| list_llm_models | List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers. | Read-only |
| llm_format_check | Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing. | Read-only |
| llm_generate | Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited. | Read-only |
| llm_json_schema_check | Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing. | Read-only |
| llm_output_validator | Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features. | Read-only |
| metamorphic_check | Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness. | Read-only |
| model_info | Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI. | Read-only |
| needle_haystack_generate | Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed. | Read-only |
| prompt_test_suite | Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation. | Read-only |
| rag_relevance_rank | Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API. | Read-only |
| rate_tool | Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." }) | Changes data |
| rerank_evaluate | Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels. Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18). | Read-only |
| response_quality_score | Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). For meaning against a reference answer, use run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown. | Read-only |
| run_eval_contract | Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model. | Read-only |
| run_semantic_tests | Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build. | Read-only |
| run_vlm_test_suite | Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, an overall PASS/WARNING/FAIL against threshold, and latency. BYOK: your key for the model's provider; the image is fetched only from a public address. | Read-only |
| similarity_score | Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode. | Read-only |
| test_skill | Test a SKILL.md (Agent Skills format). (1) lint: the spec's frontmatter rules, no model — an error returns BLOCK; gate on lint.valid. (2) triggering: a model sees only the names + descriptions of this skill and its neighbours and picks which to load (temperature 0); trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK. Your own scenarios give a reproducible gate; generated ones are a signal. Unanswered requests make the run INCONCLUSIVE; so do scenarios left when a call reaches its 22 s budget (split long runs). Default model: Groq on the server key. | Read-only |
| token_budget_calculator | Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models). | Read-only |
Change history
- find_tool: input schema changed
- find_tool: description changed (+"descriptions and, on request," +"dev," +"eval, integrations,")
- server instructions changed (+"These tools" +"operated" +"Report a verdict as a careful tester would: gate review means it could not decide, so say that and never round it to a pass; relay what a result says it did not prove, and explain why it matters to the person you work for.")
- token_budget_calculator: input schema changed
- token_budget_calculator: description changed (+"the model's context window (any" +"from list_llm_models)." -"context window. Supports 25+ models.")
- run_eval_contract: input schema changed
- run_eval_contract: description changed (+"per scenario; return passed (true, false, null if undecided)" +"a" +"(PASS, FAIL, PARTIAL). Expectations are assertions that fail")
- rate_tool: description changed (+"Give" +"Maintainers" +"ratings,")
- llm_generate: input schema changed
- llm_generate: description changed (+"openai/gpt-oss-120b, qwen/qwen3.8-27b" +"Rate-limited." -"openai/gpt-oss-120b")
- find_tool: description changed (+"Queries" +"logged 90 days, screened for credentials and personal data." -"Call this when you")
- estimate_llm_cost: description changed (+"counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models." -"counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.")
- context_window_check: input schema changed
- compare_models: input schema changed
- compare_models: input schema changed (-use_case)
- compare_models: description changed (+"Put" +"price per 1k tokens," +"reasoning, provider, pricing page. Facts only: names the cheapest")
- token_budget_calculator: input schema changed
- test_skill: input schema changed
- test_skill: description changed (+"(Agent" +"format)." +"rules, no")
- similarity_score: input schema changed
- run_vlm_test_suite: input schema changed
- run_vlm_test_suite: description changed (+"Test" +"vision model (OpenAI, Anthropic" +"Gemini) on one image: each")
- run_semantic_tests: input schema changed
- run_eval_contract: input schema changed
- run_eval_contract: description changed (+"Run an IA-EVAL" +"suite (.ia-eval.yaml contract):" +"the")
- response_quality_score: input schema changed
- rerank_evaluate: input schema changed
- rerank_evaluate: description changed (+"retrieval:" +"query," +"and")
- rate_tool: input schema changed
- rag_relevance_rank: input schema changed
- prompt_test_suite: input schema changed
- needle_haystack_generate: input schema changed
- model_info: input schema changed
- metamorphic_check: input schema changed
- metamorphic_check: description changed (+"check: does" +"stay the same" +"(case, typo,")
- llm_output_validator: input schema changed
- llm_json_schema_check: input schema changed
- llm_generate: input schema changed
- llm_format_check: input schema changed
- list_llm_models: input schema changed
- hallucination_check: input schema changed (+cases)
- get_testing_guidelines: input schema changed
- generate_eval_yaml: input schema changed
- generate_ci_workflow: input schema changed
- generate_ci_workflow: description changed (+"IA-QA:" +"(needs" +"tools")
- find_tool: input schema changed
- extract_json_from_text: input schema changed
- estimate_llm_cost: input schema changed
- embedding_similarity: input schema changed
- count_tokens: input schema changed
| Source | Listing | First seen | Last seen | Versions |
|---|---|---|---|---|
| Official MCP Registry | io.github.JcJamet/ia-qa-eval | 26 Oct 2026 | 5 Nov 2026 | 2 |