Surfaced only where it can be seen
Surfaced is observable only where the host searches or defers tools. Where every tool is loaded up front it is 1 by construction and labelled “loaded”.
“The first principle is that you must not fool yourself, and you are the easiest person to fool.”
The unit is an eligible intent: a need your product is a correct candidate for. The host then decides whether to reach for an external tool at all.
Illustrative stream, not data
the chance you are put in front of the model at all
Entered the host’s candidate set through tool search or deferred loading. Where a host loads every tool up front this is 1 by construction and labelled “loaded”.
Lost here: Not found
the chance you are chosen, given you were surfaced
Chosen as the first external integration among the eligible pool, against every competitor that could do the same job.
Lost here: Not chosen
the chance the call goes out, given you were chosen
The arguments validate, and the call is dispatched and not blocked.
Lost here: Blocked or invalid
the chance your server answers, given it was called
Your server returns a non-error, non-empty result.
Lost here: Error or empty
the chance the task is done, given your server answered
Deterministic checks pass and the calibrated judge agrees the request was met.
Lost here: Task not done
Outside Qjoined from your own outcome events, reported beside the score
The signup, order or booking that followed, joined from your outcome events. It sits outside Q: a business result, not an agent decision.
Lost here: No business event
Each stage is conditional on the one before, so their product is the chance a request completes through you.
The model itself competes. Correct self-service is reported separately, never as your success and never as a metadata defect.
Surfaced is observable only where the host searches or defers tools. Where every tool is loaded up front it is 1 by construction and labelled “loaded”.
A request with no tool is split into answered without search, searched then declined, self-served correctly and unresolved, because they mean very different things.
Correct native self-service is not a success for your product and not a defect in your metadata. It is excluded from the levers we recommend.
Competitor execution and completion are measured only for public read-only tools or participating vendors. Otherwise competitor evidence stops at selection.
A false invoke is a call on a negative prompt, where the right answer was no tool. It is reported as the False-Invoke Rate, beside the score.
Each prompt runs several times, because one run is not a measurement. Rates carry an interval and their n, and repeats are deflated for clustering.
Each metric carries its evidence classes and its sample. Classes are never pooled silently.
Public registry and directory data
Our runs through the OpenAI and Anthropic APIs and headless agents
Analytics a host gave you, uploaded by you
Your server’s telemetry: the SDK, a gateway, OTLP or a PostHog import
Business events from the outcome API, webhooks or Stripe
Opt-in human runs in the consumer apps, logged as assisted runs
Shown wherever a score appears, and never deeper than what a customer has connected.
R + S
Registry facts and API-approximated selection. Execution and completion are not measured.
R + S + sandbox E/C
Execution and task completion measured in a vendor-approved sandbox. Not production usage.
R + S + T, optional O/H
Real tool usage from the vendor’s own telemetry, joined to outcomes where connected.
A Public integration never shows a full end-to-end InvokeRank: without measured execution and completion it shows selection evidence only.
The stage rates are conditional, so their product is already the end-to-end rate. We do not weight stages: a 10% loss at any stage removes the same 10% of completed tasks.
The expected value over the posterior of Q, on a 0 to 100 scale.
Q = 0.273InvokeRank 27Selected takes 57% of the loss: the biggest lever
| Metric | Definition | Shown with |
|---|---|---|
| InvokeRank | for the integration in a cell, surface or roll-up | 90% credible interval |
| Stage rates | Posterior rates for Surfaced, Selected, Invoked, Executed and Completed | Jeffreys interval and n |
| Loss share | : where the completed tasks are lost; sums to 1 | A bar per stage |
| Lever | against the intent median or the best competitor | Points of InvokeRank available |
| False-Invoke Rate | Jeffreys interval | |
| Reliability | : the chance all three repeats succeed, per prompt | Beside InvokeRank |
| Invoke Share | Your Q over the summed Q of every eligible integration, contested intents only | Interval |
| Invoke Rating | Conditional-logit first choice with position and branded-prompt terms, on an Elo-like scale | Bootstrap interval over prompts |
| Host Self-Serve Rate | Eligible prompts the host answered correctly with no external tool ÷ eligible prompts | Interval and n |
| No-tool Decline Rate | Eligible prompts where the host searched, chose no tool and did not self-serve ÷ eligible prompts | Interval and n |
Public monitoring shows 90% intervals, to make operational uncertainty visible without pretending to be a hypothesis test. Experiment ship and no-ship decisions use 95% intervals on paired prompts.
“intervals overlap - reported as tied”
When two 90% intervals overlap, we say so and suppress the rank. It is a conservative display rule, not a formal claim that a test proved the two equal.
| To show | At least |
|---|---|
| Any rate on a public page | and an interval half-width of 15 points or less |
| A public rank position | and no overlap with the next position; otherwise a reported tie |
| A full InvokeRank on a public page | Execution and completion measured: public tools or a participating vendor |
| Otherwise | Surfaced, Selected, Invoke Share and Invoke Rating only, labelled S |
A suite written with one product in mind quietly rewards it. Category construction is therefore independent of whoever asked for the measurement.
Build a capability union from every materially relevant indexed integration in the category: tool names, descriptions, schemas, listings and public docs.
Add real user language where it exists: search queries, support questions, community threads and interviews. It is evidence of how people ask, never proof that a product is eligible.
Write neutral intent definitions from that union. No product name, tool name or description wording appears in an intent.
Record eligibility for every integration and intent as eligible, ineligible or unknown, with the reason, the evidence and the reviewer. Every change is versioned.
Score each integration on its own eligible intents. Invoke Share and Selection Share use only contested intents, where at least two integrations are independently eligible.
Reject prompts that depend on context the test does not provide, such as an email nobody attached, rather than letting them depress every score.
is the set of integrations independently eligible for intent k, at least two of them. The host’s own abilities and the no-tool path stay available in the run, but never enter the vendor-against-vendor denominator. Where competitor completion cannot be measured, the same set gives Selection Share, and it is labelled as such.
The first suite we ran was written with one integration in mind. It stays as an instrument-validation corpus, because it produced our first baselines and controls, and it is not published as a competitive benchmark until it is rebuilt under these rules.
Invoke Share answers which integration wins when tools compete. The Agent Resolution Split answers the bigger question: every eligible run ends in exactly one class, and the classes sum to 100%.
Eligible positive prompts where the host used no external tool and the completion rubric says the user was satisfied, over all eligible positive prompts. Shown with its interval and n.
A low InvokeRank on an intent the host answers well by itself is a distribution reality, not automatically a metadata problem. Levers exclude native self-service unless your product is needed for correctness, freshness, private data or an action.
Before a suite’s numbers are trusted, a round of controls checks that it measures what it claims, on the same prompts, repeats and pinned versions. Rounds repeat after any change of host, runner model or runner, and at least quarterly.
| Control | What runs | It holds when | A failure means |
|---|---|---|---|
| Placebo | The base pool, twice | The 95% paired interval of the difference covers 0 | The comparison reports effects that are not there |
| Blinded metadata | Your tool names, titles, descriptions, schema text and instructions removed | Selection falls by at least half | Runs do not read the metadata you would change, so no experiment means anything |
| Clone symmetry | Two identical copies under matched opaque labels, labels and positions swapped across rotations | The balanced share of the pair covers one half (90% interval) | Hosts favour a label or position; that bias is the resolution limit between near-identical competitors |
| Decoy | The base pool plus an unrelated public integration | Chosen on no more than 2% of prompts | The noise floor below which selection differences mean nothing |
| Real usage | The synthetic tool mix against your real tool calls on the same host, or a labelled human-assisted sample | Positive rank agreement, with its uncertainty | The prompts miss real demand, or the API differs from the consumer app |
Our first clone control ran two identical copies of one integration under different server names. The host clearly preferred one name, which showed that hosts are sensitive to server labels by themselves. We keep that result as evidence of label bias. Clone controls now use matched opaque labels, swapped across rotations together with positions, and any bias left is reported as the resolution limit between near-identical competitors.
Control copies are never public, never probed, never counted and never anyone’s competitor.
Agent distribution is full of accepted wisdom that doesn’t hold up. Here’s what we’ve learned.
A directory listing puts you on the shelf. It says nothing about whether an agent reaches for you when a customer asks for the job your product does.
Distribution happens at the moment an agent compares every tool that could do the work, and its own knowledge, and picks one. That moment is what we measure, request by request.
Agents read your tool names, descriptions and schemas, then weigh them against every competitor that could do the same job. None of that shows up in your own analytics.
So we run neutral requests for the jobs in your category and record which tool each agent considered, chose, called and finished the task with.
Ask an agent the same question twice and it can pick a different tool. A single demo proves nothing either way.
We repeat every request, report a probability with an interval, and report overlapping results as tied. Every number says how sure we are and where it came from.
ChatGPT, Claude, Cursor and Codex each find and rank tools their own way, and each changes on its own schedule.
Winning on one tells you little about the others, so each host is measured on its own terms and never blended into a single flattering score.
Copy written to trick a model into choosing you gets caught by directory review and by the next model update, and it makes agents worse for everyone.
We only recommend changes that make your tool genuinely easier to choose correctly. Rankings are never for sale, and we never automate consumer chat apps.
How we work with you from the first baseline onwards.
We map where your product is listed, build the category’s intents neutrally, and run a baseline. You learn your InvokeRank on each host, with intervals, and which competitors agents choose instead.
1.Every request is followed from shortlist to finished task, so the drop has an address: not found, found but not chosen, chosen but failed, or answered by the host itself. The biggest lever goes first.
2.A clearer description, a better schema, a fixed error. Each change runs against the same prompts as the baseline and ships only when selection moves beyond its 95% interval.
3.Models update and competitors ship new tools. InvokeSignal reruns on a schedule and alerts you only when a change falls outside its interval, with the evidence attached.
4.Synthetic runs use the OpenAI and Anthropic APIs with tool search, and are labelled “API-approximated, not the consumer app”. ChatGPT and Claude themselves can rank tools differently.
Both providers’ terms forbid it. We also never see anyone’s ChatGPT or Claude conversations: consumer-app evidence comes only from opt-in human-assisted runs or your own telemetry.
Synthetic results are compared with real telemetry, or a separately labelled human-assisted sample, before any broad claim about user behaviour. Synthetic, telemetry and assisted evidence are never pooled.
Execution and completion are measurable only for public read-only tools or participating vendors. Elsewhere we report Selection Share and say so.
Every table is scoped to a category, surface, runner model, locale, prompt version and window, and states its weights.
Early numbers carry wide intervals and a Low sample pill. Public rates appear only above the thresholds above.
Every change to the method is dated here. A change that moves numbers gets a new version, and every score names the version it was computed with.
Method frozen as v1.0. Invoke Share and Selection Share count only contested intents, where at least two integrations are independently eligible. No-tool runs split into native self-service, searched then declined, and unresolved, reported as the Agent Resolution Split with a Host Self-Serve Rate; correct self-service is excluded from metadata levers. The clone control uses matched opaque labels swapped across rotations. Overlapping 90% intervals read "intervals overlap - reported as tied", a display rule rather than an equality test.
Instrument checks before a suite is trusted: placebo, blinded metadata, clone, decoy and real-usage controls, repeated after any host, model or runner change.
Interval coverage: the strength of the shared prior is capped at the other intents’ effective trials, so intervals no longer collapse onto the shared mean when intents look alike. Point estimates are unchanged.
Method v1 published. InvokeRank is 100 times the posterior mean probability that an eligible request completes through the integration, with a 90% interval, from a beta-binomial funnel model. Replaces the weighted geometric mean of stage rates proposed in the first product draft.
The research paper on this method and its instrument checks: InvokeRank: measuring how AI agents surface, select and use tools, with an instrument-validation case study on TapTax.
Paste your MCP server, plugin or company URL for the registry evidence now. If we’re a fit, we’ll run a free baseline with intervals on the requests that matter.