Skip to content
“The first principle is that you must not fool yourself, and you are the easiest person to fool.”
Richard Feynman
Method v1.0 Updated 5 Oct 2026

The InvokeRank methodology

InvokeRank noun
The probability that an eligible request to an AI agent is completed through your product, times 100, shown with its 90% interval and the evidence behind it.
01 The funnel

Every eligible request, followed to the end

The unit is an eligible intent: a need your product is a correct candidate for. The host then decides whether to reach for an external tool at all.

Illustrative stream, not data

Stage 01 / 06

SurfacedIn the candidate set

P(D)the chance you are put in front of the model at all

Entered the host’s candidate set through tool search or deferred loading. Where a host loads every tool up front this is 1 by construction and labelled “loaded”.

Lost here: Not found

Q=Surfaced
Eligible intentHost decisionreach for a tool, or not?
External-tool pathCounts toward InvokeRank
  1. Surfaced
  2. Selected
  3. Invoked
  4. Executed
  5. Completed
  6. Outcome
Q=P(D)⁢P(S|D)⁢P(I|S)⁢P(E|I)⁢P(C|E)

Each stage is conditional on the one before, so their product is the chance a request completes through you.

No-tool pathReported separately
Answered without searchSearched, then declined
Self-served correctlyUnresolved

The model itself competes. Correct self-service is reported separately, never as your success and never as a metadata defect.

An eligible intent reaches a host decision. On the external-tool path a request is surfaced, selected, invoked, executed and completed, then joined to an outcome. On the no-tool path the host answers without searching or searches and declines, and the need is either self-served correctly or left unresolved.

Rules that keep it honest

Surfaced only where it can be seen

Surfaced is observable only where the host searches or defers tools. Where every tool is loaded up front it is 1 by construction and labelled “loaded”.

Four kinds of no tool

A request with no tool is split into answered without search, searched then declined, self-served correctly and unresolved, because they mean very different things.

Self-service is not your failure

Correct native self-service is not a success for your product and not a defect in your metadata. It is excluded from the levers we recommend.

Competitor evidence stops at what we can see

Competitor execution and completion are measured only for public read-only tools or participating vendors. Otherwise competitor evidence stops at selection.

Wrong calls count against you

A false invoke is a call on a negative prompt, where the right answer was no tool. It is reported as the False-Invoke Rate, beside the score.

Every request, repeated

Each prompt runs several times, because one run is not a measurement. Rates carry an interval and their n, and repeats are deflated for clustering.

02 Evidence

Every number says where it came from

Each metric carries its evidence classes and its sample. Classes are never pooled silently.

R: Registry

Public registry and directory data

S: Synthetic

Our runs through the OpenAI and Anthropic APIs and headless agents

H: Host

Analytics a host gave you, uploaded by you

T: Telemetry

Your server’s telemetry: the SDK, a gateway, OTLP or a PostHog import

O: Outcome

Business events from the outcome API, webhooks or Stripe

A: Assisted

Opt-in human runs in the consumer apps, logged as assisted runs

Evidence depth

Shown wherever a score appears, and never deeper than what a customer has connected.

Public

R + S

Registry facts and API-approximated selection. Execution and completion are not measured.

Verified

R + S + sandbox E/C

Execution and task completion measured in a vendor-approved sandbox. Not production usage.

Connected

R + S + T, optional O/H

Real tool usage from the vendor’s own telemetry, joined to outcomes where connected.

A Public integration never shows a full end-to-end InvokeRank: without measured execution and completion it shows selection evidence only.

How a measured score reads
Sample data
InvokeRank6290% interval 55 to 690255075100
Evidence
R Registry: presentS Synthetic: presentH Host: not connectedT Telemetry: presentO Outcome: not connectedA Assisted: not connected
Sample
2,340 runs52 prompts2 surfaces14 days
Runner
Claude Messages API (tool search)OpenAI Responses API (tool search)API-approximated, not the consumer app
03 The score

A probability, not a blend

The stage rates are conditional, so their product is already the end-to-end rate. We do not weight stages: a 10% loss at any stage removes the same 10% of completed tasks.

The end-to-end rate
Q=rD×rS×rI×rE×rC
  1. Surfaced
  2. Selected
  3. Invoked
  4. Executed
  5. Completed
InvokeRank=100×𝔼[Q]

The expected value over the posterior of Q, on a 0 to 100 scale.

Worked example: 100 eligible requests
Sample data

Q = 0.273InvokeRank 27Selected takes 57% of the loss: the biggest lever

Each stage, each intent
yik∼Binomial(nik,rik)
Successes out of trials
rik∼Beta(μkκk,(1−μk)κk)
A prior pooled across the integration’s intents
r~ik=yik+μkκknik+κk
The shrunken estimate: noisy intents move toward the pool
One intent with 9 of 12, pulled toward its pool
Sample data
0%25%50%75%100%Raw 9/12 = 75%Posterior 58%Pool 45%
  • Pooled prior
  • Posterior
  • 90% interval 42% to 73%
  1. For each stage, counts are pooled across the intents of one integration on one surface, and the shared prior is fitted by beta-binomial marginal maximum likelihood (method of moments as the fallback), after deflating for clustered repeats.
  2. The prior’s strength is capped at the other intents’ effective trials, so intervals keep their coverage when intents look alike.
  3. The stage posteriors are independent, so the posterior of Q is exact by Monte Carlo: 4,000 draws, multiplied.
  4. When a stage never fires, the stage after it reverts to its prior. No epsilon floors.
  5. Single proportions use the Jeffreys interval.
  6. Roll-ups across intents are arithmetic, ∑w×Q, with the weights stated: Q is a probability, so the roll-up is an expected value.
Published metrics
MetricDefinitionShown with
InvokeRank100×𝔼[Q] for the integration in a cell, surface or roll-up90% credible interval
Stage ratesPosterior rates for Surfaced, Selected, Invoked, Executed and CompletedJeffreys interval and n
Loss shareLk=−lnrk∑j−lnrj: where the completed tasks are lost; sums to 1A bar per stage
LeverQ×(rkrefrk−1)+ against the intent median or the best competitorPoints of InvokeRank available
False-Invoke Ratecalls on negative promptsnegative promptsJeffreys interval
Reliabilitypass3: the chance all three repeats succeed, C(c,3)C(n,3) per promptBeside InvokeRank
Invoke ShareYour Q over the summed Q of every eligible integration, contested intents onlyInterval
Invoke RatingConditional-logit first choice with position and branded-prompt terms, on an Elo-like scaleBootstrap interval over prompts
Host Self-Serve RateEligible prompts the host answered correctly with no external tool ÷ eligible promptsInterval and n
No-tool Decline RateEligible prompts where the host searched, chose no tool and did not self-serve ÷ eligible promptsInterval and n
04 Intervals

90% to watch. 95% to decide.

Public monitoring shows 90% intervals, to make operational uncertainty visible without pretending to be a hypothesis test. Experiment ship and no-ship decisions use 95% intervals on paired prompts.

“intervals overlap - reported as tied”

Reported tieSample data
Competitor A41%Your product36%20%30%40%50%60%overlap

When two 90% intervals overlap, we say so and suppress the rank. It is a conservative display rule, not a formal claim that a test proved the two equal.

Public evidence thresholds
To showAt least
Any rate on a public pageneff≥30 and an interval half-width of 15 points or less
A public rank positionneff≥96 and no overlap with the next position; otherwise a reported tie
A full InvokeRank on a public pageExecution and completion measured: public tools or a participating vendor
OtherwiseSurfaced, Selected, Invoke Share and Invoke Rating only, labelled S
05 Eligibility and fairness

Categories built without the customer’s thumb on the scale

A suite written with one product in mind quietly rewards it. Category construction is therefore independent of whoever asked for the measurement.

  1. Build a capability union from every materially relevant indexed integration in the category: tool names, descriptions, schemas, listings and public docs.

  2. Add real user language where it exists: search queries, support questions, community threads and interviews. It is evidence of how people ask, never proof that a product is eligible.

  3. Write neutral intent definitions from that union. No product name, tool name or description wording appears in an intent.

  4. Record eligibility for every integration and intent as eligible, ineligible or unknown, with the reason, the evidence and the reviewer. Every change is versioned.

  5. Score each integration on its own eligible intents. Invoke Share and Selection Share use only contested intents, where at least two integrations are independently eligible.

  6. Reject prompts that depend on context the test does not provide, such as an email nobody attached, rather than letting them depress every score.

Contested intents only

Invoke Sharei,k=Qi,k∑j∈EkQj,k

Ek is the set of integrations independently eligible for intent k, at least two of them. The host’s own abilities and the no-tool path stay available in the run, but never enter the vendor-against-vendor denominator. Where competitor completion cannot be measured, the same set gives Selection Share, and it is labelled as such.

Our own first suite is not a benchmark

The first suite we ran was written with one integration in mind. It stays as an instrument-validation corpus, because it produced our first baselines and controls, and it is not published as a competitive benchmark until it is rebuilt under these rules.

06 Agent Resolution Split

Does the host use any tool at all?

Invoke Share answers which integration wins when tools compete. The Agent Resolution Split answers the bigger question: every eligible run ends in exactly one class, and the classes sum to 100%.

Your integration completedCounts toward your InvokeRank.
A competitor completedCounts toward theirs, on the same eligible prompts.
Another external toolA tool outside the eligible set took the request.
Native self-serviceNo tool, and the same completion rubric says the need was met. Reported, never a metadata defect.
Searched, then declinedThe host looked for a tool, chose none, and did not answer correctly.
UnresolvedNo tool and the need was not met, or the host gave up after searching.
One category on one host, the same eligible prompts
Sample data
  • Any external integration completed 67%
  • Native self-service completed 21%
  • Unresolved or declined 12%

Host Self-Serve Rate

Eligible positive prompts where the host used no external tool and the completion rubric says the user was satisfied, over all eligible positive prompts. Shown with its interval and n.

Why it changes the advice

A low InvokeRank on an intent the host answers well by itself is a distribution reality, not automatically a metadata problem. Levers exclude native self-service unless your product is needed for correctness, freshness, private data or an action.

07 Instrument controls

We test the instrument before we trust it

Before a suite’s numbers are trusted, a round of controls checks that it measures what it claims, on the same prompts, repeats and pinned versions. Rounds repeat after any change of host, runner model or runner, and at least quarterly.

Instrument controls
ControlWhat runsIt holds whenA failure means
PlaceboThe base pool, twiceThe 95% paired interval of the difference covers 0The comparison reports effects that are not there
Blinded metadataYour tool names, titles, descriptions, schema text and instructions removedSelection falls by at least halfRuns do not read the metadata you would change, so no experiment means anything
Clone symmetryTwo identical copies under matched opaque labels, labels and positions swapped across rotationsThe balanced share of the pair covers one half (90% interval)Hosts favour a label or position; that bias is the resolution limit between near-identical competitors
DecoyThe base pool plus an unrelated public integrationChosen on no more than 2% of promptsThe noise floor below which selection differences mean nothing
Real usageThe synthetic tool mix against your real tool calls on the same host, or a labelled human-assisted samplePositive rank agreement, with its uncertaintyThe prompts miss real demand, or the API differs from the consumer app

Why the clone control changed

Our first clone control ran two identical copies of one integration under different server names. The host clearly preferred one name, which showed that hosts are sensitive to server labels by themselves. We keep that result as evidence of label bias. Clone controls now use matched opaque labels, swapped across rotations together with positions, and any bias left is reported as the resolution limit between near-identical competitors.

Control copies are never public, never probed, never counted and never anyone’s competitor.

08 Principles

What the method is built on

Agent distribution is full of accepted wisdom that doesn’t hold up. Here’s what we’ve learned.

Principle 01

Listing is not distribution

A directory listing puts you on the shelf. It says nothing about whether an agent reaches for you when a customer asks for the job your product does.

Distribution happens at the moment an agent compares every tool that could do the work, and its own knowledge, and picks one. That moment is what we measure, request by request.

Principle 02

A description is not a decision

Agents read your tool names, descriptions and schemas, then weigh them against every competitor that could do the same job. None of that shows up in your own analytics.

So we run neutral requests for the jobs in your category and record which tool each agent considered, chose, called and finished the task with.

Principle 03

One run is not a measurement

Ask an agent the same question twice and it can pick a different tool. A single demo proves nothing either way.

We repeat every request, report a probability with an interval, and report overlapping results as tied. Every number says how sure we are and where it came from.

Principle 04

One host is not the market

ChatGPT, Claude, Cursor and Codex each find and rank tools their own way, and each changes on its own schedule.

Winning on one tells you little about the others, so each host is measured on its own terms and never blended into a single flattering score.

Principle 05

Gaming the model does not work

Copy written to trick a model into choosing you gets caught by directory review and by the next model update, and it makes agents worse for everyone.

We only recommend changes that make your tool genuinely easier to choose correctly. Rankings are never for sale, and we never automate consumer chat apps.

Approach

How we work with you from the first baseline onwards.

Discovery and baseline

We map where your product is listed, build the category’s intents neutrally, and run a baseline. You learn your InvokeRank on each host, with intervals, and which competitors agents choose instead.

1.

Find the stage you lose at

Every request is followed from shortlist to finished task, so the drop has an address: not found, found but not chosen, chosen but failed, or answered by the host itself. The biggest lever goes first.

2.

Change one thing, measure again

A clearer description, a better schema, a fixed error. Each change runs against the same prompts as the baseline and ships only when selection moves beyond its 95% interval.

3.

Watch for drift and competitors

Models update and competitors ship new tools. InvokeSignal reruns on a schedule and alerts you only when a change falls outside its interval, with the evidence attached.

4.
09 Limitations

What these numbers cannot tell you

API runs approximate the consumer apps

Synthetic runs use the OpenAI and Anthropic APIs with tool search, and are labelled “API-approximated, not the consumer app”. ChatGPT and Claude themselves can rank tools differently.

We never automate chatgpt.com or claude.ai

Both providers’ terms forbid it. We also never see anyone’s ChatGPT or Claude conversations: consumer-app evidence comes only from opt-in human-assisted runs or your own telemetry.

Synthetic is not real usage

Synthetic results are compared with real telemetry, or a separately labelled human-assisted sample, before any broad claim about user behaviour. Synthetic, telemetry and assisted evidence are never pooled.

Competitor evidence can stop at selection

Execution and completion are measurable only for public read-only tools or participating vendors. Elsewhere we report Selection Share and say so.

Benchmark share is not market share

Every table is scoped to a category, surface, runner model, locale, prompt version and window, and states its weights.

Small samples are wide

Early numbers carry wide intervals and a Low sample pill. Public rates appear only above the thresholds above.

10 Method changelog

No hidden score changes

Every change to the method is dated here. A change that moves numbers gets a new version, and every score names the version it was computed with.

  1. v1.0

    Method frozen as v1.0. Invoke Share and Selection Share count only contested intents, where at least two integrations are independently eligible. No-tool runs split into native self-service, searched then declined, and unresolved, reported as the Agent Resolution Split with a Host Self-Serve Rate; correct self-service is excluded from metadata levers. The clone control uses matched opaque labels swapped across rotations. Overlapping 90% intervals read "intervals overlap - reported as tied", a display rule rather than an equality test.

  2. v1

    Instrument checks before a suite is trusted: placebo, blinded metadata, clone, decoy and real-usage controls, repeated after any host, model or runner change.

  3. v1

    Interval coverage: the strength of the shared prior is capped at the other intents’ effective trials, so intervals no longer collapse onto the shared mean when intents look alike. Point estimates are unchanged.

  4. v1

    Method v1 published. InvokeRank is 100 times the posterior mean probability that an eligible request completes through the integration, with a 90% interval, from a beta-binomial funnel model. Replaces the weighted geometric mean of stage rates proposed in the first product draft.

The research paper on this method and its instrument checks: InvokeRank: measuring how AI agents surface, select and use tools, with an instrument-validation case study on TapTax.

See where you stand before you change anything

Paste your MCP server, plugin or company URL for the registry evidence now. If we’re a fit, we’ll run a free baseline with intervals on the requests that matter.