InnerMatch Benchmarks — Does Online Learning Actually Out-Retrieve Static Baselines?

InnerMatch's pitch is simple to state and easy to overclaim: it starts from a principled baseline and gets better as people interact with it. Demos are a good way to feel that, but a demo is one browser session — it doesn't tell you whether the improvement holds up over hundreds of queries, whether it depends on the embedding space it's operating in, or how it stacks up against just asking a large language model to do the ranking itself. So we built two offline benchmarks that reproduce our live demos exactly, replace one browser session with a simulated user asking 500 questions in a row, and score every retrieval automatically against a known ground truth — then stress-tested our own assumptions with a battery of ablations, and benchmarked a real Claude-opus-5 model as a fifth, qualitatively different competitor.

The Setup, Once, for Both Benchmarks

Both benchmarks follow the same protocol. A simulated user asks a random query built from the corpus itself (so ground truth is known by construction), each retrieval method returns its ranked top results, and — for the methods that learn — the top result gets a simulated vote: +1 if relevant, −1 if not, exactly matching the vote our live demos send. Everything runs with a fixed random seed (42) for reproducibility, and every adaptive method accumulates its learned weights across the entire 500-query session, the same way a real browser session persists until the page resets.

Five standard information-retrieval metrics are computed per query and macro-averaged: Precision@K and Recall@K (share of the top-K results that are relevant, and share of all relevant items the top-K captured), HitRate@K (did at least one relevant item appear at all), NDCG@K (rewards relevant items ranked higher, not just present), and MRR (how close to #1 the first relevant result landed, across the full ranking, not just the top-K).

Beyond the headline comparison, each benchmark now runs 22 method variants: a static baseline, plain InnerMatch and its untrained ("cold") counterpart, a topic-routing variant that spreads feedback across multiple weight matrices instead of one, a kernel-trick variant that reformulates the same learning rule to work in an implicit nonlinear space, a combination of the two, and a continuous-kernel scoring mode — each run in both a sparse (TF-IDF) and a dense (LSA) embedding space. On top of all of that, a real claude-opus-5 model was asked to rank each corpus from scratch on every query, with no embeddings and no memory between calls, as a genuinely different point of comparison.

One thing we were careful about: both benchmarks substitute local embeddings (TF-IDF and truncated-SVD/LSA) for the paid API (Voyage AI) our live demos use in production. That substitution is disclosed everywhere in the underlying methodology, and it's the reason the numbers below should be read as evidence about the learning mechanism, not as a claim about production-embedding accuracy.

Benchmark 1 — Matching Daily-Life Tools by Description

Our four-panel MCP (Model Context Protocol) tool-matching demo lets a user describe an everyday task in plain language — planning a trip, tracking a workout, managing a playlist — and see four retrieval methods each surface their top-5 matching tools from a shared corpus. Two panels work in a keyword (TF-IDF) space, two work in a semantic (dense embedding) space; within each pair, one is a static baseline and one is InnerMatch, adapting to votes. This benchmark reproduces that comparison offline.

Corpus and Queries

121 real MCP tools across 12 "Daily Life" categories — Calendar & Scheduling, Tasks & To-dos, Notes & Writing, Email & Communication, Weather & News, Shopping & Finance, Music & Entertainment, Travel & Maps, Health & Fitness, Social Media, Food & Recipes, and Home & Smart Devices — each with a name, category, and one-sentence description (Google Calendar, Todoist, Spotify, Google Maps, Apple Health, and 116 others, hand-curated, not synthetically padded). A query is built by sampling 2–4 tokens from a randomly chosen tool's own name and description (e.g. "playlists discover music" from Spotify's description); the sampled tool's category is the ground truth, so a retrieved item counts as relevant if its category matches — this benchmark tests category-level retrieval, not exact-tool recovery.

Because the demo's keyword and semantic panels are genuinely different technologies, the benchmark needed two genuinely different local spaces: real TF-IDF for the keyword panels, and truncated SVD / LSA (50 components, 62.2% explained variance, fit on the same TF-IDF matrix) standing in for the dense embedding panels — a real co-occurrence-based semantic space, not a relabeled copy of the TF-IDF vectors.

Results

500 simulated queries, top-5 shown, seed 42:

Full run also included topic-routing, kernel-trick, combined, and continuous-kernel variants of each trained method (17 more panels), plus the LLM baseline — see the dedicated sections below.

What the Numbers Say

The semantic space beats the keyword space on every metric, learning aside. Even euclidean, the untrained semantic baseline, beats tfidf_im, the trained keyword method, on every metric except MRR. A query like "playlists discover music" shares real vocabulary with Spotify's own description, but a query sampled from a different tool in the same category may share almost no literal wording — TF-IDF misses that connection where LSA's latent structure can still catch it.

The two spaces respond to online learning in opposite directions. In the keyword space, learning helps clearly: tfidf_im climbs from 0.5200 to 0.5832 Precision@5, first half of the session to second half — a real, accumulating gain from feedback alone, with the vector space itself untouched. In the semantic space, learning doesn't help and mildly regresses: innermatch sits below its own untrained counterpart innermatch_cold on every single metric, and its within-session curve is flat rather than climbing.

A plausible mechanism: InnerMatch's feedback rule adapts one weight per vector dimension. TF-IDF dimensions are sparse and individually interpretable — "this term matters, that one doesn't" is exactly the kind of thing a per-dimension weight can learn from a single vote. LSA/SVD dimensions are dense latent factors, each an entangled mixture of many original terms with no individual meaning — there's much less clean, learnable structure for the same update rule to exploit, and each feedback step risks perturbing a dimension that was doing useful work for unrelated reasons. Online learning adds real value on top of a weaker starting point, and adds slight noise on top of an already-strong one. The question is whether that's a fact about this one corpus — the next benchmark answers it.

Benchmark 2 — Retrieving Physics Facts for a RAG Pipeline

Our second demo is a two-panel retrieval-augmented-generation setup: a user types a physics question, and two methods each surface their top-2 supporting facts. One panel (Euclidean) is a static nearest-neighbor baseline that never changes; the other (InnerMatch) starts identical but adapts to votes across the conversation.

Corpus and Queries

25 physics topics × 10 facts each = 250 facts (Rotational Dynamics, Thermodynamics, Electromagnetism, and 22 others), each topic carrying its own keyword list. A query joins 2–4 of a randomly chosen topic's keywords into a short string (e.g. "torque angular centripetal"); the sampled topic is the ground truth. As in Benchmark 1, this substitutes for the production Voyage AI embedding, but this time in two genuinely different local spaces — TF-IDF (ported line-for-line from our own tokenizer and IDF formula) and truncated-SVD/LSA (50 components, 48.5% explained variance) — run as fully independent sessions with their own candidate pools and learned weights, so this benchmark can ask the same sparse-vs-dense question Benchmark 1 did, on a completely different corpus.

Both spaces' InnerMatch variants rerank their own top-50 candidate pool, pre-filtered by Euclidean distance in that same space — the same payload-limiting step production uses. innermatch_cold/lsa_innermatch_cold are InnerMatch with the weight matrix left at its untouched starting point; they isolate how much of any InnerMatch result is just "reranking a pre-filtered pool with normalized vectors" versus the learning itself. innermatch/lsa_innermatch are the same setup with weights evolving via feedback across all 500 queries.

Results

500 simulated queries, top-2 shown, one continuous session per space, seed 42:

What the Numbers Say

Which embedding space wins here is genuinely different from Benchmark 1 — and that's informative on its own. In MCP tools, LSA beat TF-IDF on every metric, cold or trained. Here the cold baselines are much closer, and partly reversed: lsa_euclidean edges out euclidean on Precision and NDCG, but lsa_innermatch_cold trails innermatch_cold on Recall (0.1845 vs. 0.2610), HitRate (0.7831 vs. 0.8614), and MRR (0.7957 vs. 0.8313). Trained, TF-IDF pulls decisively ahead: innermatch (0.8133 P@2) beats lsa_innermatch (0.6014 P@2) by a wide margin on every metric. "Which space is better" turns out to be corpus- and metric-dependent, not a fixed property of sparse vs. dense — which makes the next finding, that does hold across both corpora, worth trusting more, not less.

TF-IDF: every metric orders the same way, static < untrained-rerank < trained, and two separate effects stack. Reranking a pre-filtered pool with normalized vectors — zero learning involved — already beats raw full-corpus Euclidean distance on every metric, most sharply on Recall (0.139 → 0.261). Online learning then adds a comparably sized further jump on top: Recall climbs to 0.327, NDCG to 0.821, Precision to 0.813. The learning curve isolates that second effect directly: Precision@2 goes from 0.759 in the first 250 queries to 0.868 in the last 250, a real, accumulating gain purely from feedback, with the starting weights fixed at the same point as innermatch_cold. Both static methods stay flat within query-sampling noise — exactly what's expected of methods that structurally cannot learn.

LSA: the same regression from Benchmark 1, on a completely different corpus. lsa_innermatch drops below its own untrained baseline on every metric — Precision 0.716 → 0.601, Recall 0.185 → 0.160, NDCG 0.721 → 0.604, MRR 0.796 → 0.675 — and its learning curve stays essentially flat and below lsa_innermatch_cold the whole session (0.590 → 0.612 P@2, never climbing back). This is the same directional result as the MCP-tools benchmark's dense panel, produced by an independently-fit SVD on an unrelated corpus of physics facts instead of software tools.

The Core Finding, Now Confirmed Twice

Put the two benchmarks side by side and one pattern survives everything else that changes between them — different corpus, different query style, different candidate pool, an independently-fit embedding space each time:

Sparse, per-dimension-interpretable feature spaces (TF-IDF) get a real, accumulating benefit from InnerMatch's online learning, twice. Dense, entangled latent spaces (LSA/SVD) don't, twice — and in Physics RAG the regression is proportionally larger than in MCP Tools, not smaller. That rules out "it's just this corpus" as the explanation. It's a genuine property of how a per-dimension linear update rule interacts with feature geometry, and it's the single most load-bearing thing either benchmark found — worth knowing before choosing a feature representation for a new InnerMatch integration, not a footnote.

Stress-Testing Our Own Assumptions

Both benchmarks also ran three additional mechanisms on every panel: topic-routing (feedback splits across several weight matrices instead of one, spawning a new one when a vote doesn't fit the existing ones well), a kernel trick (the same learning rule, reformulated to work through an implicit nonlinear RBF feature space instead of an explicit weight matrix), and a continuous proximity kernel built for dense, signed features in [-1, 1]. We ran all three specifically to see how they behave outside — and inside — the conditions each was designed for.

None of these three failures is a fluke of one run — the causal mechanism behind each was verified independently on synthetic vectors built the same way as each corpus, and each one replicates across both benchmarks. We're including this section because a benchmark that only reports the numbers that flatter the product isn't a benchmark; the kernel-scoring and kernel-trick modes are real, useful tools for the conditions they were designed for, and this is exactly the boundary condition that tells us where not to reach for them.

How Does This Compare to Just Asking an LLM?

Every method above shares one thing: it's built on precomputed embeddings and local arithmetic. We also ran a qualitatively different competitor — send the entire catalog (all 121 tools, or all 250 physics facts) plus the raw query straight to claude-opus-5 on every single call, and ask it to return the top-10 most relevant items via structured output. No embedding pipeline, no candidate-pool prefilter, no weight matrix accumulating across the session — every call is stateless and sees the full catalog fresh. Because real API calls cost money and take seconds rather than milliseconds, this baseline ran on the first 30 of the 500 queries in each benchmark, not the full set, with the catalog cached via prompt caching so repeated calls didn't re-bill the full corpus each time.

*MRR is a lower bound for the LLM specifically: the model was asked for only its top 10 items, so a relevant match ranked below that (never possible for the local methods, which always rank the full corpus) scores as "not found." No other method in either benchmark carries this structural disadvantage.

Claude-opus-5 wins on accuracy, decisively. On identical queries, it beats every same-subset static baseline, and it even out-scores the best trained InnerMatch numbers from the full 500-query runs — 0.7667 vs. tfidf_im's 0.5516 in MCP Tools, 0.9333 vs. innermatch's 0.8133 in Physics RAG — despite being completely stateless, with zero accumulated learning across its 30 calls, against methods that benefit from a whole session's worth of feedback. Two caveats belong right next to that result: the LLM ran on 30 queries against the other methods' 500, so a different 30-query slice could shift these numbers; and the MRR figures above are lower bounds for the reason in the table footnote.

The cost of that accuracy is real, and it's not close. Every embedding-based method in either benchmark is local numpy arithmetic — zero LLM tokens, sub-millisecond to low-double-digit-millisecond latency per query, measured the same way across the full 500-query runs. Claude-opus-5 spends roughly 9,900 tokens per query in MCP Tools and roughly 15,100 in Physics RAG (mostly the cached catalog, re-read rather than re-billed on every call), and its mean latency — 2.63s and 3.30s respectively — is on the order of 60,000–85,000× slower than InnerMatch's own fastest local variant, and still 90–160× slower than the slowest of the 22 local methods in either report. Run at the same rate across all 500 queries instead of 30, either benchmark's LLM baseline would take on the order of 20–30 minutes of serial API time and 15–17× the tokens actually spent here.

None of that makes an LLM-as-ranker a bad choice outright — it needs no embedding pipeline, no candidate-pool tuning, and no online-learning mechanism to reach these numbers, and for a low-volume, high-stakes retrieval task with a generous latency budget, sending the whole catalog to a frontier model on every query is a completely reasonable design. It's just not a drop-in substitute for InnerMatch's approach at any real query volume — which is exactly the gap InnerMatch's local, sub-millisecond, learning-from-feedback design is built to fill.

What This Tells Us

Taken together, these two benchmarks say something more specific than "InnerMatch works." They say: online learning is not a universal multiplier. It adds clear, measurable, accumulating value on top of sparse, per-dimension-interpretable feature spaces — confirmed on two independent corpora, not one — and its benefit is flat-to-negative on top of dense latent spaces where individual dimensions carry no standalone meaning, also confirmed twice. Topic-routing can partially recover that dense-space regression, but only when the feedback stream is heterogeneous enough to actually fragment into multiple topics — it isn't a free fix. And a real frontier LLM, asked to rank the same corpus from scratch on every single call, is the most accurate method either benchmark tested — at a token and latency cost several orders of magnitude larger than any local method, which is precisely the tradeoff a fast, adaptive, embedding-based ranker like InnerMatch exists to avoid.

It's also a demonstration of something we care about independent of any one product number: reproducibility. Both benchmarks parse their corpora directly from the same source our live demos use, run with a fixed seed, and disclose every substitution — embedding model, candidate-pool size, scoring mode, sample size for the LLM baseline — rather than tuning the setup to a favorable outcome. Where a mechanism failed (the kernel trick, the continuous kernel, dense-space learning without enough topic fragmentation), we diagnosed and reported the actual cause rather than smoothing it over. If a number here looks worse than you'd expect, that's deliberate — it's supposed to be the same benchmark whether the result flatters us or not.