f614e9bb98
Some checks failed
CI/CD / lint (pull_request) Successful in 1m4s
CI/CD / test (pull_request) Successful in 1m23s
CI/CD / typecheck (pull_request) Successful in 2m36s
CI/CD / smoke (pull_request) Failing after 1m53s
CI/CD / build (pull_request) Successful in 4m16s
CI/CD / publish (pull_request) Has been skipped
Measured winner from the DGX-Spark bake-off (toolsim.py, 145-tool catalog): a
curated favourite/<tool> shortlist + the full all/<server>/<tool> catalog + a
load-bearing "prefer favourite/ first" instruction nearly halved wander (37→20)
and 2.5x'd first-pick (2→5/8) vs a flat catalog. The instruction is load-bearing;
enriching descriptions did not help.
- New mcplocal plugin `favourite-index.ts`: composes AFTER gate (no-ops while
gated), reshapes the ungated upstream catalog into favourite/ + all/, injects
the instruction (onInitialize), and rewrites presented names back to canonical
server/tool in onToolCallBefore so normal routing + content-pipeline still run.
Gate/agent virtual tools pass through untouched; favourites are upstream-only.
- compose.ts: onInitialize now concatenates plugin instructions (was first-non-null)
so favindex can contribute its banner alongside the gate's.
- Per-project config `Project.favouriteIndex` {enabled, tools[], maxFavourites};
surfaced to the proxy via discovery; wired at project-mcp-endpoint when enabled.
- Usage derivation: mcpd tool-usage ranking over tool_call_trace events
(normalizing presented names → canonical), GET /api/v1/audit/tool-usage, and
`mcpctl favourites suggest|list`.
- CLI: `create project` gains --favourite/--favourite-index/--max-favourites;
favouriteIndex round-trips through get -o yaml | apply -f. Completions regenerated.
- Tests: plugin unit (presentation, rewrite routing, gated no-op, collisions),
compose merge, canonicalizeToolName, buildFavouriteIndex, + a live smoke test.
- Docs: docs/tool-presentation.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
87 lines
3.4 KiB
Markdown
87 lines
3.4 KiB
Markdown
# Tool presentation: the favourite-index
|
||
|
||
Big, flat MCP tool catalogs make LLMs **wander** — they call wrong tools before
|
||
the right one, or find it then keep exploring. mcpctl (the proxy) decides *what
|
||
tool list the model sees*, so presentation is a lever we can pull.
|
||
|
||
We measured it. On a 145-tool catalog with a wandering model
|
||
(`kubernetes-deployment/scripts/model-eval/toolsim.py`), the winning presentation
|
||
— **favindex** — nearly halved wander (37→20) and 2.5×'d first-pick (2→5/8) vs the
|
||
flat catalog. Enriching descriptions did **not** help; scoping/indexing did.
|
||
(Full table + method in the `feedback_llm_tool_scoping` memory.)
|
||
|
||
## What it does
|
||
|
||
When enabled for a project, the proxy presents tools in **two namespaces** plus a
|
||
**load-bearing instruction**:
|
||
|
||
- `favourite/<tool>` — a curated, usage-weighted shortlist of the commonly-used
|
||
tools, listed **first**.
|
||
- `all/<server>/<tool>` — the **full** catalog (the escape hatch).
|
||
- An `initialize` instruction: *"PREFER favourite/<tool> … use all/<server>/<tool>
|
||
only if nothing in favourites fits."*
|
||
|
||
The instruction is not optional — the same two namespaces **without** it scored
|
||
materially worse in the bake-off. Curate **and** tell the model to check
|
||
favourites first.
|
||
|
||
Favourites are a *subset*: leave rare/cloud tools in `all/` only, so the model
|
||
falls back correctly for the long tail.
|
||
|
||
## How it composes
|
||
|
||
favourite-index is a proxymodel plugin
|
||
(`src/mcplocal/src/proxymodel/plugins/favourite-index.ts`) composed **after** the
|
||
gate/default plugin:
|
||
|
||
- While a session is **gated**, only `begin_session` is visible — favourite-index
|
||
no-ops. After `begin_session` ungates, the catalog is reshaped into
|
||
`favourite/` + `all/`.
|
||
- Gate/agent virtual tools (`read_prompts`, `propose_prompt`, `agent-<name>`
|
||
chat) pass through unchanged at the top level. Favourites are drawn only from
|
||
the upstream MCP catalog.
|
||
- Calls to `favourite/…` / `all/…` are rewritten back to the canonical
|
||
`server/tool` in `onToolCallBefore`, so normal upstream routing **and** the
|
||
content-pipeline (pagination/sectioning) still run.
|
||
|
||
## Configuring it
|
||
|
||
Per-project, stored on the project as `favouriteIndex`
|
||
(`{ enabled, tools: ["server/tool", …], maxFavourites? }`). Round-trips through
|
||
`get project -o yaml | apply -f -`.
|
||
|
||
```bash
|
||
# enable + pin the common tools
|
||
mcpctl create project myproj --force --favourite-index \
|
||
--favourite k8s/get_pods --favourite k8s/get_pod_logs \
|
||
--favourite gitea/create_pull_request --max-favourites 15
|
||
|
||
# what the proxy will present
|
||
mcpctl favourites list --project myproj
|
||
```
|
||
|
||
### Deriving favourites from usage
|
||
|
||
Every tool call is recorded (`AuditEvent` `tool_call_trace`). Rank them and pin
|
||
the common ones:
|
||
|
||
```bash
|
||
mcpctl favourites suggest --project myproj --top 20 --window 30
|
||
```
|
||
|
||
This normalizes presented names back to canonical `server/tool` (so `all/…` and
|
||
`favourite/…` calls collapse), ranks by call count, and prints a ready-to-run
|
||
`create project … --favourite …` line. `★` marks tools already pinned.
|
||
|
||
## Validating a change first
|
||
|
||
Before shipping a presentation change, run it through the fake-catalog harness —
|
||
it caught "enrichment doesn't help" in ~1h instead of a wasted build:
|
||
|
||
```bash
|
||
toolsim.py <model> favindex all # vs `terse all` (flat baseline)
|
||
```
|
||
|
||
Then confirm it reproduces through the **real** proxy on a tool-heavy project
|
||
(presentation + routing: `src/mcplocal/tests/smoke/favourite-index.test.ts`).
|