mcpctl/docs/web-search.md

# Web search and documentation lookup

Four templates. All can run self-hosted without an API key or a vendor account,
and all are deployed by mcpctl like any other server.

| Template | Package / image | Needs |
|---|---|---|
| `searxng` | `mcp-searxng` (node) | a SearXNG instance |
| `firecrawl` | `firecrawl-mcp` (node) | a Firecrawl instance |
| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing |
| `docs-mcp` | `ghcr.io/arabold/docs-mcp-server` | nothing |

For agent use, pair `searxng` (finding pages) with `firecrawl` (reading them).
`duckduckgo` does both with no infrastructure, but it scrapes from your IP and
does not hold up under agent traffic — see below.

Search and docs are different jobs, not competing options. A search engine will
hand you a 2023 blog post with a stale method signature; a docs index cannot
tell you why a daemon is crash-looping. Attach both to a project that does real
engineering work.

`docs-mcp` is the open-source replacement for Context7 / Ref.tools — same job,
but the index lives on your infrastructure and can include private repos.

## No infrastructure: `duckduckgo`

The only one with nothing behind it. It scrapes DuckDuckGo's HTML endpoint
directly, so there is no engine to run and no key to hold.

```bash
mcpctl create server websearch --from-template duckduckgo --env DDG_SAFE_SEARCH=OFF
mcpctl get instances | grep websearch     # RUNNING / healthy within a few minutes
```

Tools: `search` (`query`, `max_results`, `region`) and `fetch_content` (`url`,
`start_index`, `max_length`) for pulling a result as markdown.

Fine for a person searching occasionally; not for agents. Scraping has no SLA,
and agents search in bursts — one question becomes several parallel queries.
From a single home or office IP, DuckDuckGo starts answering CAPTCHA and stays
that way, and every further request from that IP deepens the block, including
for a SearXNG instance sharing the address. `DDG_SEARCH_BACKEND=curl` survives
some bot checks but does not change that. Use `searxng` for agent search.

The readiness probe is `fetch_content` on `https://example.com` every 300s, not
`search`. A search probe is a DuckDuckGo scrape every interval — 1,440 a day at
the 60s default — from the same IP your agents depend on. The trade: `healthy`
proves the process and its outbound fetch work, not that DuckDuckGo is still
answering you.

## Better results: `searxng`

SearXNG aggregates ~25 engines and gives you real filtering, which is what keeps
search results from flooding a context window:

- `min_score` (0.0–1.0) — relevance floor. The most useful knob; start at `0.3`.
- `time_range` — `day` / `week` / `month` / `year`. Essential for "did this break
  in the last release" questions.
- `language`, `safesearch`, `pageno`.

`web_url_read` then takes `section`, `paragraphRange`, `startChar`/`maxLength`
and `readHeadings`, so you pull one section of a long page rather than all of it.
Bounded PDF extraction is included, which covers most vendor documentation.

The cost is that SearXNG is a service you have to run. It is plain
infrastructure, not an MCP server, so mcpctl has nothing to manage it with — it
belongs in your cluster or in the compose stack:

```bash
cd stack
cp .env.example .env          # set SEARXNG_SECRET
docker compose --profile websearch up -d
```

Then point the MCP server at it:

```bash
mcpctl create secret searxng-conf --data SEARXNG_URL=http://mcpctl-searxng:8080
mcpctl create server searxng --from-template searxng --env-from-secret searxng-conf
```

### The one SearXNG gotcha

SearXNG ships with `search.formats: [html]`. Every `format=json` request against
a stock instance returns **403**, and `mcp-searxng` comes back empty with no
useful error. `stack/searxng/settings.yml` exists only to add `json` to that
list — it is the one setting with no environment-variable override.

This is also why pointing the template at a *public* SearXNG instance usually
fails: nearly all of them leave the JSON API off.

The compose healthcheck probes `format=json` specifically, so a misconfigured
instance shows up as unhealthy rather than as silently empty search results.

### Probe and engines

The template's readiness probe is `searxng_instance_info`, which reads the
instance's `/config`: a pass proves SearXNG answers without sending a query to
any engine. Do not probe with `searxng_web_search` — every interval becomes a
real search fanned out to every engine, which bills API-key engines and feeds
the IP blocks on scraping ones.

Those blocks are the long-term problem. Scraping engines (duckduckgo, google,
startpage, qwant) CAPTCHA a single IP under agent traffic. SearXNG also ships
API-key engines — `braveapi`, `kagi` — that are not subject to IP reputation;
enable at least one if agents are the main caller, and cap its spend in the
vendor's dashboard.

## Reading pages: `firecrawl`

`firecrawl` is the reader to pair with `searxng`: give it a URL, get the page's
main content as markdown. Firecrawl renders pages in a real browser, so
JavaScript-heavy pages work where a plain fetch comes back empty.

```bash
mcpctl create server firecrawl --from-template firecrawl \
  --env FIRECRAWL_API_URL=http://firecrawl-api:3002
```

Point it at a self-hosted Firecrawl (no key needed) or leave `FIRECRAWL_API_URL`
unset and set `FIRECRAWL_API_KEY` for Firecrawl's cloud. The tool agents want is
`firecrawl_scrape` with `formats: ["markdown"]` and `onlyMainContent: true`.

- **Egress port.** Self-hosted Firecrawl listens on `:3002`, outside the default
  80/443 server egress, so declare it or every call times out.
- **Search stays off** unless the Firecrawl instance has `SEARXNG_ENDPOINT`
  set, so `firecrawl_search` cannot quietly become a second scraper. Use the
  `searxng` template for search.
- **Probe:** `firecrawl_scrape` of `https://example.com` every 300s. It is a
  real outbound fetch, hence the longer interval.

## Documentation: `docs-mcp`

```bash
mcpctl create server docs --from-template docs-mcp
```

Tools: `scrape_docs` (index a site, GitHub repo, npm or PyPI package, or local
files), `search_docs` (query, optionally pinned to a version), and `fetch_url`.

Index the things you actually run — Pulumi, the Kubernetes API, Grafana, the
Terraform provider docs — rather than everything.

### Persistence

The scraped index is a SQLite file (`better-sqlite3` + `sqlite-vec`) under
`/data`. There is **no external-database mode** — no `DATABASE_URL`, no
pgvector — so a Postgres cluster cannot help here. The template declares a
volume instead:

```yaml
volumes:
  - name: data
    mountPath: /data
    sizeGb: 20
```

The backing PVC is named after the *server* (`mcpctl-docs-data`), not the
instance, so editing the server or restarting the pod re-attaches to the same
index rather than starting empty. See "Persistent volumes" in the README.

The readiness probe is `list_libraries`: no arguments, read-only, and it reads
the SQLite store, so a pass proves the volume is mounted and readable rather
than merely that the process started. It answers "No libraries indexed yet." on
a fresh instance instead of erroring, which is why `search_docs` (which needs a
library that only exists after a scrape) cannot serve as the probe.

### Embeddings

Keyword search works out of the box. Semantic search is noticeably better and
needs an embedding model. To keep it local, point the server at Ollama:

```bash
ollama pull nomic-embed-text
mcpctl create server docs --from-template docs-mcp \
  --env DOCS_MCP_EMBEDDING_MODEL=openai:nomic-embed-text \
  --env OPENAI_API_BASE=http://ollama:11434/v1 \
  --env OPENAI_API_KEY=ollama
```

Ollama's OpenAI-compatible endpoint is why the provider prefix is `openai:` —
the key is a placeholder and never leaves the network.

Changing the embedding model invalidates the index: embeddings from different
models are not comparable, so everything has to be re-scraped.

## Alternatives considered

- **Tavily / Brave / Exa / Perplexity** — better formatted results, all require
  an API key and send every query to a vendor.
- **Context7** — the popular docs MCP, but a cloud index of public-docs snippets
  only. `docs-mcp` covers the same ground locally and takes private sources.
- **agent-search** — bundles SearXNG and a 10-strategy extraction cascade in one
  deploy. Attractive, but at ~67 stars it is too young to build the stack on.
  Worth revisiting.
- **Writing our own** — the hard parts are extraction heuristics and the
  embedding pipeline, and all three projects above already solved them under
  MIT.