mcpctl/docs/web-search.md

Web search and documentation lookup

Four templates. All can run self-hosted without an API key or a vendor account, and all are deployed by mcpctl like any other server.

Template	Package / image	Needs
searxng	mcp-searxng (node)	a SearXNG instance
firecrawl	firecrawl-mcp (node)	a Firecrawl instance
duckduckgo	duckduckgo-mcp-server (python)	nothing
docs-mcp	ghcr.io/arabold/docs-mcp-server	nothing

For agent use, pair searxng (finding pages) with firecrawl (reading them). duckduckgo does both with no infrastructure, but it scrapes from your IP and does not hold up under agent traffic — see below.

Search and docs are different jobs, not competing options. A search engine will hand you a 2023 blog post with a stale method signature; a docs index cannot tell you why a daemon is crash-looping. Attach both to a project that does real engineering work.

docs-mcp is the open-source replacement for Context7 / Ref.tools — same job, but the index lives on your infrastructure and can include private repos.

No infrastructure: duckduckgo

The only one with nothing behind it. It scrapes DuckDuckGo's HTML endpoint directly, so there is no engine to run and no key to hold.

mcpctl create server websearch --from-template duckduckgo --env DDG_SAFE_SEARCH=OFF
mcpctl get instances | grep websearch     # RUNNING / healthy within a few minutes

Tools: search (query, max_results, region) and fetch_content (url, start_index, max_length) for pulling a result as markdown.

Fine for a person searching occasionally; not for agents. Scraping has no SLA, and agents search in bursts — one question becomes several parallel queries. From a single home or office IP, DuckDuckGo starts answering CAPTCHA and stays that way, and every further request from that IP deepens the block, including for a SearXNG instance sharing the address. DDG_SEARCH_BACKEND=curl survives some bot checks but does not change that. Use searxng for agent search.

The readiness probe is fetch_content on https://example.com every 300s, not search. A search probe is a DuckDuckGo scrape every interval — 1,440 a day at the 60s default — from the same IP your agents depend on. The trade: healthy proves the process and its outbound fetch work, not that DuckDuckGo is still answering you.

Better results: searxng

SearXNG aggregates ~25 engines and gives you real filtering, which is what keeps search results from flooding a context window:

• min_score (0.0–1.0) — relevance floor. The most useful knob; start at 0.3.
• time_range — day / week / month / year. Essential for "did this break in the last release" questions.
• language, safesearch, pageno.

web_url_read then takes section, paragraphRange, startChar/maxLength and readHeadings, so you pull one section of a long page rather than all of it. Bounded PDF extraction is included, which covers most vendor documentation.

The cost is that SearXNG is a service you have to run. It is plain infrastructure, not an MCP server, so mcpctl has nothing to manage it with — it belongs in your cluster or in the compose stack:

cd stack
cp .env.example .env          # set SEARXNG_SECRET
docker compose --profile websearch up -d

Then point the MCP server at it:

mcpctl create secret searxng-conf --data SEARXNG_URL=http://mcpctl-searxng:8080
mcpctl create server searxng --from-template searxng --env-from-secret searxng-conf

The one SearXNG gotcha

SearXNG ships with search.formats: [html]. Every format=json request against a stock instance returns 403, and mcp-searxng comes back empty with no useful error. stack/searxng/settings.yml exists only to add json to that list — it is the one setting with no environment-variable override.

This is also why pointing the template at a public SearXNG instance usually fails: nearly all of them leave the JSON API off.

The compose healthcheck probes format=json specifically, so a misconfigured instance shows up as unhealthy rather than as silently empty search results.

Probe and engines

The template's readiness probe is searxng_instance_info, which reads the instance's /config: a pass proves SearXNG answers without sending a query to any engine. Do not probe with searxng_web_search — every interval becomes a real search fanned out to every engine, which bills API-key engines and feeds the IP blocks on scraping ones.

Those blocks are the long-term problem. Scraping engines (duckduckgo, google, startpage, qwant) CAPTCHA a single IP under agent traffic. SearXNG also ships API-key engines — braveapi, kagi — that are not subject to IP reputation; enable at least one if agents are the main caller, and cap its spend in the vendor's dashboard.

Reading pages: firecrawl

firecrawl is the reader to pair with searxng: give it a URL, get the page's main content as markdown. Firecrawl renders pages in a real browser, so JavaScript-heavy pages work where a plain fetch comes back empty.

mcpctl create server firecrawl --from-template firecrawl \
  --env FIRECRAWL_API_URL=http://firecrawl-api:3002

Point it at a self-hosted Firecrawl (no key needed) or leave FIRECRAWL_API_URL unset and set FIRECRAWL_API_KEY for Firecrawl's cloud. The tool agents want is firecrawl_scrape with formats: ["markdown"] and onlyMainContent: true.

• Egress port. Self-hosted Firecrawl listens on :3002, outside the default 80/443 server egress, so declare it or every call times out.
• Search stays off unless the Firecrawl instance has SEARXNG_ENDPOINT set, so firecrawl_search cannot quietly become a second scraper. Use the searxng template for search.
• Probe: firecrawl_scrape of https://example.com every 300s. It is a real outbound fetch, hence the longer interval.

Documentation: docs-mcp

mcpctl create server docs --from-template docs-mcp

Tools: scrape_docs (index a site, GitHub repo, npm or PyPI package, or local files), search_docs (query, optionally pinned to a version), and fetch_url.

Index the things you actually run — Pulumi, the Kubernetes API, Grafana, the Terraform provider docs — rather than everything.

Persistence

The scraped index is a SQLite file (better-sqlite3 + sqlite-vec) under /data. There is no external-database mode — no DATABASE_URL, no pgvector — so a Postgres cluster cannot help here. The template declares a volume instead:

volumes:
  - name: data
    mountPath: /data
    sizeGb: 20

The backing PVC is named after the server (mcpctl-docs-data), not the instance, so editing the server or restarting the pod re-attaches to the same index rather than starting empty. See "Persistent volumes" in the README.

The readiness probe is list_libraries: no arguments, read-only, and it reads the SQLite store, so a pass proves the volume is mounted and readable rather than merely that the process started. It answers "No libraries indexed yet." on a fresh instance instead of erroring, which is why search_docs (which needs a library that only exists after a scrape) cannot serve as the probe.

Embeddings

Keyword search works out of the box. Semantic search is noticeably better and needs an embedding model. To keep it local, point the server at Ollama:

ollama pull nomic-embed-text
mcpctl create server docs --from-template docs-mcp \
  --env DOCS_MCP_EMBEDDING_MODEL=openai:nomic-embed-text \
  --env OPENAI_API_BASE=http://ollama:11434/v1 \
  --env OPENAI_API_KEY=ollama

Ollama's OpenAI-compatible endpoint is why the provider prefix is openai: — the key is a placeholder and never leaves the network.

Changing the embedding model invalidates the index: embeddings from different models are not comparable, so everything has to be re-scraped.

Alternatives considered

• Tavily / Brave / Exa / Perplexity — better formatted results, all require an API key and send every query to a vendor.
• Context7 — the popular docs MCP, but a cloud index of public-docs snippets only. docs-mcp covers the same ground locally and takes private sources.
• agent-search — bundles SearXNG and a 10-strategy extraction cascade in one deploy. Attractive, but at ~67 stars it is too young to build the stack on. Worth revisiting.
• Writing our own — the hard parts are extraction heuristics and the embedding pipeline, and all three projects above already solved them under MIT.