fix(templates): search templates stop probing with searches; add firecrawl

Readiness probes ran real searches every interval: searxng's
searxng_web_search probe fanned a query out to every engine 1,440 times a
day, billing API-key engines (braveapi, kagi), and duckduckgo's `search`
probe scraped DuckDuckGo from the same IP agents search from.

- searxng: probe searxng_instance_info, which reads /config only
- duckduckgo: probe fetch_content on example.com every 300s
- firecrawl (new): firecrawl-mcp against a self-hosted or cloud
  Firecrawl, for reading pages as main-content markdown; probe
  firecrawl_scrape on example.com every 300s

Every probe still names a readiness tool, as templates.test.ts requires;
each was run against the real package before being written down.
docs/web-search.md now pairs searxng (search) with firecrawl (reading)
and says why duckduckgo does not hold up under agent traffic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd

docs/web-search.md

@@ -1,14 +1,19 @@
 # Web search and documentation lookup
 
-Three templates. All self-hosted, none needs an API key or a vendor account, and
-all three are deployed by mcpctl like any other server.
+Four templates. All can run self-hosted without an API key or a vendor account,
+and all are deployed by mcpctl like any other server.
 
 | Template | Package / image | Needs |
 |---|---|---|
-| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing |
 | `searxng` | `mcp-searxng` (node) | a SearXNG instance |
+| `firecrawl` | `firecrawl-mcp` (node) | a Firecrawl instance |
+| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing |
 | `docs-mcp` | `ghcr.io/arabold/docs-mcp-server` | nothing |
 
+For agent use, pair `searxng` (finding pages) with `firecrawl` (reading them).
+`duckduckgo` does both with no infrastructure, but it scrapes from your IP and
+does not hold up under agent traffic — see below.
+
 Search and docs are different jobs, not competing options. A search engine will
 hand you a 2023 blog post with a stale method signature; a docs index cannot
 tell you why a daemon is crash-looping. Attach both to a project that does real
@@ -17,23 +22,31 @@ engineering work.
 `docs-mcp` is the open-source replacement for Context7 / Ref.tools — same job,
 but the index lives on your infrastructure and can include private repos.
 
-## Start here: `duckduckgo`
+## No infrastructure: `duckduckgo`
 
-The only one with no infrastructure behind it. It scrapes DuckDuckGo's HTML
-endpoint directly, so there is no engine to run and no key to hold.
+The only one with nothing behind it. It scrapes DuckDuckGo's HTML endpoint
+directly, so there is no engine to run and no key to hold.
 
 ```bash
 mcpctl create server websearch --from-template duckduckgo --env DDG_SAFE_SEARCH=OFF
-mcpctl get instances | grep websearch     # RUNNING / healthy within ~a minute
+mcpctl get instances | grep websearch     # RUNNING / healthy within a few minutes
 ```
 
 Tools: `search` (`query`, `max_results`, `region`) and `fetch_content` (`url`,
 `start_index`, `max_length`) for pulling a result as markdown.
 
-The tradeoff is honest: scraping has no SLA. DuckDuckGo can change its markup or
-rate-limit you, and the server caps itself at 30 searches/min. Set
-`DDG_SEARCH_BACKEND=curl` if bot checks start biting. When it becomes a problem,
-move to `searxng`.
+Fine for a person searching occasionally; not for agents. Scraping has no SLA,
+and agents search in bursts — one question becomes several parallel queries.
+From a single home or office IP, DuckDuckGo starts answering CAPTCHA and stays
+that way, and every further request from that IP deepens the block, including
+for a SearXNG instance sharing the address. `DDG_SEARCH_BACKEND=curl` survives
+some bot checks but does not change that. Use `searxng` for agent search.
+
+The readiness probe is `fetch_content` on `https://example.com` every 300s, not
+`search`. A search probe is a DuckDuckGo scrape every interval — 1,440 a day at
+the 60s default — from the same IP your agents depend on. The trade: `healthy`
+proves the process and its outbound fetch work, not that DuckDuckGo is still
+answering you.
 
 ## Better results: `searxng`
 
@@ -79,6 +92,43 @@ fails: nearly all of them leave the JSON API off.
 The compose healthcheck probes `format=json` specifically, so a misconfigured
 instance shows up as unhealthy rather than as silently empty search results.
 
+### Probe and engines
+
+The template's readiness probe is `searxng_instance_info`, which reads the
+instance's `/config`: a pass proves SearXNG answers without sending a query to
+any engine. Do not probe with `searxng_web_search` — every interval becomes a
+real search fanned out to every engine, which bills API-key engines and feeds
+the IP blocks on scraping ones.
+
+Those blocks are the long-term problem. Scraping engines (duckduckgo, google,
+startpage, qwant) CAPTCHA a single IP under agent traffic. SearXNG also ships
+API-key engines — `braveapi`, `kagi` — that are not subject to IP reputation;
+enable at least one if agents are the main caller, and cap its spend in the
+vendor's dashboard.
+
+## Reading pages: `firecrawl`
+
+`firecrawl` is the reader to pair with `searxng`: give it a URL, get the page's
+main content as markdown. Firecrawl renders pages in a real browser, so
+JavaScript-heavy pages work where a plain fetch comes back empty.
+
+```bash
+mcpctl create server firecrawl --from-template firecrawl \
+  --env FIRECRAWL_API_URL=http://firecrawl-api:3002
+```
+
+Point it at a self-hosted Firecrawl (no key needed) or leave `FIRECRAWL_API_URL`
+unset and set `FIRECRAWL_API_KEY` for Firecrawl's cloud. The tool agents want is
+`firecrawl_scrape` with `formats: ["markdown"]` and `onlyMainContent: true`.
+
+- **Egress port.** Self-hosted Firecrawl listens on `:3002`, outside the default
+  80/443 server egress, so declare it or every call times out.
+- **Search stays off** unless the Firecrawl instance has `SEARXNG_ENDPOINT`
+  set, so `firecrawl_search` cannot quietly become a second scraper. Use the
+  `searxng` template for search.
+- **Probe:** `firecrawl_scrape` of `https://example.com` every 300s. It is a
+  real outbound fetch, hence the longer interval.
+
 ## Documentation: `docs-mcp`
 
 ```bash