fix(templates): search templates stop probing with searches; add firecrawl

Readiness probes ran real searches every interval: searxng's
searxng_web_search probe fanned a query out to every engine 1,440 times a
day, billing API-key engines (braveapi, kagi), and duckduckgo's `search`
probe scraped DuckDuckGo from the same IP agents search from.

- searxng: probe searxng_instance_info, which reads /config only
- duckduckgo: probe fetch_content on example.com every 300s
- firecrawl (new): firecrawl-mcp against a self-hosted or cloud
  Firecrawl, for reading pages as main-content markdown; probe
  firecrawl_scrape on example.com every 300s

Every probe still names a readiness tool, as templates.test.ts requires;
each was run against the real package before being written down.
docs/web-search.md now pairs searxng (search) with firecrawl (reading)
and says why duckduckgo does not hold up under agent traffic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd

README.md

@@ -869,17 +869,21 @@ Notes:
 
 ### Web search and docs lookup
 
-The `duckduckgo`, `searxng` and `docs-mcp` templates give an agent web search and
-version-pinned library documentation. All are self-hosted and none needs an API key.
-`duckduckgo` needs no backing service at all:
+The `searxng`, `firecrawl`, `duckduckgo` and `docs-mcp` templates give an agent web
+search, page reading and version-pinned library documentation. All can run
+self-hosted without an API key. For agents, pair `searxng` (search) with `firecrawl`
+(page reading); `duckduckgo` needs no backing service but scrapes from your IP and
+gets CAPTCHA-blocked under agent traffic:
 
 ```bash
-mcpctl create server websearch --from-template duckduckgo
+mcpctl create server searxng --from-template searxng --env SEARXNG_URL=http://searxng:8080
+mcpctl create server firecrawl --from-template firecrawl --env FIRECRAWL_API_URL=http://firecrawl-api:3002
 ```
 
 See [docs/web-search.md](docs/web-search.md) for the SearXNG engine setup (including
-the `json` format setting, without which every search silently returns nothing) and
-the `docs-mcp` index-persistence caveat.
+the `json` format setting, without which every search silently returns nothing), why
+none of the search templates probes with a search, and the `docs-mcp`
+index-persistence caveat.
 
 ## Gated Sessions
 

docs/web-search.md

@@ -1,14 +1,19 @@
 # Web search and documentation lookup
 
-Three templates. All self-hosted, none needs an API key or a vendor account, and
-all three are deployed by mcpctl like any other server.
+Four templates. All can run self-hosted without an API key or a vendor account,
+and all are deployed by mcpctl like any other server.
 
 | Template | Package / image | Needs |
 |---|---|---|
-| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing |
 | `searxng` | `mcp-searxng` (node) | a SearXNG instance |
+| `firecrawl` | `firecrawl-mcp` (node) | a Firecrawl instance |
+| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing |
 | `docs-mcp` | `ghcr.io/arabold/docs-mcp-server` | nothing |
 
+For agent use, pair `searxng` (finding pages) with `firecrawl` (reading them).
+`duckduckgo` does both with no infrastructure, but it scrapes from your IP and
+does not hold up under agent traffic — see below.
+
 Search and docs are different jobs, not competing options. A search engine will
 hand you a 2023 blog post with a stale method signature; a docs index cannot
 tell you why a daemon is crash-looping. Attach both to a project that does real
@@ -17,23 +22,31 @@ engineering work.
 `docs-mcp` is the open-source replacement for Context7 / Ref.tools — same job,
 but the index lives on your infrastructure and can include private repos.
 
-## Start here: `duckduckgo`
+## No infrastructure: `duckduckgo`
 
-The only one with no infrastructure behind it. It scrapes DuckDuckGo's HTML
-endpoint directly, so there is no engine to run and no key to hold.
+The only one with nothing behind it. It scrapes DuckDuckGo's HTML endpoint
+directly, so there is no engine to run and no key to hold.
 
 ```bash
 mcpctl create server websearch --from-template duckduckgo --env DDG_SAFE_SEARCH=OFF
-mcpctl get instances | grep websearch     # RUNNING / healthy within ~a minute
+mcpctl get instances | grep websearch     # RUNNING / healthy within a few minutes
 ```
 
 Tools: `search` (`query`, `max_results`, `region`) and `fetch_content` (`url`,
 `start_index`, `max_length`) for pulling a result as markdown.
 
-The tradeoff is honest: scraping has no SLA. DuckDuckGo can change its markup or
-rate-limit you, and the server caps itself at 30 searches/min. Set
-`DDG_SEARCH_BACKEND=curl` if bot checks start biting. When it becomes a problem,
-move to `searxng`.
+Fine for a person searching occasionally; not for agents. Scraping has no SLA,
+and agents search in bursts — one question becomes several parallel queries.
+From a single home or office IP, DuckDuckGo starts answering CAPTCHA and stays
+that way, and every further request from that IP deepens the block, including
+for a SearXNG instance sharing the address. `DDG_SEARCH_BACKEND=curl` survives
+some bot checks but does not change that. Use `searxng` for agent search.
+
+The readiness probe is `fetch_content` on `https://example.com` every 300s, not
+`search`. A search probe is a DuckDuckGo scrape every interval — 1,440 a day at
+the 60s default — from the same IP your agents depend on. The trade: `healthy`
+proves the process and its outbound fetch work, not that DuckDuckGo is still
+answering you.
 
 ## Better results: `searxng`
 
@@ -79,6 +92,43 @@ fails: nearly all of them leave the JSON API off.
 The compose healthcheck probes `format=json` specifically, so a misconfigured
 instance shows up as unhealthy rather than as silently empty search results.
 
+### Probe and engines
+
+The template's readiness probe is `searxng_instance_info`, which reads the
+instance's `/config`: a pass proves SearXNG answers without sending a query to
+any engine. Do not probe with `searxng_web_search` — every interval becomes a
+real search fanned out to every engine, which bills API-key engines and feeds
+the IP blocks on scraping ones.
+
+Those blocks are the long-term problem. Scraping engines (duckduckgo, google,
+startpage, qwant) CAPTCHA a single IP under agent traffic. SearXNG also ships
+API-key engines — `braveapi`, `kagi` — that are not subject to IP reputation;
+enable at least one if agents are the main caller, and cap its spend in the
+vendor's dashboard.
+
+## Reading pages: `firecrawl`
+
+`firecrawl` is the reader to pair with `searxng`: give it a URL, get the page's
+main content as markdown. Firecrawl renders pages in a real browser, so
+JavaScript-heavy pages work where a plain fetch comes back empty.
+
+```bash
+mcpctl create server firecrawl --from-template firecrawl \
+  --env FIRECRAWL_API_URL=http://firecrawl-api:3002
+```
+
+Point it at a self-hosted Firecrawl (no key needed) or leave `FIRECRAWL_API_URL`
+unset and set `FIRECRAWL_API_KEY` for Firecrawl's cloud. The tool agents want is
+`firecrawl_scrape` with `formats: ["markdown"]` and `onlyMainContent: true`.
+
+- **Egress port.** Self-hosted Firecrawl listens on `:3002`, outside the default
+  80/443 server egress, so declare it or every call times out.
+- **Search stays off** unless the Firecrawl instance has `SEARXNG_ENDPOINT`
+  set, so `firecrawl_search` cannot quietly become a second scraper. Use the
+  `searxng` template for search.
+- **Probe:** `firecrawl_scrape` of `https://example.com` every 300s. It is a
+  real outbound fetch, hence the longer interval.
+
 ## Documentation: `docs-mcp`
 
 ```bash

templates/duckduckgo.yaml

@@ -5,11 +5,17 @@ packageName: "duckduckgo-mcp-server"
 runtime: python
 transport: STDIO
 repositoryUrl: https://github.com/nickclyde/duckduckgo-mcp-server
+# Readiness fetches a tiny static page instead of searching. A `search` probe is
+# a DuckDuckGo scrape every interval (1,440/day at 60s) from the same IP agents
+# search from, which is how that IP ends up answering CAPTCHA. This proves the
+# process and its outbound fetch path work, not that DuckDuckGo is answering.
 healthCheck:
-  tool: search
+  tool: fetch_content
   arguments:
-    query: "mcp"
-    max_results: 1
+    url: "https://example.com"
+    max_length: 500
+  intervalSeconds: 300
+  timeoutSeconds: 20
 env:
   - name: DDG_SAFE_SEARCH
     description: Result filtering — STRICT, MODERATE or OFF

templates/firecrawl.yaml

@@ -0,0 +1,28 @@
+name: firecrawl
+version: "1.0.0"
+description: Firecrawl MCP server for reading web pages as main-content markdown, JS-rendered pages included
+packageName: "firecrawl-mcp"
+runtime: node
+transport: STDIO
+repositoryUrl: https://github.com/firecrawl/firecrawl-mcp-server
+# Readiness scrapes a tiny static page: a pass proves the Firecrawl API answers
+# and can fetch the web, without a search. Firecrawl's own search needs a
+# SearXNG endpoint on self-hosted instances; use the searxng template for
+# search. 300s because each probe is a real outbound fetch.
+healthCheck:
+  tool: firecrawl_scrape
+  arguments:
+    url: "https://example.com"
+    formats: ["markdown"]
+    onlyMainContent: true
+  intervalSeconds: 300
+  timeoutSeconds: 60
+env:
+  - name: FIRECRAWL_API_URL
+    description: >-
+      Self-hosted Firecrawl API base URL (e.g. http://firecrawl-api:3002). Leave
+      unset only when using Firecrawl's cloud with FIRECRAWL_API_KEY.
+    required: false
+  - name: FIRECRAWL_API_KEY
+    description: API key — required for Firecrawl's cloud, optional for a self-hosted instance
+    required: false

templates/searxng.yaml

@@ -4,10 +4,12 @@ description: SearXNG MCP server for private web search and URL-to-markdown readi
 packageName: "mcp-searxng"
 transport: STDIO
 repositoryUrl: https://github.com/ihor-sokoliuk/mcp-searxng
+# Readiness reads SearXNG's /config, which proves the instance answers without
+# fanning out to its engines. A searxng_web_search probe is a real search every
+# interval (1,440/day at 60s): it burns API-key engines (braveapi, kagi) and
+# feeds the IP blocks scraping engines put on the instance.
 healthCheck:
-  tool: searxng_web_search
-  arguments:
-    query: "mcp"
+  tool: searxng_instance_info
 env:
   - name: SEARXNG_URL
     description: >-