diff --git a/README.md b/README.md index 32f110d..9921d08 100644 --- a/README.md +++ b/README.md @@ -869,17 +869,21 @@ Notes: ### Web search and docs lookup -The `duckduckgo`, `searxng` and `docs-mcp` templates give an agent web search and -version-pinned library documentation. All are self-hosted and none needs an API key. -`duckduckgo` needs no backing service at all: +The `searxng`, `firecrawl`, `duckduckgo` and `docs-mcp` templates give an agent web +search, page reading and version-pinned library documentation. All can run +self-hosted without an API key. For agents, pair `searxng` (search) with `firecrawl` +(page reading); `duckduckgo` needs no backing service but scrapes from your IP and +gets CAPTCHA-blocked under agent traffic: ```bash -mcpctl create server websearch --from-template duckduckgo +mcpctl create server searxng --from-template searxng --env SEARXNG_URL=http://searxng:8080 +mcpctl create server firecrawl --from-template firecrawl --env FIRECRAWL_API_URL=http://firecrawl-api:3002 ``` See [docs/web-search.md](docs/web-search.md) for the SearXNG engine setup (including -the `json` format setting, without which every search silently returns nothing) and -the `docs-mcp` index-persistence caveat. +the `json` format setting, without which every search silently returns nothing), why +none of the search templates probes with a search, and the `docs-mcp` +index-persistence caveat. ## Gated Sessions diff --git a/docs/web-search.md b/docs/web-search.md index ff39cf6..438ff47 100644 --- a/docs/web-search.md +++ b/docs/web-search.md @@ -1,14 +1,19 @@ # Web search and documentation lookup -Three templates. All self-hosted, none needs an API key or a vendor account, and -all three are deployed by mcpctl like any other server. +Four templates. All can run self-hosted without an API key or a vendor account, +and all are deployed by mcpctl like any other server. | Template | Package / image | Needs | |---|---|---| -| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing | | `searxng` | `mcp-searxng` (node) | a SearXNG instance | +| `firecrawl` | `firecrawl-mcp` (node) | a Firecrawl instance | +| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing | | `docs-mcp` | `ghcr.io/arabold/docs-mcp-server` | nothing | +For agent use, pair `searxng` (finding pages) with `firecrawl` (reading them). +`duckduckgo` does both with no infrastructure, but it scrapes from your IP and +does not hold up under agent traffic — see below. + Search and docs are different jobs, not competing options. A search engine will hand you a 2023 blog post with a stale method signature; a docs index cannot tell you why a daemon is crash-looping. Attach both to a project that does real @@ -17,23 +22,31 @@ engineering work. `docs-mcp` is the open-source replacement for Context7 / Ref.tools — same job, but the index lives on your infrastructure and can include private repos. -## Start here: `duckduckgo` +## No infrastructure: `duckduckgo` -The only one with no infrastructure behind it. It scrapes DuckDuckGo's HTML -endpoint directly, so there is no engine to run and no key to hold. +The only one with nothing behind it. It scrapes DuckDuckGo's HTML endpoint +directly, so there is no engine to run and no key to hold. ```bash mcpctl create server websearch --from-template duckduckgo --env DDG_SAFE_SEARCH=OFF -mcpctl get instances | grep websearch # RUNNING / healthy within ~a minute +mcpctl get instances | grep websearch # RUNNING / healthy within a few minutes ``` Tools: `search` (`query`, `max_results`, `region`) and `fetch_content` (`url`, `start_index`, `max_length`) for pulling a result as markdown. -The tradeoff is honest: scraping has no SLA. DuckDuckGo can change its markup or -rate-limit you, and the server caps itself at 30 searches/min. Set -`DDG_SEARCH_BACKEND=curl` if bot checks start biting. When it becomes a problem, -move to `searxng`. +Fine for a person searching occasionally; not for agents. Scraping has no SLA, +and agents search in bursts — one question becomes several parallel queries. +From a single home or office IP, DuckDuckGo starts answering CAPTCHA and stays +that way, and every further request from that IP deepens the block, including +for a SearXNG instance sharing the address. `DDG_SEARCH_BACKEND=curl` survives +some bot checks but does not change that. Use `searxng` for agent search. + +The readiness probe is `fetch_content` on `https://example.com` every 300s, not +`search`. A search probe is a DuckDuckGo scrape every interval — 1,440 a day at +the 60s default — from the same IP your agents depend on. The trade: `healthy` +proves the process and its outbound fetch work, not that DuckDuckGo is still +answering you. ## Better results: `searxng` @@ -79,6 +92,43 @@ fails: nearly all of them leave the JSON API off. The compose healthcheck probes `format=json` specifically, so a misconfigured instance shows up as unhealthy rather than as silently empty search results. +### Probe and engines + +The template's readiness probe is `searxng_instance_info`, which reads the +instance's `/config`: a pass proves SearXNG answers without sending a query to +any engine. Do not probe with `searxng_web_search` — every interval becomes a +real search fanned out to every engine, which bills API-key engines and feeds +the IP blocks on scraping ones. + +Those blocks are the long-term problem. Scraping engines (duckduckgo, google, +startpage, qwant) CAPTCHA a single IP under agent traffic. SearXNG also ships +API-key engines — `braveapi`, `kagi` — that are not subject to IP reputation; +enable at least one if agents are the main caller, and cap its spend in the +vendor's dashboard. + +## Reading pages: `firecrawl` + +`firecrawl` is the reader to pair with `searxng`: give it a URL, get the page's +main content as markdown. Firecrawl renders pages in a real browser, so +JavaScript-heavy pages work where a plain fetch comes back empty. + +```bash +mcpctl create server firecrawl --from-template firecrawl \ + --env FIRECRAWL_API_URL=http://firecrawl-api:3002 +``` + +Point it at a self-hosted Firecrawl (no key needed) or leave `FIRECRAWL_API_URL` +unset and set `FIRECRAWL_API_KEY` for Firecrawl's cloud. The tool agents want is +`firecrawl_scrape` with `formats: ["markdown"]` and `onlyMainContent: true`. + +- **Egress port.** Self-hosted Firecrawl listens on `:3002`, outside the default + 80/443 server egress, so declare it or every call times out. +- **Search stays off** unless the Firecrawl instance has `SEARXNG_ENDPOINT` + set, so `firecrawl_search` cannot quietly become a second scraper. Use the + `searxng` template for search. +- **Probe:** `firecrawl_scrape` of `https://example.com` every 300s. It is a + real outbound fetch, hence the longer interval. + ## Documentation: `docs-mcp` ```bash diff --git a/templates/duckduckgo.yaml b/templates/duckduckgo.yaml index c21f44c..e45e69c 100644 --- a/templates/duckduckgo.yaml +++ b/templates/duckduckgo.yaml @@ -5,11 +5,17 @@ packageName: "duckduckgo-mcp-server" runtime: python transport: STDIO repositoryUrl: https://github.com/nickclyde/duckduckgo-mcp-server +# Readiness fetches a tiny static page instead of searching. A `search` probe is +# a DuckDuckGo scrape every interval (1,440/day at 60s) from the same IP agents +# search from, which is how that IP ends up answering CAPTCHA. This proves the +# process and its outbound fetch path work, not that DuckDuckGo is answering. healthCheck: - tool: search + tool: fetch_content arguments: - query: "mcp" - max_results: 1 + url: "https://example.com" + max_length: 500 + intervalSeconds: 300 + timeoutSeconds: 20 env: - name: DDG_SAFE_SEARCH description: Result filtering — STRICT, MODERATE or OFF diff --git a/templates/firecrawl.yaml b/templates/firecrawl.yaml new file mode 100644 index 0000000..dff30d6 --- /dev/null +++ b/templates/firecrawl.yaml @@ -0,0 +1,28 @@ +name: firecrawl +version: "1.0.0" +description: Firecrawl MCP server for reading web pages as main-content markdown, JS-rendered pages included +packageName: "firecrawl-mcp" +runtime: node +transport: STDIO +repositoryUrl: https://github.com/firecrawl/firecrawl-mcp-server +# Readiness scrapes a tiny static page: a pass proves the Firecrawl API answers +# and can fetch the web, without a search. Firecrawl's own search needs a +# SearXNG endpoint on self-hosted instances; use the searxng template for +# search. 300s because each probe is a real outbound fetch. +healthCheck: + tool: firecrawl_scrape + arguments: + url: "https://example.com" + formats: ["markdown"] + onlyMainContent: true + intervalSeconds: 300 + timeoutSeconds: 60 +env: + - name: FIRECRAWL_API_URL + description: >- + Self-hosted Firecrawl API base URL (e.g. http://firecrawl-api:3002). Leave + unset only when using Firecrawl's cloud with FIRECRAWL_API_KEY. + required: false + - name: FIRECRAWL_API_KEY + description: API key — required for Firecrawl's cloud, optional for a self-hosted instance + required: false diff --git a/templates/searxng.yaml b/templates/searxng.yaml index cb16d87..5ecef93 100644 --- a/templates/searxng.yaml +++ b/templates/searxng.yaml @@ -4,10 +4,12 @@ description: SearXNG MCP server for private web search and URL-to-markdown readi packageName: "mcp-searxng" transport: STDIO repositoryUrl: https://github.com/ihor-sokoliuk/mcp-searxng +# Readiness reads SearXNG's /config, which proves the instance answers without +# fanning out to its engines. A searxng_web_search probe is a real search every +# interval (1,440/day at 60s): it burns API-key engines (braveapi, kagi) and +# feeds the IP blocks scraping engines put on the instance. healthCheck: - tool: searxng_web_search - arguments: - query: "mcp" + tool: searxng_instance_info env: - name: SEARXNG_URL description: >-