fix(templates): search templates stop probing with searches; add firecrawl
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m17s
CI/CD / test (pull_request) Successful in 1m24s
CI/CD / lint (pull_request) Successful in 2m56s
CI/CD / smoke (pull_request) Failing after 1m54s
CI/CD / build (pull_request) Successful in 4m19s
CI/CD / publish (pull_request) Has been skipped
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m17s
CI/CD / test (pull_request) Successful in 1m24s
CI/CD / lint (pull_request) Successful in 2m56s
CI/CD / smoke (pull_request) Failing after 1m54s
CI/CD / build (pull_request) Successful in 4m19s
CI/CD / publish (pull_request) Has been skipped
Readiness probes ran real searches every interval: searxng's searxng_web_search probe fanned a query out to every engine 1,440 times a day, billing API-key engines (braveapi, kagi), and duckduckgo's `search` probe scraped DuckDuckGo from the same IP agents search from. - searxng: probe searxng_instance_info, which reads /config only - duckduckgo: probe fetch_content on example.com every 300s - firecrawl (new): firecrawl-mcp against a self-hosted or cloud Firecrawl, for reading pages as main-content markdown; probe firecrawl_scrape on example.com every 300s Every probe still names a readiness tool, as templates.test.ts requires; each was run against the real package before being written down. docs/web-search.md now pairs searxng (search) with firecrawl (reading) and says why duckduckgo does not hold up under agent traffic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd
This commit is contained in:
16
README.md
16
README.md
@@ -869,17 +869,21 @@ Notes:
|
||||
|
||||
### Web search and docs lookup
|
||||
|
||||
The `duckduckgo`, `searxng` and `docs-mcp` templates give an agent web search and
|
||||
version-pinned library documentation. All are self-hosted and none needs an API key.
|
||||
`duckduckgo` needs no backing service at all:
|
||||
The `searxng`, `firecrawl`, `duckduckgo` and `docs-mcp` templates give an agent web
|
||||
search, page reading and version-pinned library documentation. All can run
|
||||
self-hosted without an API key. For agents, pair `searxng` (search) with `firecrawl`
|
||||
(page reading); `duckduckgo` needs no backing service but scrapes from your IP and
|
||||
gets CAPTCHA-blocked under agent traffic:
|
||||
|
||||
```bash
|
||||
mcpctl create server websearch --from-template duckduckgo
|
||||
mcpctl create server searxng --from-template searxng --env SEARXNG_URL=http://searxng:8080
|
||||
mcpctl create server firecrawl --from-template firecrawl --env FIRECRAWL_API_URL=http://firecrawl-api:3002
|
||||
```
|
||||
|
||||
See [docs/web-search.md](docs/web-search.md) for the SearXNG engine setup (including
|
||||
the `json` format setting, without which every search silently returns nothing) and
|
||||
the `docs-mcp` index-persistence caveat.
|
||||
the `json` format setting, without which every search silently returns nothing), why
|
||||
none of the search templates probes with a search, and the `docs-mcp`
|
||||
index-persistence caveat.
|
||||
|
||||
## Gated Sessions
|
||||
|
||||
|
||||
Reference in New Issue
Block a user