fix(templates): search templates stop probing with searches; add firecrawl #131

michal · 2026-09-15T21:16:18Z

michal commented

2026-09-15 21:16:18 +00:00

Readiness probes on the search templates ran real searches every interval.

searxng: searxng_web_search probe → searxng_instance_info (reads /config, no engine fan-out). At the 60s default the old probe was 1,440 searches/day, billing API-key engines like braveapi and kagi.
duckduckgo: search probe → fetch_content on example.com every 300s, so the probe no longer scrapes DuckDuckGo from the IP agents search from.
firecrawl (new): firecrawl-mcp for reading pages as main-content markdown, self-hosted (FIRECRAWL_API_URL) or cloud (FIRECRAWL_API_KEY). Probe: firecrawl_scrape on example.com every 300s.
docs/web-search.md + README: pair searxng (search) with firecrawl (reading); explain why duckduckgo does not hold up under agent traffic and why no template probes with a search.

Verified: each probe run against the real package (firecrawl ~2.5s, duckduckgo ~0.2s); searxng probe live and healthy. vitest run tests/templates.test.ts: 72 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd

Readiness probes on the search templates ran real searches every interval. - **searxng**: `searxng_web_search` probe → `searxng_instance_info` (reads `/config`, no engine fan-out). At the 60s default the old probe was 1,440 searches/day, billing API-key engines like braveapi and kagi. - **duckduckgo**: `search` probe → `fetch_content` on example.com every 300s, so the probe no longer scrapes DuckDuckGo from the IP agents search from. - **firecrawl** (new): `firecrawl-mcp` for reading pages as main-content markdown, self-hosted (`FIRECRAWL_API_URL`) or cloud (`FIRECRAWL_API_KEY`). Probe: `firecrawl_scrape` on example.com every 300s. - docs/web-search.md + README: pair searxng (search) with firecrawl (reading); explain why duckduckgo does not hold up under agent traffic and why no template probes with a search. Verified: each probe run against the real package (firecrawl ~2.5s, duckduckgo ~0.2s); searxng probe live and healthy. `vitest run tests/templates.test.ts`: 72 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd

michal added 1 commit 2026-09-15 21:16:19 +00:00

fix(templates): search templates stop probing with searches; add firecrawl

CI/CD / typecheck (pull_request) Successful in 1m17s

Details

CI/CD / test (pull_request) Successful in 1m24s

Details

CI/CD / lint (pull_request) Successful in 2m56s

Details

CI/CD / smoke (pull_request) Failing after 1m54s

Details

CI/CD / build (pull_request) Successful in 4m19s

Details

CI/CD / publish (pull_request) Has been skipped

Details

61e52403a3

Readiness probes ran real searches every interval: searxng's
searxng_web_search probe fanned a query out to every engine 1,440 times a
day, billing API-key engines (braveapi, kagi), and duckduckgo's `search`
probe scraped DuckDuckGo from the same IP agents search from.

- searxng: probe searxng_instance_info, which reads /config only
- duckduckgo: probe fetch_content on example.com every 300s
- firecrawl (new): firecrawl-mcp against a self-hosted or cloud
  Firecrawl, for reading pages as main-content markdown; probe
  firecrawl_scrape on example.com every 300s

Every probe still names a readiness tool, as templates.test.ts requires;
each was run against the real package before being written down.
docs/web-search.md now pairs searxng (search) with firecrawl (reading)
and says why duckduckgo does not hold up under agent traffic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd

michal merged commit 3e534854f4 into main

2026-09-15 21:18:29 +00:00

michal referenced this issue from a commit

2026-09-15 21:18:30 +00:00

Merge pull request 'fix(templates): search templates stop probing with searches; add firecrawl' (#131) from fix/search-templates into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: michal/mcpctl#131