Merge origin/main into feat/agentic-teams

Conflict in src/mcpd/src/main.ts: both sides touched the import block.
Kept this branch's mapUrlToPermission plus main's bootstrapUngatedGlobal;
dropped main's SET_OWN_PASSWORD_OPERATION import — this branch moved that
usage into middleware/url-permission.ts, which imports it itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/reliability.md

@@ -137,6 +137,42 @@ Also check the *dialect*: UniFi's `controller_type` must be `classic` for a
 self-hosted controller (login `/api/login`, no `/proxy/network` prefix).
 `unifi_os` sends every request to a path that 404s.
 
+### A server that restarts instead of erroring is out of memory
+
+Server pods get **512 MiB** by default (`DEFAULT_MEMORY_LIMIT`). That is ample
+for a server that proxies an API and nowhere near enough for one that drives a
+browser or holds an index in memory.
+
+An OOMKill is the quietest failure in the fleet, because **nothing reports an
+error**. The kernel kills the container, Kubernetes restarts it, the readiness
+probe passes again, and `mcpctl get instances` reads `healthy`. Whatever the
+server was doing is simply gone — for `docs`, six scrape jobs whose queue lived
+in memory, leaving an index with 17 pages in it and no failed job to look at.
+
+The tells, in order of how fast they answer the question:
+
+```bash
+kubectl -n mcpctl-servers get pods | grep <server>     # RESTARTS climbing
+kubectl -n mcpctl-servers get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
+```
+
+`OOMKilled` there is conclusive. `mcpctl logs` will not show it: the process
+never got to say anything.
+
+Raise the ceiling per server rather than for the fleet — most servers do not
+need it, and a bigger default wastes real memory on every node:
+
+```bash
+mcpctl create server docs --memory-limit-mb 2048 --force
+```
+
+Declared in MiB, stored on the server, converted to bytes in the container
+spec. Null keeps the 512 MiB default, so existing servers are unchanged. Sizing
+rule of thumb: measure idle first (`/sys/fs/cgroup/memory.current` inside the
+pod), then leave headroom for the peak — `docs` idles at ~228 MiB and crosses
+512 MiB within seconds of a scrape, because each page render is a Chromium
+process.
+
 ## LLM-*essential* operations — failover chain
 
 Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned

docs/ungated-global.md

@@ -0,0 +1,56 @@
+# ungated-global: servers under every project, before the gate
+
+`ungated-global` is a system project with one special property: **every server
+attached to it is exposed under every project's MCP endpoint, in front of the
+gate** — its tools are listed and callable before `begin_session`, and the
+project's `prompt` is prepended to every project's instructions.
+
+Fresh mcpd instances seed it **once** with the default `calcmcp` server
+(scientific calculator, patched fork of
+[thinkitpossible/CalcMCP](https://github.com/thinkitpossible/CalcMCP) at
+<https://mysources.co.uk/michal/CalcMCP>) and a prompt telling models to never
+do math themselves and always use calcmcp.
+
+## Semantics
+
+- **Seed-once, not resurrect**: the first successful mcpd startup writes
+  `ungatedGlobalSeeded: true` into the `mcpctl-system-settings` secret. After
+  that, nothing is ever re-created — deletions are durable:
+  - `mcpctl delete server calcmcp -p ungated-global` detaches the default
+    server (delete the server resource itself to also stop its instance);
+  - deleting the whole `ungated-global` project turns the feature off.
+  To re-create the default by hand there is a `calcmcp` template:
+  `mcpctl create server calcmcp --from-template calcmcp`, then attach it to
+  a (re-created) `ungated-global` project.
+- **Merging** happens in mcpd:
+  - `GET /api/v1/projects/:id/servers` appends ungated-global's servers,
+    flagged `ungatedGlobal: true`. A server attached to the project directly
+    wins (no duplicate, no flag).
+  - `GET /api/v1/projects/:id/instructions` prepends ungated-global's
+    `prompt` and appends its servers to the server list.
+- **Gate exemption** happens in mcplocal: discovery passes the flagged names
+  into the gate plugin (`ungatedServers`); while a session is gated,
+  `tools/list` returns `begin_session` **plus** the exempt servers' tools, and
+  calls to them pass through without triggering the gate intercept or
+  ungating the session. Initialize instructions list them under
+  "Available immediately (no begin_session required)".
+
+## Editing the prompt
+
+The pre-gate directive lives on the project itself:
+
+```bash
+mcpctl patch project ungated-global --prompt "…"
+```
+
+Caveat: clients that ignore MCP initialize instructions (e.g. LibreChat uses
+config-literal serverInstructions) never see project prompts — for those, the
+carrier is the tool/server descriptions themselves, which is why calcmcp's
+server description also says "use it for every mathematical calculation".
+
+## Security note
+
+Anything attached to ungated-global is reachable from **every** project by
+**every** token that can reach any project endpoint. Only attach servers that
+are safe in every context: no secrets, no egress, no side effects. calcmcp is
+pure compute (no env, no network).

docs/web-search.md

@@ -1,14 +1,19 @@
 # Web search and documentation lookup
 
-Three templates. All self-hosted, none needs an API key or a vendor account, and
-all three are deployed by mcpctl like any other server.
+Four templates. All can run self-hosted without an API key or a vendor account,
+and all are deployed by mcpctl like any other server.
 
 | Template | Package / image | Needs |
 |---|---|---|
-| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing |
 | `searxng` | `mcp-searxng` (node) | a SearXNG instance |
+| `firecrawl` | `firecrawl-mcp` (node) | a Firecrawl instance |
+| `duckduckgo` | `duckduckgo-mcp-server` (python) | nothing |
 | `docs-mcp` | `ghcr.io/arabold/docs-mcp-server` | nothing |
 
+For agent use, pair `searxng` (finding pages) with `firecrawl` (reading them).
+`duckduckgo` does both with no infrastructure, but it scrapes from your IP and
+does not hold up under agent traffic — see below.
+
 Search and docs are different jobs, not competing options. A search engine will
 hand you a 2023 blog post with a stale method signature; a docs index cannot
 tell you why a daemon is crash-looping. Attach both to a project that does real
@@ -17,23 +22,31 @@ engineering work.
 `docs-mcp` is the open-source replacement for Context7 / Ref.tools — same job,
 but the index lives on your infrastructure and can include private repos.
 
-## Start here: `duckduckgo`
+## No infrastructure: `duckduckgo`
 
-The only one with no infrastructure behind it. It scrapes DuckDuckGo's HTML
-endpoint directly, so there is no engine to run and no key to hold.
+The only one with nothing behind it. It scrapes DuckDuckGo's HTML endpoint
+directly, so there is no engine to run and no key to hold.
 
 ```bash
 mcpctl create server websearch --from-template duckduckgo --env DDG_SAFE_SEARCH=OFF
-mcpctl get instances | grep websearch     # RUNNING / healthy within ~a minute
+mcpctl get instances | grep websearch     # RUNNING / healthy within a few minutes
 ```
 
 Tools: `search` (`query`, `max_results`, `region`) and `fetch_content` (`url`,
 `start_index`, `max_length`) for pulling a result as markdown.
 
-The tradeoff is honest: scraping has no SLA. DuckDuckGo can change its markup or
-rate-limit you, and the server caps itself at 30 searches/min. Set
-`DDG_SEARCH_BACKEND=curl` if bot checks start biting. When it becomes a problem,
-move to `searxng`.
+Fine for a person searching occasionally; not for agents. Scraping has no SLA,
+and agents search in bursts — one question becomes several parallel queries.
+From a single home or office IP, DuckDuckGo starts answering CAPTCHA and stays
+that way, and every further request from that IP deepens the block, including
+for a SearXNG instance sharing the address. `DDG_SEARCH_BACKEND=curl` survives
+some bot checks but does not change that. Use `searxng` for agent search.
+
+The readiness probe is `fetch_content` on `https://example.com` every 300s, not
+`search`. A search probe is a DuckDuckGo scrape every interval — 1,440 a day at
+the 60s default — from the same IP your agents depend on. The trade: `healthy`
+proves the process and its outbound fetch work, not that DuckDuckGo is still
+answering you.
 
 ## Better results: `searxng`
 
@@ -79,6 +92,43 @@ fails: nearly all of them leave the JSON API off.
 The compose healthcheck probes `format=json` specifically, so a misconfigured
 instance shows up as unhealthy rather than as silently empty search results.
 
+### Probe and engines
+
+The template's readiness probe is `searxng_instance_info`, which reads the
+instance's `/config`: a pass proves SearXNG answers without sending a query to
+any engine. Do not probe with `searxng_web_search` — every interval becomes a
+real search fanned out to every engine, which bills API-key engines and feeds
+the IP blocks on scraping ones.
+
+Those blocks are the long-term problem. Scraping engines (duckduckgo, google,
+startpage, qwant) CAPTCHA a single IP under agent traffic. SearXNG also ships
+API-key engines — `braveapi`, `kagi` — that are not subject to IP reputation;
+enable at least one if agents are the main caller, and cap its spend in the
+vendor's dashboard.
+
+## Reading pages: `firecrawl`
+
+`firecrawl` is the reader to pair with `searxng`: give it a URL, get the page's
+main content as markdown. Firecrawl renders pages in a real browser, so
+JavaScript-heavy pages work where a plain fetch comes back empty.
+
+```bash
+mcpctl create server firecrawl --from-template firecrawl \
+  --env FIRECRAWL_API_URL=http://firecrawl-api:3002
+```
+
+Point it at a self-hosted Firecrawl (no key needed) or leave `FIRECRAWL_API_URL`
+unset and set `FIRECRAWL_API_KEY` for Firecrawl's cloud. The tool agents want is
+`firecrawl_scrape` with `formats: ["markdown"]` and `onlyMainContent: true`.
+
+- **Egress port.** Self-hosted Firecrawl listens on `:3002`, outside the default
+  80/443 server egress, so declare it or every call times out.
+- **Search stays off** unless the Firecrawl instance has `SEARXNG_ENDPOINT`
+  set, so `firecrawl_search` cannot quietly become a second scraper. Use the
+  `searxng` template for search.
+- **Probe:** `firecrawl_scrape` of `https://example.com` every 300s. It is a
+  real outbound fetch, hence the longer interval.
+
 ## Documentation: `docs-mcp`
 
 ```bash