From 1bd5087052c227ee752594fdef72279ea44bb2c1 Mon Sep 17 00:00:00 2001 From: Michal Date: Thu, 9 Apr 2026 23:21:34 +0100 Subject: [PATCH] fix: add prompts/templates to backup + STDIO attach for docker-image servers Two bugs fixed: 1. Backup completeness: JSON backup API now includes prompts and templates. Previously these were silently dropped during backup/restore, causing data loss on migration. 2. STDIO proxy for docker-image servers: servers with dockerImage but no packageName/command (like docmost) now use k8s Attach to connect to the container's PID 1 stdin/stdout instead of exec. This fixes "has no packageName or command" errors. Changes: - backup-service.ts: add BackupPrompt/BackupTemplate types, export them - restore-service.ts: restore prompts (with project FK) and templates - mcp-proxy-service.ts: sendViaPersistentAttach for docker-image STDIO - orchestrator.ts: add attachInteractive to McpOrchestrator interface - kubernetes-orchestrator.ts: implement attachInteractive via k8s Attach - k8s-client-official.ts: expose Attach client Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/project-summary.md | 1048 +++++++++++++++++ src/mcpd/src/main.ts | 4 +- .../src/services/backup/backup-service.ts | 77 +- .../src/services/backup/restore-service.ts | 100 +- .../src/services/k8s/k8s-client-official.ts | 2 + .../services/k8s/kubernetes-orchestrator.ts | 40 + src/mcpd/src/services/mcp-proxy-service.ts | 7 +- src/mcpd/src/services/orchestrator.ts | 3 + src/mcpd/tests/k8s-orchestrator.test.ts | 5 + templates/gitea.yaml | 22 + templates/unifi-network.yaml | 25 + 11 files changed, 1327 insertions(+), 6 deletions(-) create mode 100644 docs/project-summary.md create mode 100644 templates/gitea.yaml create mode 100644 templates/unifi-network.yaml diff --git a/docs/project-summary.md b/docs/project-summary.md new file mode 100644 index 0000000..95552bd --- /dev/null +++ b/docs/project-summary.md @@ -0,0 +1,1048 @@ +# mcpctl — Comprehensive Project Summary + +**kubectl for Model Context Protocol servers.** + +mcpctl is a production-grade management system for MCP servers, providing a Kubernetes-inspired declarative interface for deploying, orchestrating, and observing MCP servers that connect to Claude and other LLM clients. + +--- + +## Table of Contents + +1. [System Architecture](#1-system-architecture) +2. [Component Overview](#2-component-overview) +3. [Resource Model & Design Decisions](#3-resource-model--design-decisions) +4. [CLI Reference](#4-cli-reference) +5. [API Surface (mcpd)](#5-api-surface-mcpd) +6. [Database Schema](#6-database-schema) +7. [Local Proxy (mcplocal)](#7-local-proxy-mcplocal) +8. [ProxyModel Plugin System](#8-proxymodel-plugin-system) +9. [Gated Sessions](#9-gated-sessions) +10. [Content Pipeline & Stages](#10-content-pipeline--stages) +11. [LLM Provider Integration](#11-llm-provider-integration) +12. [Caching](#12-caching) +13. [Authentication & RBAC](#13-authentication--rbac) +14. [Audit Infrastructure & Trust Model](#14-audit-infrastructure--trust-model) +15. [Container Orchestration](#15-container-orchestration) +16. [Deployment & Distribution](#16-deployment--distribution) +17. [Testing Strategy](#17-testing-strategy) +18. [Technology Stack](#18-technology-stack) +19. [Project Structure](#19-project-structure) +20. [Deferred & Future Work](#20-deferred--future-work) + +--- + +## 1. System Architecture + +### Three-Tier Design + +``` +Claude Code / LLM Client + | (STDIO — MCP JSON-RPC protocol) + v +mcplocal (Local Daemon — developer machine) + | (HTTP REST) + v +mcpd (Remote Daemon — server/NAS, e.g. 10.0.0.194) + | (Docker/Podman API) + v +MCP Server Containers (isolated network) +``` + +``` + ┌────────────┐ +┌─────────────────┐ HTTP ┌──────────────┐ │ PostgreSQL │ +│ mcpctl CLI │──────────────>│ mcpd │──>│ │ +│ (Commander.js) │ │ (Fastify 5) │ └────────────┘ +└─────────────────┘ └──────┬───────┘ + │ Docker/Podman API + v + ┌──────────────┐ + │ Containers │ + │ (MCP servers)│ + └──────────────┘ + +┌─────────────────┐ STDIO ┌──────────────┐ STDIO/HTTP ┌────────────┐ +│ Claude / LLM │────────────>│ mcplocal │───────────────>│ MCP Servers│ +│ │ │ (McpRouter) │ │ │ +└─────────────────┘ └──────────────┘ └────────────┘ +``` + +### Key Principle + +- **mcpd owns the database** (PostgreSQL) — the only component that talks to the DB +- **mcplocal is stateless** — config-only, no database, acts as intelligent proxy +- **mcpctl stores only credentials** — `~/.mcpctl/config.json` and `~/.mcpctl/credentials.json` +- **All MCP servers run inside the mcpd container** on the NAS via podman (container-in-container) + +--- + +## 2. Component Overview + +### mcpctl (CLI) +- kubectl-like interface for managing the entire system +- Talks to mcplocal (local daemon) via HTTP REST, or directly to mcpd with `--direct` +- Distributed as RPM/DEB package via Gitea registry +- Built with Commander.js, Ink/React for TUI, Inquirer for prompts + +### mcplocal (Local Daemon) +- Runs on developer machine as a systemd user service +- Exposes MCP protocol via STDIO to Claude +- Exposes HTTP REST API for mcpctl management commands +- Core responsibilities: + - Tool namespacing and routing (`server/tool` format) + - Gated sessions and prompt delivery + - Content pipeline (transformation stages) + - LLM integration for intelligent prompt selection + - Pipeline result caching + - Audit event collection + +### mcpd (Remote Daemon) +- Server-side daemon on NAS/cloud (Fastify 5) +- Manages MCP server containers (Docker/Podman via dockerode) +- PostgreSQL for state, audit logs, access control +- Owns credentials (never exposed to mcplocal) +- REST API for all management operations +- MCP proxy endpoint for direct tool invocation +- Health probe runner for container monitoring +- Git-based backup system + +### @mcpctl/db (Database Layer) +- Prisma ORM with PostgreSQL +- 22 models, 11 migrations +- Template seeding from YAML files at startup + +### @mcpctl/shared (Shared Utilities) +- Constants, types, validation schemas (Zod) +- Secret encryption/decryption utilities +- Zero external dependencies beyond Zod + +--- + +## 3. Resource Model & Design Decisions + +### ADR-001: Kubernetes-Style Resource Model + +| mcpctl Resource | K8s Analogy | Behavior | +|----------------|-------------|----------| +| **Server** | Deployment | Self-contained, complete definition. Contains image, command, transport, env refs, replicas. No external template dependencies at runtime. | +| **Instance** | Pod | Immutable, ephemeral, auto-managed by reconciliation loop. No `create instance` or `edit instance`. Delete triggers re-creation. | +| **Secret** | Secret | Holds sensitive key-value pairs. Servers reference via `env[].valueFrom.secretRef`. | +| **Project** | Namespace | Groups servers, configures ProxyModel and LLM provider. Generates `.mcp.json`. | +| **Prompt** | ConfigMap (sort of) | Instruction text delivered to Claude. Global or project-scoped. Priority-ranked. | +| **Template** | — | Blueprints for server creation. Used at create-time only, not runtime. | +| **RbacDefinition** | ClusterRoleBinding | Named policies with subjects and roleBindings. | + +### ADR-002: Profiles Replaced with Secrets + +The original `McpProfile` resource tried to be secrets, configmaps, and project-server links simultaneously. Environment variables declared in profiles were never actually passed to running containers. + +**Decision:** Replace with dedicated `Secret` resource following Kubernetes conventions: +```yaml +servers: + - name: ha-mcp + env: + - name: HOMEASSISTANT_TOKEN + valueFrom: + secretRef: + name: ha-credentials + key: HOMEASSISTANT_TOKEN +``` + +### ADR-003: Self-Contained Servers with Source Tracking + +Servers store complete definitions (no runtime template dependencies). Optional `source` metadata enables registry-based upgrades via 3-way diff (old snapshot vs current server vs new template). + +**Rationale:** Matches kubectl mental model. `get server X -o yaml > new.yaml && edit && apply` works naturally. Duplication is minimal (~10 lines YAML). + +### ADR-004: ConfigMaps Deferred + +Only Secrets implemented. ConfigMap separation can be added later if needed. Keeps the model simple. + +### ADR-005: Apply-Compatible YAML Round-Trip + +`mcpctl get server ha-mcp -o yaml > s.yaml && mcpctl apply -f s.yaml` must work: +- `get -o yaml/json` strips internal fields (id, createdAt, updatedAt, version, ownerId) +- Output wrapped in resource key: `{ servers: [...] }` +- `describe -o yaml/json` keeps full raw output (for debugging) + +### ADR-006: CLI Design Principles + +1. Everything possible via `apply -f` MUST also be possible via `create` CLI flags +2. Support `-o yaml` and `-o json` like kubectl +3. `describe` shows visually clean sectioned output with tables +4. Name resolution works everywhere (not just IDs) +5. Instances are immutable (like pods) — no create/edit + +--- + +## 4. CLI Reference + +### Global Options +``` +--daemon-url mcplocal daemon URL +--direct bypass mcplocal, connect directly to mcpd +-p, --project Target project +-o, --output table | json | yaml +-v, --version Show version +``` + +### Resource Operations + +| Command | Description | +|---------|-------------| +| `mcpctl get [name]` | List resources or fetch by name/ID. Supports glob patterns (`graf*`). | +| `mcpctl describe ` | Detailed view with sections and tables. | +| `mcpctl create [opts]` | Create resource. Mirrors `apply -f` capabilities. | +| `mcpctl edit ` | Open in `$EDITOR` as YAML, apply on save. | +| `mcpctl patch key=val...` | Patch individual fields without editor. | +| `mcpctl delete ` | Delete resource. | +| `mcpctl apply -f ` | Declarative YAML/JSON application (like `kubectl apply`). Supports `--dry-run`. | + +### Supported Resources + +servers, projects, instances, secrets, templates, users, groups, rbac, prompts, promptrequests, serverattachments (virtual), proxymodels (virtual, from mcplocal), all (project export) + +### Resource Aliases +``` +server/srv → servers project/proj → projects +instance/inst → instances secret/sec → secrets +template/tpl → templates prompt → prompts +user → users group → groups +rbac/rbac-definition → rbac promptrequest/pr → promptrequests +serverattachment/sa → serverattachments proxymodel/pm → proxymodels +``` + +### Lifecycle & Diagnostics + +| Command | Description | +|---------|-------------| +| `mcpctl status` | Show connectivity, auth status, LLM provider health, available models. | +| `mcpctl login` | Authenticate with mcpd (first login bootstraps initial user). | +| `mcpctl logout` | Clear stored credentials. | +| `mcpctl logs [-t N] [-i index]` | Stream container logs. Resolves server name → running instance. | +| `mcpctl cache stats` | Show pipeline cache statistics per namespace. | +| `mcpctl cache clear [ns] [--older-than N]` | Clear pipeline cache. | +| `mcpctl backup` | Show git backup status, public SSH key. | +| `mcpctl backup log [-n N]` | Show backup commit history. | +| `mcpctl backup restore list/diff/to` | Restore to specific backup commit. | + +### Console & Inspection + +| Command | Description | +|---------|-------------| +| `mcpctl console [project]` | Interactive TUI — request/response timeline, tool inspection. | +| `mcpctl console --stdin-mcp` | MCP server mode over stdin/stdout (for Claude integration). | +| `mcpctl console --audit` | Browse audit events from mcpd interactively. | + +### Configuration + +| Command | Description | +|---------|-------------| +| `mcpctl config view` | Show current configuration. | +| `mcpctl config set ` | Set config value (mcplocalUrl, mcpdUrl, registries, outputFormat, etc.). | +| `mcpctl config path` | Show config file path. | +| `mcpctl config setup` | Interactive configuration wizard. | +| `mcpctl config claude -p ` | Generate `.mcp.json` for Claude Code. | + +### Create Subcommands + +```bash +mcpctl create server [--package-name X] [--docker-image X] [--transport STDIO|SSE|STREAMABLE_HTTP] + [--runtime node|python] [--replicas N] [--env KEY=val] [--from-template name:version] + +mcpctl create secret [--data key=val ...] [--data-file path.json] + +mcpctl create project [-d desc] [--proxy-model default|gate|content-pipeline] [--server name ...] + +mcpctl create user [--password pass] [--name name] + +mcpctl create group [-d desc] [--member email ...] + +mcpctl create rbac [--subject kind:name] [--role-binding role:resource[:name]] + +mcpctl create prompt [--content text] [--project name] [--priority 1-10] [--link url] +``` + +### Apply File Format + +```yaml +secrets: + - name: my-secret + data: + KEY: value + +servers: + - name: my-server + transport: STDIO + packageName: "@modelcontextprotocol/server-example" + env: + - name: API_KEY + valueFrom: + secretRef: + name: my-secret + key: KEY + +projects: + - name: my-project + proxyModel: default + servers: + - my-server + +serverattachments: + - server: my-server + project: my-project + +prompts: + - name: my-prompt + project: my-project + content: "Instruction text..." + priority: 5 +``` + +--- + +## 5. API Surface (mcpd) + +All endpoints under `/api/v1/` require Bearer token auth except `/auth/*` and `/health*`. + +### Authentication +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/auth/bootstrap` | POST | First-user setup (creates admin + bootstrap RBAC) | +| `/auth/status` | GET | `{hasUsers: boolean}` (unauthenticated) | +| `/auth/login` | POST | Returns token + user info | +| `/auth/logout` | POST | Invalidate session | +| `/auth/me` | GET | Current user identity | +| `/auth/impersonate` | POST | Create session for another user (requires `run:impersonate`) | + +### Servers +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/servers` | GET | List all servers | +| `/servers/:id` | GET | Get server by CUID | +| `/servers` | POST | Create server (validates name uniqueness, image/package) | +| `/servers/:id` | PUT | Update server, re-reconciles replicas | +| `/servers/:id` | DELETE | Delete server + cascade-delete all instances | + +### Instances +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/instances` | GET | List (optional `?serverId=` filter) | +| `/instances/:id` | GET | Get instance | +| `/instances/:id` | DELETE | Delete instance, triggers reconciliation | +| `/instances/:id/inspect` | GET | Docker inspect output (state, port, IP) | +| `/instances/:id/logs` | GET | Container logs (`?tail=N`) | + +### Projects +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/projects` | GET | List (RBAC-filtered) | +| `/projects/:id` | GET/POST/PUT/DELETE | CRUD by CUID or name | +| `/projects/:id/mcp-config` | GET | Generate `.mcp.json` | +| `/projects/:id/instructions` | GET | Get prompt + attached servers for system message | +| `/projects/:id/servers` | GET/POST | List/attach servers | +| `/projects/:id/servers/:name` | DELETE | Detach server | + +### Prompts & Prompt Requests +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/prompts` | GET/POST | List/create approved prompts | +| `/prompts/:id` | PUT/DELETE | Update/delete (system prompts reset to default) | +| `/prompts/:id/regenerate-summary` | POST | Force re-generate summary/chapters | +| `/promptrequests` | GET/POST | List/create pending requests | +| `/promptrequests/:id/approve` | POST | Atomic delete request → create prompt | +| `/projects/:name/prompts/visible` | GET | Approved + session's pending | +| `/projects/:name/prompt-index` | GET | Compact index for gating | + +### Secrets, Users, Groups, RBAC +Standard CRUD on `/secrets`, `/users`, `/groups`, `/rbac-definitions`. + +### Health & Monitoring +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/health/overview` | GET | System health, instance counts, error rate | +| `/health/instances/:id` | GET | Instance-specific health, uptime, latency | +| `/metrics` | GET | Request counts, error counts, last request time | +| `/healthz` | GET | Liveness probe | + +### Backup, Restore, Audit +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/backup` | POST | Create encrypted bundle (servers, secrets, projects, users, groups, rbac) | +| `/restore` | POST | Restore bundle (merge/skip/overwrite strategy) | +| `/audit/events` | POST/GET | Batch insert from mcplocal / query with filters | +| `/audit/sessions` | GET | Session aggregates (first/last seen, event counts) | +| `/git/backup/init` | POST | Initialize git backup with SSH credentials | +| `/git/backup/status` | GET | Backup sync status | +| `/git/backup/sync` | POST | Manual trigger sync | + +### MCP Proxy +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/mcp/proxy` | POST | Forward JSON-RPC to running MCP server instance. Dispatches by transport (STDIO via docker exec, SSE/HTTP via direct HTTP). Maintains persistent STDIO connections. | + +--- + +## 6. Database Schema + +PostgreSQL via Prisma ORM. 22 models across 11 migrations. + +### Core Models + +**User & Auth:** +- `User` — email/password (bcrypt), role (USER/ADMIN), optional OAuth +- `Session` — Bearer token with 30-day TTL +- `Group` / `GroupMember` — user groups for RBAC + +**MCP Infrastructure:** +- `McpServer` — transport (STDIO/SSE/STREAMABLE_HTTP), docker image, package name, runtime (node/python), env vars (JSON), health check config, replicas, external URL +- `McpTemplate` — reusable blueprints for server creation (mirrors McpServer fields) +- `McpInstance` — running containers, status (STARTING/RUNNING/STOPPING/STOPPED/ERROR), container ID, port, health status, events + +**Organization:** +- `Project` — LLM config (provider, model), proxy model, gated flag, prompt instructions, server overrides +- `ProjectServer` — junction table linking projects to servers +- `Secret` — named secret bundles (data as encrypted JSON), versioned + +**Content:** +- `Prompt` — approved system prompts (global or project-scoped), priority, summary/chapters, optional link target +- `PromptRequest` — pending prompt proposals from LLM sessions + +**Audit & Backup:** +- `AuditLog` — user action trail (action, resource, resourceId, details) +- `AuditEvent` — pipeline/gate/tool trace events from mcplocal (sessionId, projectName, eventKind, correlationId, userName) +- `BackupPending` — queue for git-based backup sync +- `RbacDefinition` — named RBAC policies + +--- + +## 7. Local Proxy (mcplocal) + +### Request Flow + +``` +Claude (STDIO JSON-RPC) + ↓ +StdioProxyServer (reads from stdin) + ↓ +McpRouter.route(request) + ├→ PluginSessionContext (per-session state) + ├→ ProxyModelPlugin hooks (intercept/transform) + ├→ Upstream lookup (tool name prefix → server) + └→ Response (with optional drill-down sections) +``` + +### Router Responsibilities + +- Manages upstream connections (STDIO child processes, HTTP) +- Maps tools/resources/prompts to servers via name prefix (`servername/toolname`) +- Maintains prompt index + system prompt cache (TTL-based) +- Dispatches plugin hooks via `getOrCreatePluginContext` +- Section storage for drill-down navigation +- Audit event collection and batching +- Link resolution (relative → absolute URLs) + +### Upstream Transports + +| Transport | Implementation | +|-----------|---------------| +| STDIO | Spawns child process, bidirectional pipe, JSON-RPC over newline-delimited JSON | +| SSE | HTTP GET for event stream, POST for messages | +| Streamable HTTP | HTTP POST with JSON-RPC payloads | + +### HTTP Endpoints (mcplocal) + +| Endpoint | Description | +|----------|-------------| +| `GET /proxymodels` | List all models (YAML pipelines + TS plugins) | +| `GET /proxymodels/:name` | Single model details | +| `GET /proxymodels/stages` | List available stages | +| `POST /proxymodels/reload` | Force reload stages from disk | +| `GET /cache/stats` | Per-namespace cache statistics | +| `DELETE /cache` | Clear all or by age | +| `DELETE /cache/:namespace` | Clear specific namespace | +| `POST /mcp` | JSON-RPC request forwarding | + +--- + +## 8. ProxyModel Plugin System + +A **ProxyModel** is either a **Pipeline** (YAML) or a **Plugin** (TypeScript). + +### Plugin Interface + +| Hook | When it fires | +|------|--------------| +| `onSessionCreate` | New MCP session established | +| `onSessionDestroy` | Session ends | +| `onInitialize` | MCP initialize request — can inject instructions | +| `onToolsList` | tools/list — can filter/modify tool list | +| `onToolCallBefore` | Before forwarding a tool call — can intercept | +| `onToolCallAfter` | After receiving tool result — can transform | +| `onResourcesList` | resources/list — can filter resources | +| `onResourceRead` | resources/read — can intercept reads | +| `onPromptsList` | prompts/list — can filter prompts | +| `onPromptGet` | prompts/get — can intercept reads | + +### Built-in Plugins + +| Plugin | Extends | Gating | Content Pipeline | Use Case | +|--------|---------|:------:|:----------------:|----------| +| `gate` | — | Yes | No | Gating + prompt delivery only | +| `content-pipeline` | — | No | Yes | Content transformation only | +| `default` | gate + content-pipeline | Yes | Yes | Full pipeline (most common) | + +**Inheritance:** Plugins can extend parents. Conflicting hooks from multiple parents cause load-time errors (except chainable lifecycle hooks which run sequentially). + +### Pipeline Configuration (YAML) + +```yaml +name: default +spec: + controller: gate + controllerConfig: { byteBudget: 8192 } + stages: + - type: passthrough + - type: paginate + config: { pageSize: 8000 } + appliesTo: [prompt, toolResult] + cacheable: true +``` + +### Per-Session Context + +Each session gets a `PluginSessionContext` providing: +- Session state (`Map`) +- LLM provider, cache provider, structured logger +- Virtual tool/server registration +- Upstream routing and tool discovery +- Content processing and notifications +- Audit event emission + +--- + +## 9. Gated Sessions + +### Problem + +When Claude connects to an MCP server, it sees all tools immediately and starts using them. In a managed environment, you want to deliver relevant context (prompts/instructions) before granting tool access. + +### Solution: Keyword-Driven Prompt Retrieval + +1. **Initialize:** Instructions include prompt index + "call `begin_session` immediately" +2. **Gated `tools/list`:** Only `begin_session` visible +3. **Claude calls `begin_session`** with keywords describing the task +4. **Prompt matching:** Keywords matched against prompt summaries/chapters +5. **Ungating:** Matched prompts returned + `tools/list_changed` notification sent +6. **Full access:** All upstream tools now visible + +### Prompt Scoring + +**Formula:** `priority + (matchCount * priority)` + +- Priority alone is baseline — ensures global prompts compete for inclusion +- Tag matches multiply priority — relevant prompts score higher +- Priority 10 = always included (bypasses budget) +- 8KB byte budget cap; overflow prompts listed as index-only + +### Critical Design Lessons + +**What works:** +- One gate tool (`begin_session`), zero ambiguity +- Instructions say "check its input schema" (not naming specific parameters) +- "immediately" and "required" prevent Claude from exploring first +- Tool names listed as preview in instructions (helps keyword generation) +- `tools/list_changed` notification mandatory after ungating +- Auto-ungate fallback if Claude bypasses gate + +**What fails:** +- Naming parameters that don't match the schema +- Complex conditional instructions (Claude prefers simple paths) +- Multiple tools in gated state (Claude skips the gate) +- Gate instructions only in tool description (must be in initialize response) +- Burying the call-to-action after 200 lines of context + +### Complete Flow + +``` +Client mcplocal upstream + │── initialize ────────>│ + │<── instructions ──────│ (gate instructions + prompt index + tool preview) + │── tools/list ────────>│ + │<── [begin_session] ───│ (ONLY begin_session visible) + │── tools/call ────────>│ + │ begin_session │── match prompts ─────────>│ + │ {tags:[...]} │<── prompt content ────────│ + │<── matched prompts ───│ (full content + encouragement) + │<── notification ──────│ (tools/list_changed) + │── tools/list ────────>│ + │<── [108+ tools] ──────│ (ALL tools now visible) + │ │ + │ Claude proceeds with full tool access +``` + +--- + +## 10. Content Pipeline & Stages + +### How It Works + +Tool results pass through an ordered sequence of stages before reaching Claude: + +1. Each stage receives previous stage's content +2. Returns `{content, sections?, metadata?}` +3. Sections enable drill-down navigation +4. Stage errors are caught — pipeline continues with previous content + +### Built-in Stages + +| Stage | Purpose | +|-------|---------| +| `passthrough` | Identity transform (testing/baseline) | +| `paginate` | Split large content into numbered pages (8KB default). LLM-generated page titles (cached). | +| `section-split` | Split by structure: JSON arrays/objects → elements/keys, YAML → keys, prose → `##` headers, code → function/class boundaries. Merges tiny sections, re-splits oversized ones. | +| `summarize-tree` | Hierarchical LLM-generated section summaries. Groups sections into trees. Cached. | + +### Section Drill-Down + +After pipeline produces sections: +1. Full content replaced with compact table of contents + `_resultId` +2. Sections stored in session-scoped store (5-minute TTL) +3. Client calls same tool with `_resultId` + `_section` to retrieve specific section +4. Supports hierarchical navigation (sections within sections) + +### Custom Stages + +Drop `.js` files in `~/.mcpctl/stages/`: +```javascript +export default async function myStage(input, context) { + // context.llm, context.cache, context.log available + return { content: transformedContent, sections: [...] }; +} +``` + +Hot-reload with 300ms file watch debounce. Built-in stages take precedence. + +--- + +## 11. LLM Provider Integration + +### Supported Providers + +| Provider | Type | Tier | +|----------|------|------| +| Gemini CLI | Local | Fast | +| Ollama | Local | Fast | +| DeepSeek | API | Fast/Heavy | +| OpenAI | API | Heavy | +| Anthropic | API | Heavy | +| vLLM | Local | Configurable | +| vLLM Managed | Auto-managed local | Configurable | + +### Tier System + +- **Fast tier:** Quick, cheap models for pipeline stages and keyword extraction +- **Heavy tier:** Full models for complex prompt selection and summarization +- **Legacy active:** Single default provider (fallback) + +### LLM Adapter + +Stages use a simple interface: +```typescript +interface LLMProvider { + complete(prompt: string, options?): Promise; + available(): boolean; +} +``` + +Resolution order: named provider → fast tier → heavy tier → active provider. + +### Multi-Provider Configuration + +```json +{ + "llm": { + "providers": [ + { "name": "fast-local", "type": "ollama", "model": "llama3", "tier": "fast" }, + { "name": "heavy-api", "type": "openai", "model": "gpt-4", "tier": "heavy" } + ] + } +} +``` + +--- + +## 12. Caching + +### Architecture: L1 Memory + L2 Disk + +- **L1 in-memory:** LRU map (default 500 entries) for fast lookups +- **L2 disk:** `~/.mcpctl/cache//.dat` + - Namespace: `provider--model--proxymodel` (e.g., `openai--gpt-4o--content-pipeline`) + - Key: 16-char hex SHA256 prefix of content + - Value: raw content (no JSON wrapper) + +### Configuration + +| Option | Default | Examples | +|--------|---------|---------| +| `maxSize` | 256MB | `"1GB"`, `"10%"` (of partition), `536870912` (bytes) | +| `ttlMs` | 30 days | Any millisecond value | +| `maxMemoryEntries` | 500 | L1 LRU cap | +| `dir` | `~/.mcpctl/cache` | Custom path | + +### Management + +```bash +mcpctl cache stats # Per-namespace breakdown +mcpctl cache clear # Clear everything +mcpctl cache clear openai--gpt-4--default # Clear specific namespace +mcpctl cache clear --older-than 7 # Clear entries older than 7 days +``` + +### HTTP API +``` +GET /cache/stats # Per-namespace stats +DELETE /cache # Clear all (or ?olderThan=N) +DELETE /cache/:namespace # Clear specific namespace +``` + +--- + +## 13. Authentication & RBAC + +### Auth Flow + +1. `mcpctl login` → prompts for email/password +2. First login (no users in system) → bootstrap: creates admin user + admin group + bootstrap RBAC +3. POST `/auth/login` → returns 30-day bearer token +4. Token stored in `~/.mcpctl/credentials.json` +5. CLI passes token to mcplocal config +6. mcplocal attaches `Authorization: Bearer ` to all mcpd requests +7. mcpd validates token against Session table + +### RBAC Model + +**Subjects:** User (by email), Group, ServiceAccount + +**Roles & Capabilities:** + +| Role | Grants | +|------|--------| +| `edit` | view, create, delete, edit, expose | +| `view` | view | +| `create` | create | +| `delete` | delete | +| `run` | run (for operations) | +| `expose` | expose, view | + +**Resources:** `*`, servers, instances, secrets, projects, templates, users, groups, rbac, prompts, promptrequests + +**Binding Types:** +- **Resource binding:** `{role: 'edit', resource: 'servers', name?: 'my-server'}` + - With `name`: user can only access that specific resource + - Without `name`: user can access all resources of that type +- **Operation binding:** `{role: 'run', action: 'impersonate'}` + - Grants permission for named operations (backup, restore, audit-purge, logs, impersonate) + +### Resolution + +1. CLI resolves name → CUID client-side before API calls +2. RBAC hook resolves CUID → name before checking bindings +3. List filtering: `getAllowedScope()` computes allowed names, `preSerialization` hook filters arrays +4. Wildcard scope: user has unscoped binding → sees all resources +5. Named scope: user has only name-scoped bindings → filtered to allowed names + +--- + +## 14. Audit Infrastructure & Trust Model + +### Event Kinds + +| Event Kind | Description | +|------------|-------------| +| `pipeline_execution` | Full pipeline run summary (duration, stage count, sizes) | +| `stage_execution` | Individual stage detail (duration, input/output size, error) | +| `gate_decision` | Gate open/close with client intent and matched prompts | +| `prompt_delivery` | Which prompts were sent, match scores | +| `tool_call_trace` | Tool call with server + timing + result size | +| `rbac_decision` | Access control decisions | +| `session_bind` | Session initialization | + +### Trust Model + +| Source | Verified | Meaning | +|--------|----------|---------| +| `client` | false | Client LLM claims (begin_session intent, tags) | +| `mcplocal` | true | Server-side data (prompt matches, pipeline transforms) | +| `mcpd` | true | mcpd-originated events | + +### AuditCollector + +Fire-and-forget batching: 50 events max, 5-second flush interval. POSTs to mcpd. Non-blocking — audit failures don't affect tool calls. + +### Correlation & Causality + +- `correlationId` links related events (all events from one tool call) +- `parentEventId` enables causal chains (gate_decision → pipeline_execution) +- `userName` tracks which user triggered the event +- Designed for future graphiti knowledge graph ingestion + +### Per-Server Targeting + +Different servers in a project can have different proxymodel configs via `serverOverrides` on the project resource. Resolution: server override → project default → null. + +### Future (Designed, Not Implemented) + +- **Virtual MCP Audit Server:** mcpd-hosted virtual server providing `query_audit_log`, `get_session_timeline` tools. Claude can directly query audit data. +- **Graphiti Integration:** Causal graph with entity types (Session, Tool, Server, ProxyModel, Prompt, Stage) and edges (`triggered_by`, `transformed_by`, `verified_by`). +- **Lab Parameter Simulation:** Select any pipeline event, retrieve original input, re-run with different proxyModel/LLM/stages, side-by-side diff. +- **Audit Level Config:** Per-server `auditLevel: 'full' | 'hash-only' | 'disabled'`. + +--- + +## 15. Container Orchestration + +### Orchestrator Interface + +`McpOrchestrator` abstracts container management (Docker/Podman today, Kubernetes in the future): +- `pullImage`, `createContainer`, `stopContainer`, `removeContainer` +- `inspectContainer`, `getContainerLogs`, `execInContainer`, `ping` + +### Container Management + +- Labels: `mcpctl.managed=true` for filtering +- Network: `mcp-servers` (configurable via `MCPD_MCP_NETWORK`) +- Resource limits: 512MB RAM, 0.5 CPU (configurable) +- Internal container IP exposed via inspect + +### Runtime Spawn Commands + +| Runtime | Command | +|---------|---------| +| Node | `npx --prefer-offline -y ` | +| Python | `uvx ` | +| Custom | Explicit `command` field | + +### Health Probes + +Periodic MCP tool-call probes (like K8s livenessProbe): +- Default interval: 15 seconds +- Dispatch by transport: STDIO (docker exec), HTTP (JSON-RPC) +- Failure threshold: 3 consecutive failures → unhealthy +- Updates instance `healthStatus` and `lastHealthCheck` + +### Reconciliation Loop + +Maintains desired replica count: +- If running < desired → start new instances +- If running > desired → stop excess instances +- Detects crashed containers → marks ERROR → triggers re-creation + +### Persistent STDIO Connections + +For STDIO transport, mcpd maintains long-lived exec sessions (`PersistentStdioClient`) to avoid repeated `docker exec` overhead. Bidirectional streaming for interactive sessions. + +--- + +## 16. Deployment & Distribution + +### Production Deployment + +**mcpd runs on 10.0.0.194** (NAS, managed via Portainer), NOT on the dev machine. + +```bash +# Full deploy (preferred after merging) +bash fulldeploy.sh +``` + +`fulldeploy.sh` runs three steps: +1. `scripts/build-mcpd.sh` — build + push Docker image to `mysources.co.uk/michal/mcpctl-mcpd` +2. `deploy.sh` — deploy stack to production via Portainer API at `http://10.0.0.194:9000` +3. `scripts/release.sh` — build RPM + publish to Gitea + install locally + smoke tests + +### Docker Images + +| Image | Purpose | +|-------|---------| +| `mcpctl-mcpd` | Multi-stage build: Node 20 Alpine, includes git/ssh, Prisma | +| `mcpctl-node-runner` | Node 20 slim, runs `npx -y` for npm packages | +| `mcpctl-python-runner` | Python 3.12 slim, uses `uv` for Python packages | + +All pushed to `mysources.co.uk/michal/` registry. + +### Stack Services (Production) + +- `postgres` — PostgreSQL 16 (port 5432) +- `mcpd` — Daemon (port 3100) +- `node-runner`, `python-runner` — Base images +- Networks: `mcpctl` (management), `mcp-servers` (container communication) + +### RPM/DEB Distribution + +```bash +source .env && bash scripts/release.sh +``` + +Installs via nfpm: +- `/usr/bin/mcpctl` — CLI binary (bun compiled) +- `/usr/bin/mcpctl-local` — Local proxy binary (bun compiled) +- `/usr/share/fish/vendor_completions.d/mcpctl.fish` — Fish completions +- `/usr/share/bash-completion/completions/mcpctl` — Bash completions +- `/usr/lib/systemd/user/mcplocal.service` — Systemd user service + +User install: +```bash +dnf config-manager --add-repo https://mysources.co.uk/api/packages/michal/rpm.repo +dnf install mcpctl +``` + +### Git & PR Workflow + +- Gitea at `http://10.0.0.194:3012` (internal) / `https://mysources.co.uk/michal/mcpctl` (public) +- `pr.sh` in project root creates PRs via Gitea API +- `gh` CLI not installed — use `pr.sh` or direct API calls + +--- + +## 17. Testing Strategy + +### Test Tiers + +| Tier | Tool | Scope | When | +|------|------|-------|------| +| Unit tests | Vitest | Package-level, mocked dependencies | `pnpm test:run` | +| DB tests | Vitest | Full Prisma + test PostgreSQL | `pnpm --filter db exec vitest run` (separate) | +| Smoke tests | Vitest | Live mcplocal + mcpd (not mocked) | `pnpm test:smoke` (post-deploy) | + +### Convention + +- Every new feature MUST include smoke tests +- Smoke tests live in `src/mcplocal/tests/smoke/` +- Use `SmokeMcpSession` from `tests/smoke/mcp-client.ts` for MCP protocol interactions +- Smoke tests run automatically in the build/deploy pipeline + +### Critical Rules + +- **NEVER pipe pnpm test output** to `tail`, `grep`, `head` — pnpm hangs when it detects non-TTY +- Always capture full output with `2>&1` and read directly +- DB tests excluded from workspace-root vitest (need test database) +- Tests integrated into pipeline: `build-rpm.sh` runs unit tests; `release.sh` runs smoke tests + +--- + +## 18. Technology Stack + +| Layer | Technology | +|-------|-----------| +| CLI Framework | Commander.js, Ink/React (TUI), Inquirer | +| API Server | Fastify 5, TypeScript strict mode | +| Database | PostgreSQL 16, Prisma ORM v6 | +| Container Runtime | Docker/Podman via dockerode | +| MCP Protocol | @modelcontextprotocol/sdk | +| Validation | Zod schemas everywhere | +| LLM Providers | OpenAI, Anthropic, Google Gemini, Ollama, DeepSeek, Groq, Mistral, OpenRouter, Azure | +| Testing | Vitest, coverage via v8 | +| Build | TypeScript project references, pnpm workspaces | +| Compilation | Bun (binary compilation for RPM) | +| Packaging | nfpm (RPM/DEB), Docker multi-stage | +| CI/CD | fulldeploy.sh → Portainer API + Gitea packages | +| Shell Completions | Fish + Bash (auto-generated via `scripts/generate-completions.ts`) | + +### Design Patterns + +1. **Monorepo** — pnpm workspaces with shared base TypeScript config +2. **Layered architecture** — Routes → Services → Repositories (Prisma) +3. **Interface-based repositories** — all data access through interfaces for testability +4. **Dependency injection** — services receive dependencies via constructor +5. **Zod validation** — all input validated at API boundary +6. **Plugin inheritance** — composable ProxyModel plugins with conflict detection +7. **Content-addressed caching** — SHA256 hash keys for deduplication +8. **TTL-based stores** — prompt index (60s), system prompts (5min), sections (5min) +9. **Fire-and-forget audit** — non-blocking event collection +10. **Declarative config** — kubectl-style YAML/JSON for all resource management + +--- + +## 19. Project Structure + +``` +mcpctl/ +├── src/ +│ ├── cli/ @mcpctl/cli CLI (Commander.js) +│ │ ├── src/commands/ 22 command handlers +│ │ ├── src/registry/ MCP server registry client +│ │ ├── src/formatters/ Output formatting (table/json/yaml) +│ │ └── src/auth/ Credential storage +│ ├── mcpd/ @mcpctl/mcpd Daemon (Fastify 5) +│ │ ├── src/routes/ 18 route handlers +│ │ ├── src/services/ 13 services +│ │ ├── src/repositories/ Data access layer +│ │ ├── src/middleware/ Auth, logging, error handling +│ │ └── src/validation/ Zod schemas, RBAC rules +│ ├── mcplocal/ @mcpctl/mcplocal Local proxy +│ │ ├── src/gate/ Session gating + tag matching +│ │ ├── src/proxymodel/ Plugin system + stages + cache +│ │ ├── src/providers/ 6 LLM providers +│ │ ├── src/upstream/ STDIO + HTTP upstream connections +│ │ ├── src/audit/ Event collection + batching +│ │ └── src/health/ Health monitoring +│ ├── db/ @mcpctl/db Database (Prisma) +│ │ ├── prisma/schema.prisma 22 models +│ │ └── prisma/migrations/ 11 migrations +│ └── shared/ @mcpctl/shared Constants, types, validation +├── deploy/ Dockerfiles + entrypoint +├── stack/ Production docker-compose + env +├── scripts/ Build, release, deploy scripts +├── completions/ Fish + Bash completions +├── templates/ MCP server YAML templates +├── docs/ Architecture + design docs +├── fulldeploy.sh Full build → deploy → release +├── deploy.sh Portainer stack deploy +├── pr.sh Gitea PR creation +├── nfpm.yaml RPM/DEB package metadata +├── vitest.config.ts Root test config +├── vitest.workspace.ts Workspace test config +├── tsconfig.base.json Base TypeScript config (strict) +└── pnpm-workspace.yaml Monorepo workspace definition +``` + +--- + +## 20. Deferred & Future Work + +### Deferred Tasks + +| ID | Description | Status | +|----|-------------|--------| +| 88 | Rename proxyMode: filtered → proxy | Deferred | +| 105-109 | Model Studio TUI | Deferred | +| 110 | RBAC for ProxyModels | Deferred | +| 113 | Model Studio docs | Deferred | + +### Future Architecture + +- **Virtual MCP Audit Server** — Claude-queryable audit tools +- **Graphiti Knowledge Graph** — causal graph from audit events +- **Lab Parameter Simulation** — re-run pipelines with different configs +- **Kubernetes Orchestrator** — beyond Docker/Podman +- **ConfigMaps** — non-sensitive config separate from Secrets +- **Multi-provider failover** — automatic LLM provider cascading + +### Completed Major Features + +- Project structure + monorepo setup +- MCP Registry Client (official, glama, smithery — 53 tests) +- Health Probe Runner (STDIO, SSE, Streamable HTTP — 12 tests) +- Container orchestration with reconciliation +- Full RBAC with name-scoped bindings +- Gated sessions with prompt scoring +- ProxyModel plugin system with inheritance +- Content pipeline with 4 built-in stages +- Pipeline cache (L1 memory + L2 disk) +- Audit infrastructure with trust model +- Git-based backup and restore +- Shell completions (Fish + Bash) +- RPM/DEB packaging and distribution +- Smoke test framework +- Console inspector for debugging diff --git a/src/mcpd/src/main.ts b/src/mcpd/src/main.ts index a7c212d..aac4493 100644 --- a/src/mcpd/src/main.ts +++ b/src/mcpd/src/main.ts @@ -287,8 +287,6 @@ async function main(): Promise { const auditEventService = new AuditEventService(auditEventRepo); const metricsCollector = new MetricsCollector(); const healthAggregator = new HealthAggregator(metricsCollector, orchestrator); - const backupService = new BackupService(serverRepo, projectRepo, secretRepo, userRepo, groupRepo, rbacDefinitionRepo); - const restoreService = new RestoreService(serverRepo, projectRepo, secretRepo, userRepo, groupRepo, rbacDefinitionRepo); const authService = new AuthService(prisma); const templateService = new TemplateService(templateRepo); const mcpProxyService = new McpProxyService(instanceRepo, serverRepo, orchestrator); @@ -301,6 +299,8 @@ async function main(): Promise { const promptRuleRegistry = new ResourceRuleRegistry(); promptRuleRegistry.register(systemPromptVarsRule); const promptService = new PromptService(promptRepo, promptRequestRepo, projectRepo, promptRuleRegistry); + const backupService = new BackupService(serverRepo, projectRepo, secretRepo, userRepo, groupRepo, rbacDefinitionRepo, promptRepo, templateRepo); + const restoreService = new RestoreService(serverRepo, projectRepo, secretRepo, userRepo, groupRepo, rbacDefinitionRepo, promptRepo, templateRepo); // Auth middleware for global hooks const authMiddleware = createAuthMiddleware({ diff --git a/src/mcpd/src/services/backup/backup-service.ts b/src/mcpd/src/services/backup/backup-service.ts index c4a468a..d3b1f89 100644 --- a/src/mcpd/src/services/backup/backup-service.ts +++ b/src/mcpd/src/services/backup/backup-service.ts @@ -3,6 +3,8 @@ import type { IProjectRepository } from '../../repositories/project.repository.j import type { IUserRepository } from '../../repositories/user.repository.js'; import type { IGroupRepository } from '../../repositories/group.repository.js'; import type { IRbacDefinitionRepository } from '../../repositories/rbac-definition.repository.js'; +import type { IPromptRepository } from '../../repositories/prompt.repository.js'; +import type { ITemplateRepository } from '../../repositories/template.repository.js'; import { encrypt, isSensitiveKey } from './crypto.js'; import type { EncryptedPayload } from './crypto.js'; import { APP_VERSION } from '@mcpctl/shared'; @@ -18,6 +20,8 @@ export interface BackupBundle { users?: BackupUser[]; groups?: BackupGroup[]; rbacBindings?: BackupRbacBinding[]; + prompts?: BackupPrompt[]; + templates?: BackupTemplate[]; encryptedSecrets?: EncryptedPayload; } @@ -25,10 +29,16 @@ export interface BackupServer { name: string; description: string; packageName: string | null; + runtime: string | null; dockerImage: string | null; transport: string; repositoryUrl: string | null; + externalUrl: string | null; + command: unknown; + containerPort: number | null; + replicas: number; env: unknown; + healthCheck: unknown; } export interface BackupSecret { @@ -65,9 +75,31 @@ export interface BackupRbacBinding { roleBindings: unknown; } +export interface BackupPrompt { + name: string; + content: string; + projectName: string | null; + priority: number; + summary: string | null; + chapters: unknown; + linkTarget: string | null; +} + +export interface BackupTemplate { + name: string; + description: string; + packageName: string | null; + dockerImage: string | null; + transport: string; + command: unknown; + containerPort: number | null; + env: unknown; + healthCheck: unknown; +} + export interface BackupOptions { password?: string; - resources?: Array<'servers' | 'secrets' | 'projects' | 'users' | 'groups' | 'rbac'>; + resources?: Array<'servers' | 'secrets' | 'projects' | 'users' | 'groups' | 'rbac' | 'prompts' | 'templates'>; } export class BackupService { @@ -78,10 +110,12 @@ export class BackupService { private userRepo?: IUserRepository, private groupRepo?: IGroupRepository, private rbacRepo?: IRbacDefinitionRepository, + private promptRepo?: IPromptRepository, + private templateRepo?: ITemplateRepository, ) {} async createBackup(options?: BackupOptions): Promise { - const resources = options?.resources ?? ['servers', 'secrets', 'projects', 'users', 'groups', 'rbac']; + const resources = options?.resources ?? ['servers', 'secrets', 'projects', 'users', 'groups', 'rbac', 'prompts', 'templates']; let servers: BackupServer[] = []; let secrets: BackupSecret[] = []; @@ -96,10 +130,16 @@ export class BackupService { name: s.name, description: s.description, packageName: s.packageName, + runtime: s.runtime, dockerImage: s.dockerImage, transport: s.transport, repositoryUrl: s.repositoryUrl, + externalUrl: s.externalUrl, + command: s.command, + containerPort: s.containerPort, + replicas: s.replicas, env: s.env, + healthCheck: s.healthCheck, })); } @@ -151,6 +191,37 @@ export class BackupService { })); } + let prompts: BackupPrompt[] = []; + let templates: BackupTemplate[] = []; + + if (resources.includes('prompts') && this.promptRepo) { + const allPrompts = await this.promptRepo.findAll(); + prompts = allPrompts.map((p) => ({ + name: p.name, + content: p.content, + projectName: (p as unknown as { project?: { name: string } }).project?.name ?? null, + priority: p.priority, + summary: p.summary, + chapters: p.chapters, + linkTarget: p.linkTarget, + })); + } + + if (resources.includes('templates') && this.templateRepo) { + const allTemplates = await this.templateRepo.findAll(); + templates = allTemplates.map((t) => ({ + name: t.name, + description: t.description, + packageName: t.packageName, + dockerImage: t.dockerImage, + transport: t.transport, + command: t.command, + containerPort: t.containerPort, + env: t.env, + healthCheck: t.healthCheck, + })); + } + const bundle: BackupBundle = { version: '1', mcpctlVersion: APP_VERSION, @@ -162,6 +233,8 @@ export class BackupService { users, groups, rbacBindings, + prompts, + templates, }; if (options?.password && secrets.length > 0) { diff --git a/src/mcpd/src/services/backup/restore-service.ts b/src/mcpd/src/services/backup/restore-service.ts index 683c1cc..6771948 100644 --- a/src/mcpd/src/services/backup/restore-service.ts +++ b/src/mcpd/src/services/backup/restore-service.ts @@ -3,6 +3,8 @@ import type { IProjectRepository } from '../../repositories/project.repository.j import type { IUserRepository } from '../../repositories/user.repository.js'; import type { IGroupRepository } from '../../repositories/group.repository.js'; import type { IRbacDefinitionRepository } from '../../repositories/rbac-definition.repository.js'; +import type { IPromptRepository } from '../../repositories/prompt.repository.js'; +import type { ITemplateRepository } from '../../repositories/template.repository.js'; import type { RbacRoleBinding } from '../../validation/rbac-definition.schema.js'; import { decrypt } from './crypto.js'; import type { BackupBundle } from './backup-service.js'; @@ -27,6 +29,10 @@ export interface RestoreResult { groupsSkipped: number; rbacCreated: number; rbacSkipped: number; + promptsCreated: number; + promptsSkipped: number; + templatesCreated: number; + templatesSkipped: number; errors: string[]; } @@ -38,6 +44,8 @@ export class RestoreService { private userRepo?: IUserRepository, private groupRepo?: IGroupRepository, private rbacRepo?: IRbacDefinitionRepository, + private promptRepo?: IPromptRepository, + private templateRepo?: ITemplateRepository, ) {} validateBundle(bundle: unknown): bundle is BackupBundle { @@ -67,6 +75,10 @@ export class RestoreService { groupsSkipped: 0, rbacCreated: 0, rbacSkipped: 0, + promptsCreated: 0, + promptsSkipped: 0, + templatesCreated: 0, + templatesSkipped: 0, errors: [], }; @@ -159,12 +171,17 @@ export class RestoreService { name: server.name, description: server.description, transport: server.transport as 'STDIO' | 'SSE' | 'STREAMABLE_HTTP', - replicas: (server as { replicas?: number }).replicas ?? 1, + replicas: server.replicas ?? 1, env: (server.env ?? []) as Array<{ name: string; value?: string; valueFrom?: { secretRef: { name: string; key: string } } }>, }; if (server.packageName) createData.packageName = server.packageName; + if (server.runtime) createData.runtime = server.runtime; if (server.dockerImage) createData.dockerImage = server.dockerImage; if (server.repositoryUrl) createData.repositoryUrl = server.repositoryUrl; + if (server.externalUrl) createData.externalUrl = server.externalUrl; + if (server.command) createData.command = server.command as string[]; + if (server.containerPort) createData.containerPort = server.containerPort; + if (server.healthCheck) createData.healthCheck = server.healthCheck as Parameters[0]['healthCheck']; await this.serverRepo.create(createData); result.serversCreated++; } catch (err) { @@ -337,6 +354,87 @@ export class RestoreService { } } + // Restore prompts (after projects, so projectId can be resolved) + if (bundle.prompts && this.promptRepo) { + for (const prompt of bundle.prompts) { + try { + // Resolve project by name + let projectId: string | undefined; + if (prompt.projectName) { + const project = await this.projectRepo.findByName(prompt.projectName); + if (project) projectId = project.id; + } + + const existing = await this.promptRepo.findByNameAndProject(prompt.name, projectId ?? null); + if (existing) { + if (strategy === 'fail') { + result.errors.push(`Prompt "${prompt.name}" already exists`); + return result; + } + if (strategy === 'skip') { + result.promptsSkipped++; + continue; + } + // overwrite + const updateData: { content: string; priority: number; summary?: string } = { + content: prompt.content, + priority: prompt.priority, + }; + if (prompt.summary) updateData.summary = prompt.summary; + await this.promptRepo.update(existing.id, updateData); + result.promptsCreated++; + continue; + } + + const createData: { name: string; content: string; projectId?: string; priority?: number; linkTarget?: string } = { + name: prompt.name, + content: prompt.content, + }; + if (projectId) createData.projectId = projectId; + if (prompt.priority !== 5) createData.priority = prompt.priority; + if (prompt.linkTarget) createData.linkTarget = prompt.linkTarget; + await this.promptRepo.create(createData); + result.promptsCreated++; + } catch (err) { + result.errors.push(`Failed to restore prompt "${prompt.name}": ${err instanceof Error ? err.message : String(err)}`); + } + } + } + + // Restore templates + if (bundle.templates && this.templateRepo) { + for (const tmpl of bundle.templates) { + try { + const existing = await this.templateRepo.findByName(tmpl.name); + if (existing) { + if (strategy === 'skip') { + result.templatesSkipped++; + continue; + } + // overwrite or fail handled by upsert + result.templatesSkipped++; + continue; + } + + const tmplData: Record = { + name: tmpl.name, + description: tmpl.description, + transport: tmpl.transport as 'STDIO' | 'SSE' | 'STREAMABLE_HTTP', + }; + if (tmpl.packageName) tmplData.packageName = tmpl.packageName; + if (tmpl.dockerImage) tmplData.dockerImage = tmpl.dockerImage; + if (tmpl.command) tmplData.command = tmpl.command; + if (tmpl.containerPort) tmplData.containerPort = tmpl.containerPort; + if (tmpl.env) tmplData.env = tmpl.env; + if (tmpl.healthCheck) tmplData.healthCheck = tmpl.healthCheck; + await this.templateRepo.create(tmplData as Parameters[0]); + result.templatesCreated++; + } catch (err) { + result.errors.push(`Failed to restore template "${tmpl.name}": ${err instanceof Error ? err.message : String(err)}`); + } + } + } + return result; } diff --git a/src/mcpd/src/services/k8s/k8s-client-official.ts b/src/mcpd/src/services/k8s/k8s-client-official.ts index 9ba8271..44db3b8 100644 --- a/src/mcpd/src/services/k8s/k8s-client-official.ts +++ b/src/mcpd/src/services/k8s/k8s-client-official.ts @@ -21,6 +21,7 @@ export class K8sOfficialClient { readonly kc: k8s.KubeConfig; readonly core: k8s.CoreV1Api; readonly exec: k8s.Exec; + readonly attach: k8s.Attach; readonly log: k8s.Log; readonly serversNamespace: string; @@ -36,6 +37,7 @@ export class K8sOfficialClient { this.core = this.kc.makeApiClient(k8s.CoreV1Api); this.exec = new k8s.Exec(this.kc); + this.attach = new k8s.Attach(this.kc); this.log = new k8s.Log(this.kc); this.serversNamespace = opts?.serversNamespace ?? process.env['MCPD_SERVERS_NAMESPACE'] diff --git a/src/mcpd/src/services/k8s/kubernetes-orchestrator.ts b/src/mcpd/src/services/k8s/kubernetes-orchestrator.ts index 711cf30..515aa46 100644 --- a/src/mcpd/src/services/k8s/kubernetes-orchestrator.ts +++ b/src/mcpd/src/services/k8s/kubernetes-orchestrator.ts @@ -257,6 +257,46 @@ export class KubernetesOrchestrator implements McpOrchestrator { }; } + /** + * Attach to a running container's main process (PID 1) stdin/stdout. + * Used for docker-image STDIO servers where the entrypoint IS the MCP server. + */ + async attachInteractive( + containerId: string, + ): Promise { + const containerName = await this.getContainerName(containerId); + const stdout = new PassThrough(); + const stdinStream = new PassThrough(); + + const stderrStream = new Writable({ + write(_chunk: Buffer, _encoding, callback) { + callback(); + }, + }); + + const ws = await this.client.attach.attach( + this.namespace, + containerId, + containerName, + stdout, + stderrStream, + stdinStream, + false, // tty + ); + + return { + stdout, + write(data: string) { + stdinStream.write(data); + }, + close() { + stdinStream.end(); + stdout.destroy(); + ws.close(); + }, + }; + } + async listContainers(namespace?: string): Promise { const ns = namespace ?? this.namespace; const podList = await this.client.core.listNamespacedPod({ diff --git a/src/mcpd/src/services/mcp-proxy-service.ts b/src/mcpd/src/services/mcp-proxy-service.ts index 3ea4365..b67af77 100644 --- a/src/mcpd/src/services/mcp-proxy-service.ts +++ b/src/mcpd/src/services/mcp-proxy-service.ts @@ -140,8 +140,13 @@ export class McpProxyService { } const packageName = server.packageName as string | null; const command = server.command as string[] | null; + if (!packageName && (!command || command.length === 0)) { - throw new InvalidStateError(`Server '${server.id}' has no packageName or command for STDIO transport`); + throw new InvalidStateError( + `Server '${server.name}' (${server.id}) uses STDIO transport with a docker image ` + + `but has no command. Set 'command' to the image's entrypoint ` + + `(e.g. mcpctl edit server ${server.name} --command node --command build/index.js)` + ); } // Build the spawn command based on runtime diff --git a/src/mcpd/src/services/orchestrator.ts b/src/mcpd/src/services/orchestrator.ts index 8c06815..e6767c1 100644 --- a/src/mcpd/src/services/orchestrator.ts +++ b/src/mcpd/src/services/orchestrator.ts @@ -71,6 +71,9 @@ export interface McpOrchestrator { /** Start a long-running interactive exec session (bidirectional stdio stream). */ execInteractive?(containerId: string, cmd: string[]): Promise; + /** Attach to a running container's main process stdin/stdout (PID 1). */ + attachInteractive?(containerId: string): Promise; + /** Check if the orchestrator runtime is available */ ping(): Promise; } diff --git a/src/mcpd/tests/k8s-orchestrator.test.ts b/src/mcpd/tests/k8s-orchestrator.test.ts index ff7c308..2f983eb 100644 --- a/src/mcpd/tests/k8s-orchestrator.test.ts +++ b/src/mcpd/tests/k8s-orchestrator.test.ts @@ -67,6 +67,10 @@ vi.mock('@kubernetes/client-node', () => { exec = vi.fn(); } + class MockAttach { + attach = vi.fn(); + } + class MockLog { log = vi.fn(); } @@ -75,6 +79,7 @@ vi.mock('@kubernetes/client-node', () => { KubeConfig: MockKubeConfig, CoreV1Api: class {}, Exec: MockExec, + Attach: MockAttach, Log: MockLog, // Export test helpers __testHelpers: { setHandler, getHandler, clearHandlers, mockCore }, diff --git a/templates/gitea.yaml b/templates/gitea.yaml new file mode 100644 index 0000000..fc1d7da --- /dev/null +++ b/templates/gitea.yaml @@ -0,0 +1,22 @@ +name: gitea +version: "1.0.0" +description: Gitea MCP server for repositories, issues, PRs, and code management +dockerImage: "docker.gitea.com/gitea-mcp-server:latest" +transport: STDIO +repositoryUrl: https://gitea.com/gitea/gitea-mcp +command: + - /app/gitea-mcp + - -t + - stdio +# Health check disabled: STDIO health probe requires packageName (npm-based servers). +# This server uses a custom dockerImage. Probe support for dockerImage STDIO servers is TODO. +env: + - name: GITEA_HOST + description: Gitea instance URL (e.g. https://gitea.example.com) + required: true + - name: GITEA_ACCESS_TOKEN + description: Gitea personal access token + required: true + - name: GITEA_INSECURE + description: Allow self-signed certificates (true/false, default false) + required: false diff --git a/templates/unifi-network.yaml b/templates/unifi-network.yaml new file mode 100644 index 0000000..8d5c14a --- /dev/null +++ b/templates/unifi-network.yaml @@ -0,0 +1,25 @@ +name: unifi-network +version: "1.0.0" +description: UniFi Network MCP server for managing UniFi network devices, clients, and configuration +packageName: "unifi-network-mcp" +runtime: python +transport: STDIO +repositoryUrl: https://github.com/sirkirby/unifi-mcp +# Health check disabled: STDIO health probe requires packageName (npm-based servers). +# This server uses the Python runner. Probe support for Python runner STDIO servers is TODO. +env: + - name: UNIFI_HOST + description: UniFi controller hostname or IP (e.g. unifi.example.com — without https://) + required: true + - name: UNIFI_USERNAME + description: UniFi local admin username + required: true + - name: UNIFI_PASSWORD + description: UniFi admin password + required: true + - name: UNIFI_NETWORK_PORT + description: UniFi controller port (default 443, use 8443 for standalone UniFi Controller) + required: false + - name: UNIFI_NETWORK_VERIFY_SSL + description: Verify SSL certificate (true/false, default true — set false for self-signed certs) + required: false