mcpctl/.taskmaster/tasks/task_018.md

# Task ID: 18

**Title:** Create End-to-End Integration and Documentation

**Status:** pending

**Dependencies:** 9, 13, 14, 15, 16

**Priority:** medium

**Description:** Build comprehensive integration tests, usage documentation, and example workflows for the complete mcpctl system.

**Details:**

Create E2E tests and documentation:

```typescript
// tests/e2e/full-workflow.test.ts
describe('mcpctl E2E', () => {
  test('complete workflow: setup to Claude usage', async () => {
    // 1. Start mcpd server
    const mcpd = await startMcpd();

    // 2. Setup MCP server via CLI
    await exec('mcpctl setup slack --non-interactive --token=test-token');

    // 3. Create project
    await exec('mcpctl project create weekly_reports --profiles slack-read-only jira-read-only');

    // 4. Add to Claude config
    await exec('mcpctl claude add-mcp-project weekly_reports');
    const mcpJson = JSON.parse(fs.readFileSync('.mcp.json', 'utf8'));
    expect(mcpJson.mcpServers['mcpctl-proxy']).toBeDefined();

    // 5. Start local proxy
    const proxy = await startLocalProxy();

    // 6. Simulate Claude request through proxy
    const response = await proxy.callTool('slack_get_messages', {
      channel: 'team',
      _context: 'Find security-related messages'
    });

    // 7. Verify response is filtered
    expect(response.content.length).toBeLessThan(originalData.length);

    // 8. Verify audit log
    const audit = await exec('mcpctl audit --limit 1');
    expect(audit).toContain('mcp_call');
  });
});
```

Documentation structure:
```
docs/
├── getting-started.md
├── installation.md
├── configuration.md
├── cli-reference.md
├── mcp-servers/
│   ├── slack.md
│   ├── jira.md
│   └── github.md
├── architecture.md
├── local-llm-setup.md
├── deployment/
│   ├── docker-compose.md
│   └── kubernetes.md
└── examples/
    ├── weekly-reports.md
    └── terraform-docs.md
```

Example workflow documentation:
```markdown
# Weekly Reports Workflow

## Setup
```bash
# Install mcpctl
npm install -g mcpctl

# Configure server
mcpctl config set-server http://your-nas:3000

# Setup MCPs
mcpctl setup slack
mcpctl setup jira

# Create project
mcpctl project create weekly_reports --profiles slack-team jira-myproject

# Add to Claude
mcpctl claude add-mcp-project weekly_reports
```

## Usage with Claude
In your Claude session:
> "Write me a weekly report. Get all messages from Slack related to my team and security, and all Jira tickets I worked on this week."

The local proxy will filter thousands of messages to only the relevant ones.
```

**Test Strategy:**

Run full E2E test suite. Test all documented workflows work as described. Validate documentation accuracy with fresh setup. Test error scenarios and recovery.

## Subtasks

### 18.1. Build E2E Test Infrastructure with Docker Compose Local Environment

**Status:** pending  
**Dependencies:** None  

Create the complete E2E test infrastructure using docker-compose that runs mcpd, PostgreSQL, mock MCP servers, and local LLM proxy entirely locally without external dependencies.

**Details:**

Create tests/e2e directory structure with:

**docker-compose.e2e.yml:**
```yaml
version: '3.8'
services:
  postgres-e2e:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: mcpctl_test
      POSTGRES_PASSWORD: test_password
      POSTGRES_DB: mcpctl_test
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U mcpctl_test']
  mcpd:
    build: ../../src/mcpd
    depends_on:
      postgres-e2e:
        condition: service_healthy
    environment:
      DATABASE_URL: postgresql://mcpctl_test:test_password@postgres-e2e:5432/mcpctl_test
  mock-slack-mcp:
    build: ./mocks/slack-mcp
    ports: ['9001:9001']
  mock-jira-mcp:
    build: ./mocks/jira-mcp
    ports: ['9002:9002']
  ollama:
    image: ollama/ollama:latest
    volumes: ['ollama-data:/root/.ollama']
  local-proxy:
    build: ../../src/local-proxy
    depends_on: [mcpd, ollama]
volumes:
  ollama-data:
```

**tests/e2e/setup.ts:**
- startE2EEnvironment(): Start all containers, wait for health
- stopE2EEnvironment(): Stop and cleanup containers
- resetDatabase(): Truncate all tables between tests
- getMcpdClient(): Return configured API client for mcpd
- getProxyClient(): Return configured MCP client for local-proxy

**tests/e2e/mocks/slack-mcp/**: Dockerfile and Node.js mock implementing MCP protocol returning configurable test data (1000+ messages for filtering tests)

**tests/e2e/mocks/jira-mcp/**: Similar mock for Jira with test tickets

**tests/e2e/fixtures/**: Test data files (slack-messages.json, jira-tickets.json) with realistic but synthetic data

**tests/e2e/vitest.config.ts:**
```typescript
export default defineConfig({
  test: {
    globalSetup: './setup.ts',
    testTimeout: 120000,
    hookTimeout: 60000,
    setupFiles: ['./test-utils.ts']
  }
});
```

Add scripts to root package.json:
- "test:e2e": "vitest run --config tests/e2e/vitest.config.ts"
- "test:e2e:up": "docker-compose -f tests/e2e/docker-compose.e2e.yml up -d"
- "test:e2e:down": "docker-compose -f tests/e2e/docker-compose.e2e.yml down -v"

### 18.2. Implement Full Workflow E2E Tests with Security Validation

**Status:** pending  
**Dependencies:** 18.1  

Create comprehensive E2E test suites covering the complete user workflow from CLI setup through proxy usage, plus security-focused tests verifying no credential leakage, proper auth flows, and permission boundary enforcement.

**Details:**

Create tests/e2e/workflows directory with test files:

**tests/e2e/workflows/full-workflow.test.ts:**
```typescript
describe('mcpctl E2E: Complete Workflow', () => {
  test('setup to Claude usage', async () => {
    // 1. Start mcpd server (via docker-compose)
    expect(await getMcpdHealth()).toBe('ok');
    
    // 2. Setup MCP server via CLI
    const setupResult = await exec('mcpctl setup slack --non-interactive --token=xoxb-test-token');
    expect(setupResult.exitCode).toBe(0);
    
    // 3. Create project with profiles
    await exec('mcpctl project create weekly_reports --profiles slack-read-only jira-read-only');
    const project = await getMcpdClient().getProject('weekly_reports');
    expect(project.profiles).toHaveLength(2);
    
    // 4. Add to Claude config
    await exec('mcpctl claude add-mcp-project weekly_reports');
    const mcpJson = JSON.parse(fs.readFileSync('.mcp.json', 'utf8'));
    expect(mcpJson.mcpServers['mcpctl-proxy']).toBeDefined();
    expect(mcpJson.mcpServers['mcpctl-proxy'].env.SLACK_BOT_TOKEN).toBeUndefined(); // No secrets!
    
    // 5. Simulate proxy request with context filtering
    const response = await getProxyClient().callTool('slack_get_messages', {
      channel: 'team',
      _context: 'Find security-related messages'
    });
    expect(response.content.length).toBeLessThan(1000); // Filtered from 1000+ test messages
    
    // 6. Verify audit log
    const audit = await exec('mcpctl audit --limit 1 --format json');
    expect(JSON.parse(audit.stdout)[0].action).toBe('mcp_call');
  });
});
```

**tests/e2e/security/credential-leakage.test.ts:**
```typescript
describe('Security: No Credential Leakage', () => {
  test('.mcp.json never contains actual secrets', async () => {
    await exec('mcpctl setup slack --token=xoxb-real-token');
    await exec('mcpctl claude add-mcp-project test_project');
    const content = fs.readFileSync('.mcp.json', 'utf8');
    expect(content).not.toContain('xoxb-');
    expect(content).not.toMatch(/[A-Za-z0-9]{32,}/);
  });
  
  test('audit logs scrub sensitive data', async () => {
    await exec('mcpctl setup jira --token=secret-api-token');
    const logs = await prisma.auditLog.findMany({ where: { action: 'mcp_server_setup' }});
    logs.forEach(log => {
      expect(JSON.stringify(log.details)).not.toContain('secret-api-token');
    });
  });
  
  test('CLI history does not contain tokens', async () => {
    // Verify --token values are masked in any logged commands
  });
});
```

**tests/e2e/security/auth-flows.test.ts:**
```typescript
describe('Security: Authentication Flows', () => {
  test('API rejects requests without valid token', async () => {
    const response = await fetch(`${MCPD_URL}/api/projects`, {
      headers: { 'Authorization': 'Bearer invalid-token' }
    });
    expect(response.status).toBe(401);
  });
  
  test('expired sessions are rejected', async () => {
    const expiredSession = await createExpiredSession();
    const response = await authenticatedFetch('/api/projects', expiredSession.token);
    expect(response.status).toBe(401);
  });
});
```

**tests/e2e/security/permission-boundaries.test.ts:**
```typescript
describe('Security: Permission Boundaries', () => {
  test('read-only profile cannot call write operations', async () => {
    await exec('mcpctl project create readonly_test --profiles slack-read-only');
    const response = await getProxyClient().callTool('slack_post_message', {
      channel: 'general',
      text: 'test'
    });
    expect(response.error).toContain('permission denied');
  });
});
```

**tests/e2e/workflows/error-recovery.test.ts:** Test scenarios for network failures, container restarts, database disconnections with proper recovery

### 18.3. Create User and Technical Documentation Suite

**Status:** pending  
**Dependencies:** None  

Build comprehensive documentation covering getting started, installation, configuration, CLI reference, architecture overview, and local LLM setup guides with proper markdown structure.

**Details:**

Create docs/ directory structure:

**docs/getting-started.md:**
- Quick 5-minute setup guide
- Prerequisites (Node.js, Docker, pnpm)
- Install mcpctl globally: `npm install -g mcpctl`
- Start mcpd: `docker-compose up -d` or `mcpctl daemon start`
- Configure first MCP server: `mcpctl setup slack`
- Create first project: `mcpctl project create my_assistant --profiles slack-read-only`
- Add to Claude: `mcpctl claude add-mcp-project my_assistant`
- Verify with `mcpctl status`

**docs/installation.md:**
- NPM global install: `npm install -g mcpctl`
- Docker deployment: Using provided docker-compose.yml
- Kubernetes deployment: Helm chart reference (link to deployment/kubernetes.md)
- Building from source: Clone, pnpm install, pnpm build
- Verifying installation: `mcpctl version`, `mcpctl doctor`

**docs/configuration.md:**
- Environment variables reference (DATABASE_URL, MCPD_URL, LOG_LEVEL, etc.)
- Configuration file locations (~/.mcpctl/config.yaml, .mcpctl.yaml)
- Per-project configuration (.mcpctl.yaml in project root)
- Secrets management (keyring integration, environment variables, --token flags)
- Example configurations for different environments

**docs/cli-reference.md:**
- Complete command reference with examples
- `mcpctl setup <server>` - Configure MCP server
- `mcpctl project create|list|delete|status` - Project management
- `mcpctl profile list|describe|apply` - Profile management
- `mcpctl claude add-mcp-project|remove-mcp-project` - Claude integration
- `mcpctl instance start|stop|restart|logs|status` - Instance lifecycle
- `mcpctl audit [--limit N] [--format json|table]` - Audit log queries
- `mcpctl config get|set` - Configuration management
- Global flags: --server, --format, --verbose, --quiet

**docs/architecture.md:**
- High-level system diagram (ASCII or Mermaid)
- Component descriptions: CLI, mcpd, local-proxy, database
- Data flow: Claude -> .mcp.json -> local-proxy -> mcpd -> MCP servers
- Security model: Token validation, audit logging, credential isolation
- Scalability: Stateless mcpd, PostgreSQL HA, horizontal scaling

**docs/local-llm-setup.md:**
- Ollama installation and configuration
- Model recommendations for filtering (llama3.2, qwen2.5)
- Gemini CLI setup as alternative
- vLLM for high-throughput deployments
- DeepSeek API configuration
- Performance tuning and benchmarks

**docs/mcp-servers/:**
- slack.md: Slack MCP setup, required scopes, profile examples
- jira.md: Jira Cloud/Server setup, API token creation
- github.md: GitHub token scopes, repository access
- terraform.md: Terraform docs MCP configuration
- Each includes: Prerequisites, Setup steps, Available profiles, Troubleshooting

**docs/deployment/:**
- docker-compose.md: Production docker-compose configuration
- kubernetes.md: Helm chart installation, values.yaml reference

### 18.4. Create SRE Runbooks and Network Topology Documentation

**Status:** pending  
**Dependencies:** 18.3  

Write operational runbooks for common SRE scenarios including restart procedures, credential rotation, scaling, and diagnostics, plus network topology documentation for enterprise deployments with proxy, firewall, and DNS considerations.

**Details:**

Create docs/operations/ directory:

**docs/operations/runbooks/:**

**restart-failed-instance.md:**
```markdown
# Runbook: Restart Failed MCP Instance

## Symptoms
- `mcpctl instance status <name>` shows 'error' or 'stopped'
- Audit logs show repeated connection failures
- Claude reports MCP tool unavailable

## Diagnosis
1. Check instance status: `mcpctl instance status <name> --verbose`
2. View recent logs: `mcpctl instance logs <name> --tail 100`
3. Check container health: `docker inspect mcpctl-<name> | jq '.[0].State'`

## Resolution Steps
1. Stop the instance: `mcpctl instance stop <name>`
2. Check for resource exhaustion: `docker stats --no-stream`
3. Restart: `mcpctl instance start <name>`
4. Verify health: `mcpctl instance status <name> --wait-healthy`
5. Test connectivity: `mcpctl instance test <name>`

## Escalation
- If repeated failures: Check network connectivity to external APIs
- If OOM: Increase container memory limits in profile configuration
```

**rotate-credentials.md:** Steps for rotating Slack tokens, Jira API keys, GitHub PATs without downtime

**scale-up.md:** Adding mcpd instances, database read replicas, load balancer configuration

**diagnose-connectivity.md:** Network troubleshooting between proxy, mcpd, and MCP servers

**backup-restore.md:** PostgreSQL backup procedures, disaster recovery

**security-incident.md:** Credential exposure response, audit log analysis, revocation procedures

**docs/operations/network-topology.md:**
```markdown
# Network Topology and Enterprise Deployment

## Architecture Diagram
[Mermaid diagram showing: Claude Desktop -> local-proxy (localhost) -> Corporate Proxy -> mcpd (internal network) -> MCP Servers (Slack API, Jira API, etc.)]

## Network Requirements

### Local Proxy (runs on developer machine)
- Listens on localhost:9229 (configurable)
- Outbound: HTTPS to mcpd server (configurable URL)
- No direct internet access required

### mcpd Server (internal deployment)
- Inbound: HTTPS from corporate network (developer machines)
- Outbound: HTTPS to MCP server APIs (Slack, Jira, GitHub)
- PostgreSQL: Port 5432 to database server

### Firewall Rules
| Source | Destination | Port | Protocol | Purpose |
|--------|-------------|------|----------|--------|
| Developer workstations | mcpd | 443 | HTTPS | API access |
| mcpd | PostgreSQL | 5432 | TCP | Database |
| mcpd | api.slack.com | 443 | HTTPS | Slack MCP |
| mcpd | *.atlassian.net | 443 | HTTPS | Jira MCP |
| mcpd | api.github.com | 443 | HTTPS | GitHub MCP |

### Proxy Configuration
- If corporate proxy required: Set HTTP_PROXY/HTTPS_PROXY for mcpd container
- No-proxy list: Database server, internal services
- SSL inspection: May require custom CA certificate injection

### DNS Configuration
- mcpd server should be resolvable: mcpd.internal.company.com
- Or use IP address in mcpctl config: `mcpctl config set-server https://10.0.0.50:443`

### TLS/Certificate Requirements
- mcpd should use valid TLS certificate (Let's Encrypt or internal CA)
- Certificate SANs should include all access hostnames
- For self-signed: Export CA and configure in mcpctl: `mcpctl config set-ca /path/to/ca.pem`
```

**docs/operations/troubleshooting-network.md:**
- Common issues: Connection refused, certificate errors, proxy authentication
- Diagnostic commands: `mcpctl doctor`, `mcpctl test-connection`
- tcpdump/Wireshark guidance for packet inspection
- Proxy debugging with curl equivalents

### 18.5. Implement Data Team Example Workflows with Automated Validation

**Status:** pending  
**Dependencies:** 18.1, 18.2, 18.3  

Create example workflow documentation for data analysts and engineers including weekly report generation, data pipeline monitoring, and documentation querying, with automated E2E tests validating each workflow works as documented.

**Details:**

Create docs/examples/ directory with workflow documentation:

**docs/examples/weekly-reports.md:**
```markdown
# Weekly Reports Workflow

## Use Case
Generate weekly team reports by aggregating Slack discussions and Jira ticket updates.

## Setup
```bash
# Install mcpctl (if not already installed)
npm install -g mcpctl

# Configure mcpd server connection
mcpctl config set-server http://your-nas:3000

# Setup MCP servers with appropriate tokens
mcpctl setup slack --token $SLACK_BOT_TOKEN
mcpctl setup jira --url https://company.atlassian.net --token $JIRA_API_TOKEN

# Create project with read-only profiles
mcpctl project create weekly_reports --profiles slack-team jira-myproject

# Add to Claude Desktop
mcpctl claude add-mcp-project weekly_reports
```

## Usage with Claude
In your Claude session, say:
> "Write me a weekly report for the security team. Get all Slack messages from #security-team mentioning incidents or vulnerabilities this week, and all Jira tickets I worked on with status changes."

The local proxy will:
1. Intercept the Slack API request
2. Use local LLM to identify relevant messages (filtering 1000s to ~50)
3. Return only pertinent data to Claude
4. Log the operation for audit compliance

## Expected Output
- Weekly summary with categorized Slack discussions
- Jira ticket status updates with time spent
- Action items extracted from conversations
```

**docs/examples/data-pipeline-monitoring.md:**
- Setup for monitoring Airflow/dbt pipelines via Slack alerts
- Integration with Jira for incident tracking
- Example Claude prompts for pipeline health checks

**docs/examples/documentation-querying.md:**
- Setup Terraform docs MCP for infrastructure documentation
- GitHub MCP for code documentation querying
- Example: "Find all S3 buckets with public access in our Terraform configs"

**tests/e2e/examples/ directory with automated validation:**

**tests/e2e/examples/weekly-reports.test.ts:**
```typescript
describe('Example Workflow: Weekly Reports', () => {
  test('follows documented setup steps', async () => {
    // Parse setup commands from docs/examples/weekly-reports.md
    const setupCommands = extractCodeBlocks('docs/examples/weekly-reports.md', 'bash');
    
    for (const cmd of setupCommands) {
      // Skip comments and variable-dependent commands for test
      if (cmd.startsWith('#') || cmd.includes('$SLACK')) continue;
      
      // Execute with test tokens
      const result = await exec(cmd.replace('$SLACK_BOT_TOKEN', 'test-token'));
      expect(result.exitCode).toBe(0);
    }
  });
  
  test('proxy filters messages as described', async () => {
    // Setup as documented
    await exec('mcpctl setup slack --non-interactive --token=test-token');
    await exec('mcpctl project create weekly_reports --profiles slack-read-only');
    
    // Simulate Claude request matching documented usage
    const response = await getProxyClient().callTool('slack_search_messages', {
      query: 'security incidents vulnerabilities',
      _context: 'Find security-related messages for weekly report'
    });
    
    // Verify filtering works as documented
    expect(response.messages.length).toBeLessThan(100); // Filtered from 1000+
    expect(response.messages.every(m => 
      m.text.toLowerCase().includes('security') || 
      m.text.toLowerCase().includes('incident') ||
      m.text.toLowerCase().includes('vulnerability')
    )).toBe(true);
  });
  
  test('audit log records operation', async () => {
    const auditResult = await exec('mcpctl audit --limit 1 --format json');
    const lastAudit = JSON.parse(auditResult.stdout)[0];
    expect(lastAudit.action).toBe('mcp_call');
    expect(lastAudit.resource).toContain('slack');
  });
});
```

**tests/e2e/examples/data-pipeline-monitoring.test.ts:** Similar validation for pipeline monitoring workflow

**tests/e2e/examples/documentation-querying.test.ts:** Validation for Terraform/GitHub docs workflow

Each test file:
1. Parses the corresponding markdown file for setup commands
2. Executes commands (with test credentials) to verify they work
3. Simulates the documented Claude usage pattern
4. Verifies expected outcomes match documentation claims