71 lines
2.3 KiB
Markdown
71 lines
2.3 KiB
Markdown
|
# Task ID: 32
|
||
|
|
|
||
|
|
**Title:** Implement smart context optimization
|
||
|
|
|
||
|
|
**Status:** done
|
||
|
|
|
||
|
|
**Dependencies:** 29 ✓
|
||
|
|
|
||
|
|
**Priority:** medium
|
||
|
|
|
||
|
|
**Description:** Add token counting and decision logic to intelligently skip LLM filtering when responses are small enough, and cache filtering decisions for repeated queries.
|
||
|
|
|
||
|
|
**Details:**
|
||
|
|
|
||
|
|
1. Create `src/mcplocal/src/llm/token-counter.ts`:
|
||
|
|
```typescript
|
||
|
|
export function estimateTokens(text: string): number {
|
||
|
|
// Simple estimation: ~4 chars per token for English
|
||
|
|
// More accurate: use tiktoken or similar library
|
||
|
|
return Math.ceil(text.length / 4);
|
||
|
|
}
|
||
|
|
```
|
||
|
|
2. Create `src/mcplocal/src/llm/filter-cache.ts`:
|
||
|
|
```typescript
|
||
|
|
export class FilterCache {
|
||
|
|
private cache: LRUCache<string, FilterDecision>;
|
||
|
|
|
||
|
|
shouldFilter(toolName: string, params: unknown, responseSize: number): boolean {
|
||
|
|
const key = this.computeKey(toolName, params);
|
||
|
|
const cached = this.cache.get(key);
|
||
|
|
if (cached) return cached.shouldFilter;
|
||
|
|
// No cache hit - use default threshold logic
|
||
|
|
return responseSize > this.tokenThreshold;
|
||
|
|
}
|
||
|
|
|
||
|
|
recordDecision(toolName: string, params: unknown, decision: FilterDecision): void {
|
||
|
|
const key = this.computeKey(toolName, params);
|
||
|
|
this.cache.set(key, decision);
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
3. Add configuration options:
|
||
|
|
- `tokenThreshold`: number (default 1000 tokens)
|
||
|
|
- `filterCacheSize`: number (default 1000 entries)
|
||
|
|
- `filterCacheTtl`: number (default 3600 seconds)
|
||
|
|
4. Integrate into LlmProcessor:
|
||
|
|
```typescript
|
||
|
|
async filterResponse(...) {
|
||
|
|
const tokens = estimateTokens(JSON.stringify(rawResponse));
|
||
|
|
if (tokens < this.config.tokenThreshold) {
|
||
|
|
// Not worth filtering - return as-is
|
||
|
|
return { filtered: false, response: rawResponse };
|
||
|
|
}
|
||
|
|
// Proceed with LLM filtering
|
||
|
|
}
|
||
|
|
```
|
||
|
|
5. Add metrics tracking:
|
||
|
|
- Total tokens processed
|
||
|
|
- Tokens saved by filtering
|
||
|
|
- Filter cache hit rate
|
||
|
|
- Average latency added by filtering
|
||
|
|
|
||
|
|
**Test Strategy:**
|
||
|
|
|
||
|
|
1. Unit test: Token estimation is reasonably accurate
|
||
|
|
2. Unit test: Cache correctly stores and retrieves decisions
|
||
|
|
3. Unit test: Threshold logic skips filtering for small responses
|
||
|
|
4. Unit test: Cache TTL expiration works correctly
|
||
|
|
5. Integration test: Metrics are recorded accurately
|
||
|
|
6. Performance test: Cache improves latency for repeated queries
|