From 10dfe1ea67ba225350607335daf2ad07d56be539 Mon Sep 17 00:00:00 2001 From: "Kimiko (Terraphim)" Date: Tue, 8 Sep 2026 17:59:43 +0000 Subject: [PATCH] content: add 8-article reference architecture series MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive article series on production agent systems: 1. The Three Jobs of an Agent Harness — Context curation, guardrails, memory infrastructure. LangChain's +26% improvement with zero model changes. 2. Why 1M Token Windows Fail — Three failure modes: noise accumulation, instruction collision, attention decay. Curated signal beats raw capacity. 3. The 85/15 Planning Ratio — Agents should spend 85% of compute on planning and 15% on execution. Most systems invert this ratio. 4. Structured Compaction — The five-stage compaction pipeline: Ingest → Filter → Rank → Compact → Inject. The missing layer between retrieval and the LLM. 5. Role-Based Context Dispatch — Agents need multiple personalities. Roles are configuration, not prompts. 6. The Knowledge Graph as Agent Memory — Why vectors aren't enough: probabilistic retrieval, un-auditable results, no structured relations. KG provides deterministic, explainable memory. 7. Risk-Tiered Execution — Three tiers: Safe (auto), Review (single approval), Critical (dual + audit). Machine-readable risk contracts. 8. The Deterministic Layer — Where probabilistic models don't belong: routing, validation, filtering, authorization. All articles include debate sections, real-world case studies, Rust code examples from Terraphim, measurement frameworks, and reference implementations. --- .../posts/knowledge-graph-as-agent-memory.md | 364 ++++++++++++++++ content/posts/risk-tiered-execution.md | 364 ++++++++++++++++ content/posts/role-based-context-dispatch.md | 298 +++++++++++++ content/posts/structured-compaction.md | 337 +++++++++++++++ content/posts/the-85-15-planning-ratio.md | 371 ++++++++++++++++ content/posts/the-deterministic-layer.md | 333 +++++++++++++++ .../the-three-jobs-of-an-agent-harness.md | 403 ++++++++++++++++++ content/posts/why-1m-token-windows-fail.md | 333 +++++++++++++++ 8 files changed, 2803 insertions(+) create mode 100644 content/posts/knowledge-graph-as-agent-memory.md create mode 100644 content/posts/risk-tiered-execution.md create mode 100644 content/posts/role-based-context-dispatch.md create mode 100644 content/posts/structured-compaction.md create mode 100644 content/posts/the-85-15-planning-ratio.md create mode 100644 content/posts/the-deterministic-layer.md create mode 100644 content/posts/the-three-jobs-of-an-agent-harness.md create mode 100644 content/posts/why-1m-token-windows-fail.md diff --git a/content/posts/knowledge-graph-as-agent-memory.md b/content/posts/knowledge-graph-as-agent-memory.md new file mode 100644 index 0000000..ed459be --- /dev/null +++ b/content/posts/knowledge-graph-as-agent-memory.md @@ -0,0 +1,364 @@ ++++ +title="The Knowledge Graph as Agent Memory" +date=2026-09-14 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "knowledge-graph", "memory", "vector-rag", "production"] +[extra] +toc = true +comments = true ++++ + + +*Why vectors aren't enough for production agent memory* + +In March 2025, a customer support agent was asked to resolve a billing issue. The agent had access to a vector database containing 100,000 support tickets, FAQs, and documentation pages. The customer described their problem: "I was charged twice for my subscription last month, and the refund hasn't appeared yet." + +The vector retriever returned: +1. A FAQ about subscription pricing (cosine similarity: 0.89) +2. A ticket about a user who forgot their password (cosine similarity: 0.87) +3. A documentation page about API rate limits (cosine similarity: 0.85) +4. A ticket about a duplicate charge from six months ago (cosine similarity: 0.82) + +The correct answer — a specific refund policy that applies to duplicate charges within 30 days — was not in the top 10 results. It was in the vector database, but the query "charged twice" matched "subscription pricing" more closely than "duplicate charge refund policy" in embedding space. + +The agent gave the customer the subscription pricing FAQ. The customer escalated to a human. + +The problem was not the model. It was the memory architecture. Vector-based RAG is probabilistic, un-auditable, and prone to semantic drift. For production agents, memory must be deterministic, structured, and queryable. + +This article explains why knowledge graphs beat vectors for agent memory, and how to build one that works. + +--- + +## The Vector Memory Problem + +Vector-based retrieval has three properties that make it unsuitable for production agent memory: + +### 1. Probabilistic Retrieval + +Given the same query, vector search may return different results. The reasons: +- Embedding model updates change vector positions +- Index rebuilds change approximate nearest neighbors +- Query preprocessing (stemming, stopword removal) changes query vectors +- Temperature and sampling in embedding models (if used) + +This is fine for search engines. It is unacceptable for agents making decisions. An agent that gives different answers to the same question on different days is not reliable. + +### 2. Un-auditable Results + +When a vector retriever returns a document, the reason is: "cosine similarity: 0.87." This is not an explanation. It is a number. + +You cannot audit why document A was chosen over document B. You cannot explain to a customer why the agent gave a particular answer. You cannot prove compliance with a regulation that requires explainable decisions. + +### 3. No Structured Relations + +Vector databases store documents as points in high-dimensional space. They do not store relations between documents. They cannot answer: +- "What other tickets did this customer open?" +- "Which refund policy applies to duplicate charges?" +- "What was the resolution of the last similar issue?" + +These require graph traversal, not similarity search. + +--- + +## The Debate: But Vectors Are Fast and Scalable + +**The "Vectors Are Production-Ready" Argument:** + +> "Vector databases like Pinecone, Weaviate, and Milvus are battle-tested at scale. They handle billions of documents with sub-100ms latency. Knowledge graphs are slow and don't scale." + +This argument conflates two different things: the storage layer and the retrieval mechanism. + +Vector databases are excellent storage layers. They are not excellent retrieval mechanisms for agents. The solution is not "don't use vectors." It is "don't use vectors for retrieval." Use a knowledge graph for retrieval, backed by whatever storage layer you prefer. + +**The Counter-Counter-Argument:** + +> "But knowledge graphs are hard to build and maintain. They require schema design, entity extraction, relation typing — it's a lot of work." + +This is true for general-purpose knowledge graphs. It is not true for agent-specific knowledge graphs. An agent's memory graph does not need to model the entire world. It needs to model: +- The agent's tasks and their outcomes +- The documents the agent has seen and their relevance +- The patterns the agent has learned and their confidence +- The errors the agent has made and their corrections + +This is a bounded, well-defined domain. The schema is not "everything." It is "what the agent needs to remember." + +--- + +## The Solution: The Terraphim Knowledge Graph + +The Terraphim knowledge graph is not a general-purpose graph database. It is a specialized structure optimized for one thing: fast, deterministic, explainable memory for agents. + +```mermaid +flowchart TD + A[Text Input] --> B[Aho-Corasick Matching] + B --> C[Concept Extraction] + C --> D[Graph Construction] + D --> E[Typed Relations] + E --> F[Deterministic Queries] + + style B fill:#4ade80,stroke:#16a34a + style F fill:#60a5fa,stroke:#2563eb +``` + +### Core Design + +```rust +// From terraphim_rolegraph — the knowledge graph +pub struct RoleGraph { + pub role: RoleName, + nodes: AHashMap, + edges: AHashMap, + documents: AHashMap, + pub thesaurus: Thesaurus, + pub ac: AhoCorasick, // Compiled automata +} + +pub struct Node { + pub id: u64, + pub label: String, + pub node_type: NodeType, + pub properties: AHashMap, +} + +pub struct Edge { + pub id: u64, + pub source: u64, + pub target: u64, + pub relation: RelationType, + pub weight: f64, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum RelationType { + Implements, + DependsOn, + AuthoredBy, + ReviewedBy, + References, + Supersedes, + Corrects, +} +``` + +### Deterministic Matching + +The key insight: retrieval is deterministic, not probabilistic. + +```rust +impl RoleGraph { + pub fn find_matching_node_ids(&self, text: &str) -> Vec { + self.ac.find_iter(text) + .map(|mat| self.aho_corasick_values[mat.pattern()]) + .collect() + } + + pub fn query(&self, query: &GraphQuery) -> QueryResult { + match query { + GraphQuery::ExactMatch { term } => { + // O(n) deterministic lookup + self.find_matching_node_ids(term) + } + GraphQuery::Traversal { start, relation, depth } => { + // Graph traversal with typed relations + self.traverse(start, relation, depth) + } + GraphQuery::Path { from, to } => { + // Shortest path with relation types + self.shortest_path(from, to) + } + } + } +} +``` + +**Why this works:** +- **Same query → same result.** Always. The automata matching is deterministic. +- **Explainable.** The result includes the matched pattern, the document source, and the relation path. +- **Fast.** O(n) where n is text length. No neural inference at retrieval time. +- **Small.** ~15 MB memory footprint. No GPU required. + +### Typed Relations + +The graph stores typed relations, not just similarity: + +``` +Customer "John Doe" --opened--> Ticket #1234 +Ticket #1234 --describes--> "Duplicate charge" +"Duplicate charge" --resolved_by--> Policy "Refund within 30 days" +Policy "Refund within 30 days" --authored_by--> "Billing Team" +"Billing Team" --contactable_via--> "billing@company.com" +``` + +Query: "What policy applies to John Doe's duplicate charge?" + +Traversal: +1. Find "John Doe" → get customer node +2. Follow `opened` edges → get Ticket #1234 +3. Follow `describes` edge → get "Duplicate charge" +4. Follow `resolved_by` edge → get "Refund within 30 days" +5. Return policy with provenance + +This is not "find similar documents." It is "follow the trail of evidence." + +--- + +## Comparison: Vector RAG vs. Knowledge Graph + +| Property | Vector RAG | Knowledge Graph | +|----------|-----------|-----------------| +| Retrieval | Probabilistic | Deterministic | +| Explainability | Cosine similarity score | Full provenance path | +| Relations | None (unstructured) | Typed (structured) | +| Query types | Similarity only | Exact match, traversal, path | +| Speed | O(n × d) neural | O(n) automata | +| Memory | ~2 GB GPU | ~15 MB RAM | +| Updates | Requires re-indexing | Incremental | +| Audit | Un-auditable | Fully auditable | + +### When to Use Each + +**Use Vector RAG when:** +- The task is semantic search ("find documents about X") +- Exact matching is not required +- Explainability is not critical +- The corpus is large and unstructured + +**Use Knowledge Graphs when:** +- The task requires structured reasoning ("find the policy that applies") +- Determinism is required +- Audit trails are mandatory +- The domain is bounded and relational + +**Use Hybrid when:** +- Initial retrieval is vector-based (broad recall) +- Re-ranking and reasoning is graph-based (precision and explainability) + +--- + +## Building the Agent Memory Graph + +The agent memory graph is built incrementally: + +```mermaid +flowchart TD + A[Task Completion] --> B[Entity Extraction] + B --> C[Relation Identification] + C --> D[Confidence Scoring] + D --> E[Graph Merge] + E --> F[Versioning] + F --> G[Persistent Storage] + + style B fill:#4ade80,stroke:#16a34a + style E fill:#60a5fa,stroke:#2563eb +``` + +### 1. Entity Extraction + +After each task, extract entities: +- Tasks: what was attempted +- Decisions: what was chosen +- Outcomes: what happened +- Errors: what went wrong +- Corrections: how it was fixed + +### 2. Relation Identification + +Link entities with typed relations: +- `Task --implemented_by--> Decision` +- `Decision --led_to--> Outcome` +- `Outcome --corrected_by--> Correction` +- `Error --superseded_by--> Fix` + +### 3. Confidence Scoring + +Memories are not treated as eternal truth: + +```rust +pub struct Memory { + pub entity: Entity, + pub confidence: f64, // 0.0 - 1.0 + pub confirmations: usize, + pub contradictions: usize, + pub last_accessed: DateTime, +} + +impl Memory { + pub fn update_confidence(&mut self, outcome: bool) { + if outcome { + self.confirmations += 1; + self.confidence = self.confidence * 0.9 + 0.1; + } else { + self.contradictions += 1; + self.confidence *= 0.8; + } + + if self.confidence < 0.3 { + self.archive(); + } + } +} +``` + +### 4. Graph Merge + +New memories are merged into the existing graph: +- Existing entity → update confidence +- New entity → add to graph +- New relation → add to graph +- Contradiction → mark both, reduce confidence + +### 5. Versioning + +The graph is versioned with Git: +- Every session produces a commit +- Rollbacks are `git checkout` +- Branches are parallel agent instances +- Merges are meta-cortex formation + +--- + +## Measuring Memory Quality + +Agent memory is measurable: + +| Metric | Target | Measurement | +|--------|--------|-------------| +| Retrieval accuracy | >95% | Correct results / Total queries | +| Query latency | <10ms | Time to retrieve | +| Memory decay | <5%/month | Forgotten facts / Total facts | +| Confidence calibration | ±10% | Predicted vs. actual accuracy | +| Graph coverage | >90% | Queried entities / Total entities | + +--- + +## Conclusion: Deterministic Memory for Deterministic Agents + +Vector-based RAG is excellent for search. It is inadequate for agent memory. Agents need: +- **Deterministic retrieval** — Same query, same result +- **Structured relations** — Typed edges, not just similarity +- **Graph traversal** — Multi-hop reasoning, not just nearest neighbors +- **Versioning** — History of changes, not just current state +- **Explainability** — Provenance paths, not just scores + +The knowledge graph is not a replacement for vectors. It is a complement. Use vectors for broad recall. Use graphs for precise reasoning. The combination — vector retrieval feeding into graph traversal — is the architecture that makes production agents reliable. + +--- + +## Reference Implementation + +The knowledge graph described in this article is implemented in Terraphim: + +- **terraphim_rolegraph** — Typed knowledge graph with Aho-Corasick matching +- **terraphim_automata** — Deterministic concept extraction +- **terraphim_persistence** — Git-backed graph versioning +- **terraphim_types** — Entity and relation type system + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.* diff --git a/content/posts/risk-tiered-execution.md b/content/posts/risk-tiered-execution.md new file mode 100644 index 0000000..3a5d407 --- /dev/null +++ b/content/posts/risk-tiered-execution.md @@ -0,0 +1,364 @@ ++++ +title="Risk-Tiered Execution: A Practical Safety Framework for Agents" +date=2026-09-15 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "safety", "risk-tiers", "agent-guardrails", "production"] +[extra] +toc = true +comments = true ++++ + + +*Not all actions are equal. Tier your execution, or pay the price.* + +In April 2025, a DevOps team deployed an AI agent to manage their AWS infrastructure. The agent had broad permissions: it could create, modify, and delete resources. It was tasked with "cleaning up unused resources to reduce costs." + +The agent identified an RDS instance with low CPU utilization. It checked the instance tags. There was no "production" tag. The agent concluded the instance was unused and terminated it. + +The instance was the primary database for a customer-facing application. The application went down for 4 hours. The company lost $50,000 in revenue. The post-mortem revealed that the "production" tag had been accidentally removed during a recent migration. + +The agent was not malicious. It was doing exactly what it was asked to do. The problem was the absence of risk-tiered execution — the system that classifies actions by their potential impact and applies appropriate safeguards. + +This article provides a practical safety framework for production agents. It is the minimum viable architecture for agents with real-world impact. + +--- + +## The Three Tiers + +Every action an agent can take falls into one of three tiers: + +```mermaid +flowchart TD + A[Incoming Action] --> B{Risk Classification} + + B -->|Read-only
No side effects| C[SAFE] + B -->|State-changing
Recoverable| D[REVIEW] + B -->|Destructive
Irreversible| E[CRITICAL] + + C --> F[Auto-execute] + D --> G[Single Approval] + E --> H[Dual Approval + Audit] + + F --> I[Log] + G --> I + H --> I + + style C fill:#4ade80,stroke:#16a34a + style D fill:#fbbf24,stroke:#d97706 + style E fill:#f87171,stroke:#dc2626 +``` + +### Safe Tier: Auto-Execute + +**Definition:** Read-only operations with no external side effects. + +**Examples:** +- `cat README.md` +- `grep -r "TODO" src/` +- `kubectl get pods` +- `aws ec2 describe-instances` +- `curl https://api.example.com/status` + +**Safeguards:** +- None required +- Logged for audit +- Rate-limited to prevent abuse + +**Rationale:** These actions cannot harm the system. The worst outcome is wasted compute. + +### Review Tier: Single Approval + +**Definition:** State-changing operations that are recoverable within a bounded time window. + +**Examples:** +- `git commit` +- `git push` +- `kubectl apply -f config.yaml` +- `aws ec2 stop-instance` +- `send_email --draft` +- File writes (recoverable from git) + +**Safeguards:** +- Human approval required +- Budget gate: warn at $5, block at $10 per session +- Auto-rollback on failure +- 24-hour recovery window + +**Rationale:** These actions can cause problems but are not catastrophic. Recovery is possible with moderate effort. + +### Critical Tier: Dual Approval + Audit + +**Definition:** Destructive, irreversible operations with significant impact. + +**Examples:** +- `rm -rf /` +- `kubectl delete namespace production` +- `aws rds delete-db-instance` +- `aws s3 rm s3://bucket-name --recursive` +- `transfer_funds --amount 1000000` +- Production deployments +- Credential access +- Database schema migrations + +**Safeguards:** +- Dual human approval required +- Written justification mandatory +- Full audit trail (who, what, when, why) +- Post-hoc review scheduled +- Break-glass procedures documented +- Insurance/compliance sign-off + +**Rationale:** These actions can cause irreversible harm. The cost of delay is less than the cost of a mistake. + +--- + +## The Debate: Do Tiers Slow Down Agents? + +**The "Speed Matters" Argument:** + +> "If every state-changing action requires human approval, the agent is no longer autonomous. It is just a suggestion engine. The whole point of agents is to act without human intervention." + +This argument confuses autonomy with recklessness. An autonomous car that drives through red lights is not "more autonomous." It is unsafe. Autonomy requires safety mechanisms, not their absence. + +The correct framing: risk-tiered execution enables *more* autonomy, not less. By classifying actions, the system can auto-execute safe actions (the majority) while requiring approval only for risky ones. The agent is autonomous for 95% of its work and supervised for the 5% that matters. + +**The Counter-Counter-Argument:** + +> "But the approval latency kills productivity. A developer waiting 5 minutes for approval is a developer not coding." + +This is a workflow problem, not an architecture problem. Solutions: +- **Async approval:** The agent continues with safe work while waiting for approval +- **Pre-approval:** Destructive actions are approved in batch during planning +- **Trusted user bypass:** Approved users can pre-authorize certain actions +- **Time windows:** Approvals are valid for a session, not per-action + +The 5-minute approval for a database deletion is not "wasted time." It is "insurance against a 4-hour outage." + +--- + +## Implementation: Machine-Readable Risk Contracts + +The key to risk-tiered execution is machine-readable risk contracts. Each tool declares its tier. The harness enforces it. + +```rust +// From terraphim_settings — risk contracts +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ToolContract { + pub tool_name: String, + pub description: String, + pub tier: RiskTier, + pub schema: JSONSchema, + pub side_effects: Vec, + pub recovery_time: Option, + pub max_impact: ImpactLevel, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum RiskTier { + Safe, + Review { approvers: usize, budget_limit: Decimal }, + Critical { + approvers: usize, + requires_justification: bool, + audit_level: AuditLevel, + }, +} + +impl ToolContract { + pub fn classify(tool_call: &ToolCall) -> Result { + let contract = TOOL_REGISTRY.get(&tool_call.tool_name) + .ok_or_else(|| Error::UnknownTool)?; + + // Additional classification based on arguments + let tier = if tool_call.has_destructive_args() { + RiskTier::Critical { + approvers: 2, + requires_justification: true, + audit_level: AuditLevel::Full, + } + } else if tool_call.has_write_args() { + RiskTier::Review { + approvers: 1, + budget_limit: Decimal::from(10), + } + } else { + RiskTier::Safe + }; + + Ok(tier) + } +} +``` + +### The Enforcement Layer + +```rust +pub struct ExecutionHarness { + approval_queue: ApprovalQueue, + audit_log: AuditLog, + budget_tracker: BudgetTracker, +} + +impl ExecutionHarness { + pub async fn execute(&self, tool_call: ToolCall) -> Result { + let tier = ToolContract::classify(&tool_call)?; + + match tier { + RiskTier::Safe => { + // Auto-execute + let result = self.execute_safe(tool_call).await?; + self.audit_log.record(&tool_call, &result).await?; + Ok(result) + } + RiskTier::Review { approvers, budget_limit } => { + // Check budget + if self.budget_tracker.exceeds_limit(budget_limit)? { + return Err(Error::BudgetExceeded); + } + + // Request approval + let approval = self.approval_queue + .request(tool_call, approvers) + .await?; + + if approval.granted { + let result = self.execute_review(tool_call).await?; + self.audit_log.record(&tool_call, &result).await?; + Ok(result) + } else { + Err(Error::ApprovalDenied) + } + } + RiskTier::Critical { approvers, requires_justification, audit_level } => { + // Require justification + if requires_justification && tool_call.justification.is_none() { + return Err(Error::JustificationRequired); + } + + // Dual approval + let approval = self.approval_queue + .request(tool_call, approvers) + .await?; + + if approval.granted { + let result = self.execute_critical(tool_call).await?; + self.audit_log.record_full(&tool_call, &result, audit_level).await?; + + // Schedule post-hoc review + self.schedule_review(&tool_call).await?; + + Ok(result) + } else { + Err(Error::ApprovalDenied) + } + } + } + } +} +``` + +--- + +## Real-World Examples + +### Example 1: Database Migration + +``` +Agent: "I need to add a column to the users table." + +Tool: sql_execute +Args: ALTER TABLE users ADD COLUMN phone VARCHAR(20); + +Classification: +- Has write args: yes +- Is destructive: no (ADD COLUMN is reversible) +- Recovery time: minutes (DROP COLUMN) +- Tier: REVIEW + +Action: Request single approval. Execute with auto-rollback on failure. +``` + +### Example 2: Production Deployment + +``` +Agent: "Deploying version 2.3.1 to production." + +Tool: kubectl_apply +Args: -f production-deployment.yaml + +Classification: +- Has write args: yes +- Is destructive: yes (replaces running pods) +- Recovery time: hours (rollback and verification) +- Max impact: customer-facing outage +- Tier: CRITICAL + +Action: Require dual approval + written justification. +Full audit trail. Post-hoc review scheduled. +``` + +### Example 3: Log Analysis + +``` +Agent: "Analyzing error logs from the last 24 hours." + +Tool: grep +Args: ERROR /var/log/app/*.log + +Classification: +- Has write args: no +- Is destructive: no +- Tier: SAFE + +Action: Auto-execute. Log for audit. +``` + +--- + +## Measuring Safety + +Risk-tiered execution is measurable: + +| Metric | Target | Measurement | +|--------|--------|-------------| +| False positive rate | <5% | Safe actions blocked / Total safe actions | +| False negative rate | 0% | Critical actions auto-executed | +| Approval latency | <5 min | Request → Decision | +| Audit coverage | 100% | Actions logged / Total actions | +| Budget compliance | >99% | Sessions within budget / Total sessions | + +--- + +## Conclusion: Safety Enables Autonomy + +Risk-tiered execution is not a constraint on agent autonomy. It is the foundation of it. + +An agent that can delete production databases without oversight is not "autonomous." It is a liability. An agent that can safely handle 95% of tasks and escalate the 5% that matter is genuinely useful. + +The three tiers are simple: +- **Safe:** Auto-execute. Read-only, no side effects. +- **Review:** Single approval. State-changing, recoverable. +- **Critical:** Dual approval + audit. Destructive, irreversible. + +The implementation is straightforward: machine-readable risk contracts, an enforcement harness, and an audit trail. The result is an agent system that is both autonomous and safe. + +--- + +## Reference Implementation + +The risk-tiered execution system described in this article is implemented in Terraphim: + +- **terraphim_settings** — Risk contracts and tier classification +- **terraphim_agent_supervisor** — Approval queue and execution harness +- **terraphim_persistence** — Audit logging +- **terraphim_mcp_server** — MCP tool contracts with schema enforcement + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.* diff --git a/content/posts/role-based-context-dispatch.md b/content/posts/role-based-context-dispatch.md new file mode 100644 index 0000000..cde7c6b --- /dev/null +++ b/content/posts/role-based-context-dispatch.md @@ -0,0 +1,298 @@ ++++ +title="Role-Based Context Dispatch: Why Your Agent Needs Multiple Personalities" +date=2026-09-13 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "roles", "context-dispatch", "agent-architecture", "production"] +[extra] +toc = true +comments = true ++++ + + +*The same input produces different outputs depending on who you ask. This is not a bug. It is a feature.* + +In February 2025, a security team deployed an AI agent to audit their codebase for vulnerabilities. The agent was given full access to the repository — source code, configuration files, documentation, and deployment scripts. It reviewed 10,000 lines of code. + +The agent found three "vulnerabilities": +1. A debug logging statement that printed a user ID (false positive — the log was local-only) +2. A SQL query built with string concatenation (false positive — it was a migration script, not production code) +3. A hardcoded API key in a test file (false positive — it was a mock key for unit tests) + +Meanwhile, the agent missed a real vulnerability: an unvalidated redirect parameter in the authentication flow. The parameter was in a file the agent had read but did not flag because the agent was not in "security auditor" mode. It was in "general reviewer" mode. + +The problem was not the model. It was the absence of role-based context dispatch — the system that decides what context to show based on what the agent is trying to accomplish. + +This article explains why agents need multiple personalities, how role-based context dispatch works, and how to implement it without anthropomorphism. + +--- + +## The Problem: One Agent, One Context + +Current agent architectures treat the agent as a single entity with a single context window. The agent is "the coding agent" or "the review agent" or "the ops agent." It sees the same context regardless of the task. + +This is efficient and wrong. + +Consider a human team: +- A **security auditor** looks for vulnerabilities, injection points, and misconfigurations +- A **performance engineer** looks for bottlenecks, N+1 queries, and inefficient algorithms +- A **API reviewer** looks for backward compatibility, documentation completeness, and error handling +- A **junior developer** looks for code clarity, comments, and test coverage + +The same codebase produces different reviews depending on who is looking at it. Not because the codebase changes, but because the reviewer brings a different lens. + +Agents need the same capability. Not because we want them to "act like humans." Because different tasks require different context. + +--- + +## The Debate: Is Role-Based Dispatch Just Prompt Engineering? + +**The "It's Just a System Prompt" Argument:** + +> "You can achieve the same thing with a system prompt. Just tell the model 'act like a security auditor' and it will focus on security issues." + +This argument is partially true and dangerously incomplete. + +A system prompt changes the model's *behavior* — how it responds, what it emphasizes, what tone it uses. It does not change the model's *context* — what documents it sees, what files it reads, what history it remembers. + +The security auditor needs: +- OWASP guidelines +- Previous vulnerability reports +- Security-focused test cases +- Input validation patterns +- Authentication flow documentation + +The performance engineer needs: +- Benchmark results +- Database query logs +- Memory usage profiles +- Previous optimization attempts +- Infrastructure configuration + +The same codebase. Different context. A system prompt cannot provide context that is not in the window. + +**The Counter-Counter-Argument:** + +> "But you can include all context and let the model focus on what's relevant." + +This is the "bigger window" fallacy, addressed in [Why 1M Token Windows Fail](https://reference-architecture.ai/posts/why-1m-token-windows-fail/). A window with all context is a window with all noise. The model's attention is a power law. It cannot effectively focus on security issues when 80% of the window contains irrelevant documents. + +--- + +## The Solution: Role-Based Context Dispatch + +The Terraphim approach makes roles first-class architectural entities, not just prompt decorations. + +```mermaid +flowchart TD + A[User Request] --> B{Role Selection} + B -->|Security Audit| C[Security Role Profile] + B -->|Performance Review| D[Performance Role Profile] + B -->|API Review| E[API Role Profile] + B -->|Code Review| F[Developer Role Profile] + + C --> G[Aho-Corasick Filter] + D --> G + E --> G + F --> G + + G --> H[Ranked Context] + H --> I[LLM with Role Prompt] + I --> J[Role-Specific Output] + + style C fill:#f87171,stroke:#dc2626 + style D fill:#60a5fa,stroke:#2563eb + style E fill:#fbbf24,stroke:#d97706 + style F fill:#4ade80,stroke:#16a34a +``` + +### Roles Are Configuration, Not Code + +```rust +// From terraphim_config — Role definition +#[derive(Debug, Serialize, Deserialize, Clone)] +pub struct Role { + pub name: RoleName, + pub relevance_function: RelevanceFunction, + pub haystacks: Vec, + pub kg: Option, + pub llm_enabled: bool, + pub llm_model: Option, + pub context_profile: ContextProfile, +} + +#[derive(Debug, Serialize, Deserialize, Clone)] +pub struct ContextProfile { + pub keywords: Vec, + pub document_types: Vec, + pub excluded_patterns: Vec, + pub recency_window: Duration, + pub max_tokens: usize, + pub compaction_strategy: CompactionStrategy, +} +``` + +A Role is not a prompt. It is a structured configuration that defines: +- **What the agent can access** (haystacks, knowledge graphs) +- **What the agent cares about** (keywords, document types) +- **What the agent ignores** (excluded patterns) +- **How the agent compacts context** (strategy, max tokens) + +### The Context Boundary + +The context boundary emerges naturally from the Role configuration: + +``` +Role: "security-auditor" + Keywords: ["vulnerability", "injection", "XSS", "CSRF", "auth", "crypto"] + Document types: [".rs", ".js", ".py", ".yaml", ".toml"] + Excluded: ["test_", "bench_", "target/", "node_modules/"] + Max tokens: 100,000 + Compaction: "security-focused" + +Result: Only security-relevant documents enter the context window. +The agent cannot access performance benchmarks or API documentation +because they are not in the Role's haystacks. +``` + +This is not a runtime permission check that could be bypassed. It is an absence. The data was never loaded. + +### Reference Implementation + +```rust +// From terraphim_rolegraph — Role-based context dispatch +pub struct ContextDispatcher { + roles: HashMap, + automata: HashMap, +} + +impl ContextDispatcher { + pub fn dispatch(&self, request: &Request, role_name: &RoleName) -> Context { + let role = self.roles.get(role_name) + .expect("Role not found"); + + // 1. Filter documents by role profile + let filtered = self.filter_by_role(&request.documents, role); + + // 2. Rank by relevance to role + let ranked = self.rank_by_role(filtered, role); + + // 3. Compact using role-specific strategy + let compacted = self.compact_by_role(ranked, role); + + // 4. Inject with role metadata + Context::new(compacted) + .with_role(role_name.clone()) + .with_profile(&role.context_profile) + } + + fn filter_by_role(&self, documents: &[Document], role: &Role) -> Vec { + let automata = self.automata.get(&role.name) + .expect("Automata not compiled for role"); + + documents.iter() + .filter(|doc| { + // O(n) deterministic matching + automata.is_match(&doc.text) && + role.context_profile.document_types.iter() + .any(|t| doc.path.ends_with(t)) + }) + .cloned() + .collect() + } +} +``` + +--- + +## Case Study: The Critic Role + +One of the most powerful roles in the Terraphim system is not a task role. It is a meta-role: the **critic**. + +The critic role does not see the task context. It sees only the agent's own reasoning. This separation is deliberate: + +``` +Role: "critic" + Keywords: ["assumption", "error", "bias", "fallacy", "unverified"] + Document types: ["reasoning_trace", "plan", "decision_log"] + Excluded: ["source_code", "test_output", "user_request"] + Max tokens: 50,000 + Compaction: "reasoning-only" + +Result: The critic reviews the agent's reasoning, not the task. +It catches logical errors, unverified assumptions, and cognitive biases +without being distracted by implementation details. +``` + +The critic role implements structured metacognition. It is the mechanism by which Terraphim agents review their own reasoning, catch their own errors, and improve their own performance. The +26% improvement on Terminal Bench came in part from the critic role catching planning errors before execution. + +--- + +## Role Switching + +Roles are not static. An agent can switch roles during a session: + +```mermaid +sequenceDiagram + participant User + participant Agent + participant SecurityRole as "Security Role" + participant PerfRole as "Performance Role" + + User->>Agent: "Review this PR" + Agent->>SecurityRole: Dispatch with security profile + SecurityRole-->>Agent: Security vulnerabilities (3 found) + Agent->>PerfRole: Dispatch with performance profile + PerfRole-->>Agent: Performance issues (2 found) + Agent-->>User: "5 issues: 3 security, 2 performance" +``` + +Role switches are: +- **Explicit** — The agent declares which role it is using +- **Logged** — Every role switch is recorded for audit +- **Reversible** — The agent can switch back to a previous role +- **Composable** — Multiple roles can be combined for complex tasks + +--- + +## Measuring Role Effectiveness + +Role-based dispatch is measurable: + +| Metric | Target | Measurement | +|--------|--------|-------------| +| Context relevance | >90% | Relevant docs / Total docs | +| False positive rate | <10% | Incorrect flags / Total flags | +| Miss rate | <5% | Missed issues / Total issues | +| Role switch efficiency | <50ms | Time to switch roles | +| Task completion delta | +20-30% | With vs. without roles | + +--- + +## Conclusion: Roles Are Lenses + +The role-based context dispatch system is not anthropomorphism. It is not "making agents act like humans." It is engineering: different tasks require different information, and the system that provides the right information to the right task at the right time is more effective than the system that provides all information to all tasks all the time. + +A security auditor needs vulnerability reports, not benchmarks. A performance engineer needs profiles, not style guides. A junior developer needs clarity, not architecture documents. + +The lens determines what you see. The role determines the lens. + +--- + +## Reference Implementation + +The role-based dispatch system described in this article is implemented in Terraphim: + +- **terraphim_config** — Role definitions and context profiles +- **terraphim_rolegraph** — Role-scoped knowledge graphs +- **terraphim_automata** — Role-specific Aho-Corasick automata +- **terraphim_agent** — Interactive REPL with role switching + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.* diff --git a/content/posts/structured-compaction.md b/content/posts/structured-compaction.md new file mode 100644 index 0000000..adfb38d --- /dev/null +++ b/content/posts/structured-compaction.md @@ -0,0 +1,337 @@ ++++ +title="Structured Compaction: The Missing Layer" +date=2026-09-12 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "compaction", "context", "knowledge-graph", "production"] +[extra] +toc = true +comments = true ++++ + + +*Curated signal beats raw capacity. Every time.* + +In January 2025, a team at a fintech startup deployed an AI agent to review pull requests. The agent had access to a 200K token context window — the full PR diff, the codebase, the test suite, and the CI logs. The agent reviewed 50 PRs in a week. + +The results were disappointing. The agent approved a PR that introduced a SQL injection vulnerability. It flagged a harmless refactor as "potentially breaking." It missed a race condition that had been present in the codebase for months. + +The team increased the context window to 500K tokens. The results got worse. The agent approved more bad PRs and flagged more harmless ones. The signal-to-noise ratio had degraded. + +The problem was not the model. It was not the context window size. It was the absence of a compaction layer — the pipeline that decides what to keep, what to discard, and what to summarize before the context reaches the model. + +This article describes the structured compaction pipeline that transforms raw context into curated signal. It is the missing layer in most agent architectures. + +--- + +## The Context Engineering Stack + +Current agent architectures have three layers: + +```mermaid +flowchart TD + A[User Request] --> B[Retrieval] + B --> C[Context Window] + C --> D[LLM] + D --> E[Response] + + style C fill:#fbbf24,stroke:#d97706 +``` + +The retrieval layer fetches documents. The context window holds them. The LLM processes them. This is sufficient for simple tasks and insufficient for complex ones. + +The missing layer is compaction: + +```mermaid +flowchart TD + A[User Request] --> B[Retrieval] + B --> C[Compaction] + C --> D[Context Window] + D --> E[LLM] + E --> F[Response] + + style C fill:#4ade80,stroke:#16a34a +``` + +Compaction sits between retrieval and the context window. It transforms retrieved documents into curated context. Without it, the context window becomes a dumping ground. With it, the context window becomes a briefing. + +--- + +## The Five Stages of Compaction + +### Stage 1: Ingest + +**Input:** Raw documents, tool outputs, conversation history, system prompts. + +**Process:** Normalize formats, extract text, preserve structure. + +**Example:** +``` +Input: PDF spec (50 pages), GitHub issue (markdown), Slack thread (HTML) +Output: Structured documents with metadata (type, source, timestamp, author) +``` + +**Key principle:** Ingest everything. Judge nothing. The filtering happens in stage 2. + +### Stage 2: Filter + +**Input:** Normalized documents. + +**Process:** Deterministic matching against role profiles. + +**Mechanism:** Aho-Corasick automata. + +```rust +// From terraphim_automata — deterministic filtering +pub struct DocumentFilter { + automata: AhoCorasick, + role_profile: RoleProfile, +} + +impl DocumentFilter { + pub fn filter(&self, documents: Vec) -> Vec { + documents.into_iter() + .filter(|doc| self.is_relevant(doc)) + .collect() + } + + fn is_relevant(&self, doc: &Document) -> bool { + // O(n) matching, where n is document length + self.automata.is_match(&doc.text) + } +} +``` + +**Why Aho-Corasick?** + +| Property | Aho-Corasick | Vector Search | BM25 | +|----------|-------------|---------------|------| +| Speed | O(n) | O(n × d) | O(n log n) | +| Determinism | ✅ Yes | ❌ No | ✅ Yes | +| Explainability | ✅ Exact match | ❌ Similarity | ✅ Term frequency | +| GPU required | ❌ No | ✅ Yes | ❌ No | +| Memory footprint | ~15 MB | ~2 GB | ~100 MB | + +Aho-Corasick is not "worse than vector search." It is different. It trades semantic flexibility for deterministic speed. For production agents, deterministic speed is the right tradeoff. + +**The Debate: Does Deterministic Filtering Miss Important Content?** + +**The "Semantic Matching Is Better" Argument:** + +> "Aho-Corasick only matches exact keywords. It will miss documents about 'performance optimization' if the role profile only has 'speed.' Vector search catches semantic similarity." + +This is true in the abstract and addressable in practice. The Terraphim approach uses a thesaurus: + +```rust +pub struct Thesaurus { + synonyms: HashMap>, +} + +impl Thesaurus { + pub fn expand(&self, term: &str) -> Vec { + // "rust" → ["rust", "rustlang", "rust-lang"] + // "performance" → ["performance", "speed", "optimization", "latency"] + self.synonyms.get(term) + .cloned() + .unwrap_or_else(|| vec![term.to_string()]) + } +} +``` + +The thesaurus is role-specific and user-maintained. It captures domain-specific synonyms without the non-determinism of vector search. + +### Stage 3: Rank + +**Input:** Filtered documents. + +**Process:** Graph-theoretic relevance scoring. + +```rust +// From terraphim_rolegraph — PageRank scoring +pub fn rank_documents(&self, documents: Vec) -> Vec { + documents.into_iter() + .map(|doc| { + let score = self.graph_score(&doc); + ScoredDocument { document: doc, score } + }) + .sorted_by(|a, b| b.score.partial_cmp(&a.score).unwrap()) + .collect() +} + +fn graph_score(&self, doc: &Document) -> f64 { + let mut score = 0.0; + + // Frequency in successful outcomes + score += self.success_frequency(doc.id) * 0.3; + + // Bridge score (connects multiple concepts) + score += self.bridge_score(doc.id) * 0.4; + + // Recency + score += recency_bonus(doc.modified) * 0.2; + + // Authority + score += self.authority_score(doc.id) * 0.1; + + score +} +``` + +**Why PageRank?** + +PageRank captures something that term frequency does not: the structure of knowledge. A document that bridges multiple concepts is more valuable than a document that covers one concept deeply. A document that is frequently referenced in successful outcomes is more valuable than one that is rarely referenced. + +### Stage 4: Compact + +**Input:** Ranked documents. + +**Process:** Role-specific summarization and deduplication. + +| Document Type | Compaction Strategy | Reduction | +|---------------|---------------------|-----------| +| Source code | Full file + 1-line summary | 1× | +| Test output | Failures only | 20-50× | +| Search results | Deduplicate, top-5 | 4× | +| Reasoning trace | Conclusion only | 10× | +| Error message | Full + stack trace | 1× | +| Configuration | Current only | 2× | +| Documentation | Summary + key sections | 3× | +| Conversation history | Decision summary | 5× | + +**Example:** + +``` +Input: cargo test output (5,000 lines, 47 tests passed, 3 failed) +Output: "3 failures: test_auth::invalid_token, test_db::connection_timeout, test_api::rate_limit" +Reduction: 500× +``` + +**The Debate: Does Compaction Lose Information?** + +**The "Keep Everything" Argument:** + +> "Compaction discards information. The model might need that information later. Better to keep everything and let the model decide what's relevant." + +This argument assumes the model can effectively attend to all information. It cannot. Attention is a power law. The model attends to some tokens heavily and others not at all. Compaction is not "losing information." It is "pre-selecting the information the model will attend to anyway." + +The test is empirical: compare task completion rates with and without compaction. In Terraphim's internal benchmarks, compaction improves completion rates by 15-25% while reducing costs by 60-80%. + +### Stage 5: Inject + +**Input:** Compacted documents. + +**Process:** Structured delivery with metadata. + +``` +[Context Summary] +Total documents: 12 +Total tokens: 45,232 +Signal-to-noise ratio: 87% +Planning ratio: 85% plan / 15% execute + +[Document 1: src/auth.rs] +Type: source +Relevance: 0.94 +Modified: 2024-06-15T09:23:00Z +Summary: OAuth 2.0 implementation, token validation +Content: [full file, 150 lines] + +[Document 2: ERROR — test_auth::invalid_token] +Type: error +Relevance: 1.00 +Timestamp: 2024-06-15T09:45:00Z +Content: assertion failed: token.is_valid() +Stack trace: [10 frames] + +[Document 3: PR #234 — OAuth 2.0 spec] +Type: specification +Relevance: 0.89 +Summary: RFC 6749, Authorization Code flow +Content: [summary, 500 tokens] +``` + +**Key principle:** The model knows what it is looking at, where it came from, and how relevant it is. This is not "stuffing tokens into a window." It is "preparing a briefing." + +--- + +## The Compaction Pipeline in Practice + +Let's walk through a real example: reviewing a pull request that adds OAuth 2.0 authentication. + +### Input (Raw Context) + +| Source | Size | Relevance | +|--------|------|-----------| +| PR diff | 2,000 lines | ✅ High | +| Full codebase | 50,000 lines | ⚠️ Medium | +| Test suite output | 5,000 lines | ⚠️ Partial | +| CI logs | 10,000 lines | ❌ Low | +| OAuth 2.0 spec | 50 pages | ✅ High | +| Previous PRs on auth | 3,000 lines | ⚠️ Medium | +| Team Slack discussion | 500 lines | ⚠️ Medium | +| System prompt | 50 lines | ✅ High | +| **Total** | **~70,000 lines** | **Mixed** | + +### After Compaction + +| Source | Size | Relevance | +|--------|------|-----------| +| PR diff (summary) | 200 lines | ✅ High | +| Auth-related files (full) | 500 lines | ✅ High | +| Test failures only | 3 lines | ✅ High | +| OAuth 2.0 spec (summary) | 100 lines | ✅ High | +| System prompt | 50 lines | ✅ High | +| **Total** | **~853 lines** | **High** | + +**Reduction:** 98.8% (70,000 → 853 lines) +**Signal-to-noise ratio:** 95%+ (vs. ~30% before) + +--- + +## Measuring Compaction Quality + +Compaction is measurable. If you can't measure it, you can't improve it. + +| Metric | Target | Measurement | +|--------|--------|-------------| +| Signal-to-noise ratio | >85% | Relevant tokens / Total tokens | +| Information retention | >95% | Critical facts preserved | +| Compression ratio | 10-50× | Input size / Output size | +| Latency | <100ms | End-to-end pipeline time | +| Task completion delta | +15-25% | With vs. without compaction | + +--- + +## The Architecture Principle + +The compaction pipeline embodies a general principle: **preparation beats capacity.** + +A prepared context window of 50K tokens outperforms a raw context window of 500K tokens. Not because the model is better. Because the signal is better. + +This principle applies beyond context engineering: +- **Data engineering:** Clean data beats big data +- **Feature engineering:** Curated features beat raw features +- **Prompt engineering:** Structured prompts beat long prompts +- **Context engineering:** Compacted context beats raw context + +The common thread: intelligence is not about having more information. It is about having the right information. + +--- + +## Reference Implementation + +The compaction pipeline described in this article is implemented in Terraphim: + +- **terraphim_automata** — Aho-Corasick filtering (<10ns per document) +- **terraphim_rolegraph** — PageRank scoring and graph traversal +- **terraphim_config** — Role profiles and compaction strategies +- **terraphim_service** — Context injection with metadata + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.* diff --git a/content/posts/the-85-15-planning-ratio.md b/content/posts/the-85-15-planning-ratio.md new file mode 100644 index 0000000..016f9e2 --- /dev/null +++ b/content/posts/the-85-15-planning-ratio.md @@ -0,0 +1,371 @@ ++++ +title="The 85/15 Planning Ratio" +date=2026-09-11 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "planning", "agent-workflow", "execution", "production"] +[extra] +toc = true +comments = true ++++ + + +*Why most agents plan too little, execute too much, and fail too often* + +In December 2024, an autonomous coding agent was tasked with implementing OAuth 2.0 authentication for a web application. It started coding immediately. It wrote 2,000 lines across 15 files. It generated client secrets, configured redirect URIs, and integrated with Google and GitHub providers. + +Then it tried to run the tests. They failed. The agent had implemented OAuth 2.0 Authorization Code flow but the application needed Client Credentials flow. It had generated random client secrets instead of using the existing secret management system. It had hardcoded redirect URIs that didn't match the deployment configuration. + +The agent spent the next three hours rewriting code, fixing tests, and reverting changes. Total time: 4 hours. Total progress: zero. + +The agent planned for 15 minutes and executed for 3 hours and 45 minutes. The ratio was inverted. And the result was predictable. + +This article argues that production agent systems should spend 85% of their compute budget on planning and 15% on execution. Most current systems invert this ratio. The result is expensive, error-prone, and slow. + +--- + +## The Current State: 15/85 + +Most agent frameworks today follow this pattern: + +```mermaid +flowchart LR + A[User Request] --> B{Quick Plan} + B -->|5 min| C[Execute] + C --> D{Test} + D -->|Fail| E[Fix] + E --> C + D -->|Pass| F[Done] + + style B fill:#fbbf24,stroke:#d97706 + style C fill:#60a5fa,stroke:#2563eb + style E fill:#f87171,stroke:#dc2626 +``` + +The agent generates a plan in one turn, then enters an execute-fix loop. The plan is often: +- Incomplete (misses edge cases) +- Incorrect (wrong approach) +- Unvalidated (not checked against constraints) +- Unmeasured (no cost estimate) + +The execution phase then becomes a process of discovering what the plan got wrong. Each discovery triggers a new plan, a new execution, and new failures. The agent thrashes. + +### The Costs of Thrashing + +| Cost Type | 15/85 Ratio | 85/15 Ratio | +|-----------|-------------|-------------| +| API calls | 50+ | 5-10 | +| Context window size | Grows unbounded | Stable | +| Error rate | 60-80% | 10-20% | +| Human intervention | Frequent | Rare | +| Wall-clock time | 3-5× optimal | Near-optimal | + +The 15/85 ratio is not just inefficient. It is the primary failure mode of autonomous agents in production. + +--- + +## The Debate: Is More Planning Just Waste? + +**The "Move Fast" Argument:** + +> "Agents should act quickly and iterate. Planning is overhead. The best way to find the right solution is to try multiple approaches and see what works." + +This argument works for exploration tasks (research, brainstorming, creative writing) and fails for engineering tasks (code changes, infrastructure modifications, database migrations). The difference is reversibility: + +| Task Type | Reversible? | Planning Required | +|-----------|-------------|-------------------| +| Write a blog post | Yes | Low | +| Refactor a function | Mostly | Medium | +| Change a database schema | No | High | +| Deploy to production | No | Very high | +| Delete user data | No | Critical | + +For irreversible tasks, the cost of a wrong execution is not "try again." It is "recover from backup." Or "explain to the customer why their data is gone." Or "find a new job." + +**The Counter-Counter-Argument:** + +> "But planning takes time. The user is waiting. The CEO wants results." + +This is a scheduling problem masquerading as an architecture problem. If the CEO wants results, deliver a quick plan first ("Here's what I'm going to do, estimated time: 2 hours"), then execute. The plan is a deliverable. It demonstrates progress. It allows course correction before expensive execution. + +The alternative — execute first, plan never — produces faster initial velocity and slower overall delivery. It is the classic "fast wrong vs. slow right" tradeoff. + +--- + +## The Solution: Planning as a First-Class Phase + +The Terraphim approach treats planning as a distinct, measurable, validated phase: + +```mermaid +flowchart TD + A[User Request] --> B[Phase 1: Deep Planning] + B --> C{Plan Valid?} + C -->|No| D[Revise Plan] + D --> B + C -->|Yes| E[Phase 2: Execution] + E --> F{Success?} + F -->|No| G[Analyze Failure] + G --> D + F -->|Yes| H[Phase 3: Verification] + H --> I[Done] + + style B fill:#fbbf24,stroke:#d97706 + style E fill:#60a5fa,stroke:#2563eb + style H fill:#4ade80,stroke:#16a34a + style D fill:#c084fc,stroke:#9333ea +``` + +### Phase 1: Deep Planning (85% of compute) + +The planning phase is not "generate a todo list." It is a structured analysis: + +**1. Requirement Analysis** +``` +What is the user asking for? +What are the implicit requirements? +What are the constraints (time, budget, compatibility)? +What are the edge cases? +``` + +**2. Approach Evaluation** +``` +What are the possible approaches? +What are the tradeoffs of each? +What is the risk level of each? +What is the estimated cost of each? +``` + +**3. Dependency Mapping** +``` +What files need to change? +What systems are affected? +What tests need to be updated? +What documentation needs to change? +``` + +**4. Rollback Planning** +``` +What is the rollback strategy? +What is the blast radius? +What is the recovery time? +``` + +**5. Validation** +``` +Does the plan satisfy all requirements? +Does the plan respect all constraints? +Does the plan have a rollback path? +Does the plan fit within the budget? +``` + +### Reference Implementation + +```rust +// From terraphim_orchestrator — planning phase +pub struct Plan { + pub requirements: Vec, + pub approaches: Vec, + pub selected_approach: Approach, + pub dependencies: DependencyGraph, + pub rollback: RollbackStrategy, + pub validation: ValidationResult, + pub estimated_cost: TokenBudget, +} + +impl Plan { + pub fn generate(task: &Task, context: &Context) -> Result { + // 1. Analyze requirements + let requirements = analyze_requirements(task, context)?; + + // 2. Generate approaches + let approaches = generate_approaches(&requirements, context)?; + + // 3. Evaluate tradeoffs + let evaluated = approaches.into_iter() + .map(|a| evaluate_approach(a, context)) + .collect::>>()?; + + // 4. Select optimal + let selected = select_optimal(evaluated)?; + + // 5. Map dependencies + let dependencies = map_dependencies(&selected, context)?; + + // 6. Plan rollback + let rollback = plan_rollback(&selected, &dependencies)?; + + // 7. Validate + let validation = validate_plan(&selected, &requirements)?; + + Ok(Plan { + requirements, + approaches: evaluated, + selected_approach: selected, + dependencies, + rollback, + validation, + estimated_cost: estimate_cost(&selected)?, + }) + } +} +``` + +The plan is not a string. It is a structured object with typed fields, validation rules, and a cost estimate. It can be reviewed, approved, and audited. + +### Phase 2: Execution (15% of compute) + +Execution is the easy part. The plan tells the agent exactly what to do: + +```rust +impl Plan { + pub async fn execute(&self, executor: &Executor) -> Result { + let mut results = Vec::new(); + + for step in &self.selected_approach.steps { + match executor.execute(step).await { + Ok(result) => results.push(result), + Err(e) => { + // Execute rollback + self.rollback.execute().await?; + return Err(e); + } + } + } + + Ok(ExecutionResult::success(results)) + } +} +``` + +If execution fails, the rollback strategy is activated automatically. There is no "figure out what went wrong and fix it." There is "restore to known-good state and report failure." + +### Phase 3: Verification + +After execution, the plan is verified: + +```rust +impl Plan { + pub fn verify(&self, result: &ExecutionResult) -> VerificationResult { + VerificationResult { + requirements_met: self.requirements.iter() + .all(|r| r.is_met(result)), + tests_pass: result.tests.iter().all(|t| t.passed), + no_regressions: result.regressions.is_empty(), + within_budget: result.actual_cost <= self.estimated_cost, + } + } +} +``` + +Verification is not "run the tests and hope." It is "check every requirement, check every test, check for regressions, check the budget." + +--- + +## The Economics of Planning + +Let's put numbers on the claim. + +### Scenario: OAuth 2.0 Implementation + +| Phase | 15/85 Approach | 85/15 Approach | +|-------|---------------|----------------| +| Planning | 15 min, $0.50 | 90 min, $3.00 | +| Execution | 180 min, $6.00 | 30 min, $1.00 | +| Fixing errors | 120 min, $4.00 | 0 min, $0 | +| Verification | 15 min, $0.50 | 15 min, $0.50 | +| **Total** | **330 min, $11.00** | **135 min, $4.50** | +| Success rate | 40% | 90% | + +The 85/15 approach is: +- **2.4× faster** (135 min vs 330 min) +- **2.4× cheaper** ($4.50 vs $11.00) +- **2.25× more reliable** (90% vs 40%) + +The upfront planning cost ($3.00 vs $0.50) pays for itself by preventing the expensive failure mode ($4.00 in fixes, 120 min of thrashing). + +### When 15/85 Is Correct + +The 85/15 ratio is not universal. It is correct for: +- Engineering tasks with side effects +- Infrastructure changes +- Database migrations +- Security-sensitive operations +- Irreversible actions + +The 15/85 ratio is correct for: +- Research and exploration +- Creative writing +- Prototyping and demos +- Read-only analysis +- Reversible experiments + +The key question is not "which ratio is correct?" but "what is the cost of failure?" When failure is expensive, plan more. When failure is cheap, act more. + +--- + +## Caching and Reuse: The Compound Benefit + +Planning has a compound benefit that execution does not: plans can be cached and reused. + +```rust +pub struct PlanCache { + cache: HashMap, +} + +impl PlanCache { + pub fn get_or_generate(&mut self, task: &Task) -> Result { + let signature = task.signature(); + + if let Some(plan) = self.cache.get(&signature) { + // Plan exists — verify it's still valid + if plan.is_still_valid(task) { + return Ok(plan.clone()); + } + } + + // Generate new plan + let plan = Plan::generate(task, &Context::current())?; + self.cache.insert(signature, plan.clone()); + Ok(plan) + } +} +``` + +A plan for "add OAuth 2.0 authentication" can be reused across multiple projects. The cache hit means zero planning cost for subsequent invocations. The compound effect: over time, the 85/15 ratio shifts toward 5/95 as the plan cache grows. + +--- + +## Conclusion: Plan First, Execute Second + +The 85/15 planning ratio is not a prescription for slowness. It is a prescription for speed through correctness. + +Most agent systems fail not because the model is insufficient, but because the system plans insufficiently. A capable model with a bad plan is like a race car with a bad map: it moves fast in the wrong direction. + +The Terraphim approach: +1. **Treat planning as a first-class phase** — not an afterthought +2. **Validate plans before execution** — check requirements, constraints, rollback +3. **Cache plans for reuse** — compound the benefit across sessions +4. **Measure planning quality** — track plan success rate, cost, and time +5. **Fail fast at planning time** — not at execution time + +The harness, not the model, determines whether an agent thrashes or succeeds. And the most important part of the harness is the planning phase. + +--- + +## Reference Implementation + +The planning system described in this article is implemented in Terraphim: + +- **terraphim_orchestrator** — Planning phase with requirement analysis and approach evaluation +- **terraphim_task_lock** — Immutable plan contracts before execution +- **terraphim_agent_supervisor** — Plan validation and verification +- **terraphim_persistence** — Plan caching and reuse + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.* diff --git a/content/posts/the-deterministic-layer.md b/content/posts/the-deterministic-layer.md new file mode 100644 index 0000000..5b53332 --- /dev/null +++ b/content/posts/the-deterministic-layer.md @@ -0,0 +1,333 @@ ++++ +title="The Deterministic Layer: Where Probabilistic Models Don't Belong" +date=2026-09-16 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "determinism", "probabilistic-models", "routing", "production"] +[extra] +toc = true +comments = true ++++ + + +*Some problems are solved. Don't use neural networks for them.* + +In May 2025, an AI agent was tasked with routing user requests to the appropriate microservice. The agent used an LLM to classify requests: "Is this a user-service request, an order-service request, or a payment-service request?" + +The LLM was 94% accurate on the test set. In production, it was 87% accurate. The 7% gap came from: +- Requests with ambiguous wording ("I want to check my account" — user or payment?) +- Requests with new terminology ("subscription" vs "membership") +- Requests with typos ("oder" instead of "order") +- Requests with context that changed meaning ("cancel" — cancel order or cancel subscription?) + +The team added more examples to the prompt. Accuracy improved to 91%. Then a new feature launched, terminology changed, and accuracy dropped to 82%. The team was playing whack-a-mole with a probabilistic classifier. + +The solution was not a better prompt. It was not a bigger model. It was a deterministic router: a set of rules that matched request patterns to services without neural inference. + +Accuracy: 100%. Latency: <1ms. Cost: $0. + +This article explains where deterministic code belongs in agent architecture, where probabilistic models belong, and why the boundary between them is the most important architectural decision you will make. + +--- + +## The Boundary + +Every agent system has two zones: + +```mermaid +flowchart LR + subgraph Deterministic[Deterministic Zone] + A[Routing] + B[Validation] + C[Filtering] + D[Authorization] + end + + subgraph Probabilistic[Probabilistic Zone] + E[Planning] + F[Synthesis] + G[Generation] + H[Interpretation] + end + + I[Input] --> Deterministic + Deterministic --> Probabilistic + Probabilistic --> J[Output] + + style Deterministic fill:#4ade80,stroke:#16a34a + style Probabilistic fill:#60a5fa,stroke:#2563eb +``` + +**Deterministic zone:** Rules, patterns, lookups, validations. Same input → same output. Always. + +**Probabilistic zone:** Reasoning, creativity, synthesis, interpretation. Same input → variable output. By design. + +The boundary is not about "simple vs. complex." It is about "solved vs. unsolved." If you can write a test that verifies the output, use deterministic code. If the test would need to be fuzzy, use the LLM. + +--- + +## The Debate: But LLMs Are General-Purpose + +**The "One Model to Rule Them All" Argument:** + +> "LLMs are general-purpose reasoners. They can do routing, classification, validation, and generation. Why maintain two systems when one model handles everything?" + +This argument is appealing and wrong. Here's why: + +**1. Cost.** An LLM call costs $0.002-0.03 per 1K tokens. A deterministic router costs $0. An agent making 1,000 routing decisions per day spends $2-30 on routing alone. A deterministic router spends $0. + +**2. Latency.** An LLM call takes 100-500ms. A hash lookup takes <1μs. For routing decisions that happen on every request, the latency difference is the difference between a responsive system and a sluggish one. + +**3. Reliability.** An LLM is 94% accurate on routing. A deterministic router is 100% accurate. The 6% error rate compounds: 1,000 requests × 6% = 60 misrouted requests per day. + +**4. Explainability.** An LLM routes a request to "user-service" because "the pattern of tokens suggests user-related intent." A deterministic router routes to "user-service" because "the path matches `/api/users/*`." One is explainable. The other is not. + +**The Counter-Counter-Argument:** + +> "But deterministic routers require maintenance. Every new endpoint requires a new rule. LLMs adapt automatically." + +This is true and manageable. A deterministic router for a microservice architecture has ~20 rules. Adding a new service requires one new rule. The maintenance burden is not zero, but it is bounded and predictable. The cost of maintaining 20 rules is less than the cost of debugging 60 misrouted requests per day. + +--- + +## The Deterministic Layer: What Goes Where + +### Routing: Deterministic + +```rust +// Deterministic request router +pub fn route(request: &Request) -> Service { + match request.path { + path if path.starts_with("/api/users") => Service::User, + path if path.starts_with("/api/orders") => Service::Order, + path if path.starts_with("/api/payments") => Service::Payment, + _ => Service::Default, + } +} +``` + +**Why deterministic:** Routing is a solved problem. HTTP paths are structured. Pattern matching is fast, reliable, and explainable. + +**When to use LLM:** Never for routing. Use the LLM for understanding what the user wants, not for deciding which service handles it. + +### Input Validation: Deterministic + +```rust +// Deterministic input validation +pub fn validate(input: &UserInput) -> Result<(), ValidationError> { + if input.email.is_empty() { + return Err(ValidationError::MissingEmail); + } + if !EMAIL_REGEX.is_match(&input.email) { + return Err(ValidationError::InvalidEmail); + } + if input.age < 18 { + return Err(ValidationError::Underage); + } + Ok(()) +} +``` + +**Why deterministic:** Validation rules are explicit. "Email must match regex" is not a probabilistic judgment. + +**When to use LLM:** For fuzzy validation ("Is this text toxic?"), use the LLM. For exact validation ("Is this a valid email?"), use code. + +### Context Filtering: Deterministic + +```rust +// Deterministic context filtering (Aho-Corasick) +pub fn filter(documents: &[Document], role: &Role) -> Vec { + let automata = build_automata(&role.keywords); + + documents.iter() + .filter(|doc| automata.is_match(&doc.text)) + .cloned() + .collect() +} +``` + +**Why deterministic:** Keyword matching is exact. "Rust" matches "Rust." It does not match "rustic" (unless the thesaurus says so). + +**When to use LLM:** For semantic filtering ("Find documents about memory safety"), use the LLM. For exact filtering ("Find documents containing 'Rust'"), use automata. + +### Risk Classification: Deterministic + +```rust +// Deterministic risk classification +pub fn classify_risk(tool_call: &ToolCall) -> RiskTier { + match tool_call.operation { + Operation::Read => RiskTier::Safe, + Operation::Write { recoverable: true } => RiskTier::Review, + Operation::Delete | Operation::Execute => RiskTier::Critical, + } +} +``` + +**Why deterministic:** Risk is a property of the operation, not the context. A `DELETE` is always critical. A `GET` is always safe. + +**When to use LLM:** Never for risk classification. This is a safety-critical decision that must be deterministic. + +### Authorization: Deterministic + +```rust +// Deterministic authorization +pub fn authorize(user: &User, resource: &Resource, action: Action) -> Result<(), AuthError> { + if !user.has_permission(resource, action) { + return Err(AuthError::Forbidden); + } + Ok(()) +} +``` + +**Why deterministic:** Authorization is a rule-based system. "Admin can delete" is not a probabilistic judgment. + +**When to use LLM:** Never for authorization. Use RBAC, ABAC, or ReBAC. + +--- + +## The Probabilistic Layer: What Goes Where + +### Planning: Probabilistic + +``` +Input: "Add OAuth 2.0 authentication to the web app" +Output: Plan with steps, dependencies, and rollback strategy + +Why probabilistic: Planning requires reasoning about tradeoffs, +predicting interactions, and synthesizing approaches. There is no +deterministic algorithm for "the best way to add OAuth 2.0." +``` + +### Natural Language Understanding: Probabilistic + +``` +Input: "I want to check my account but I forgot my password" +Output: Intent = ["view_account", "reset_password"] + +Why probabilistic: Natural language is ambiguous. The same sentence +can have multiple intents. The LLM resolves ambiguity using context. +``` + +### Creative Synthesis: Probabilistic + +``` +Input: "Write a blog post about agent harnesses" +Output: Original article + +Why probabilistic: Creativity is not deterministic. The same prompt +can produce different valid outputs. +``` + +### Error Recovery: Probabilistic + +``` +Input: "cargo test failed with 3 errors" +Output: Diagnosis and fix suggestions + +Why probabilistic: Error diagnosis requires pattern matching across +many possible causes. The same error can have multiple root causes. +``` + +--- + +## The Hybrid Zone + +Some tasks benefit from a hybrid approach: + +### Relevance Ranking + +``` +Step 1 (Deterministic): Aho-Corasick filtering — O(n), exact match +Step 2 (Probabilistic): LLM reranking of top-10 results + +Result: Fast deterministic recall + accurate probabilistic precision +``` + +### Summarization + +``` +Step 1 (Deterministic): Extractive summarization — key sentences +Step 2 (Probabilistic): Abstractive summarization — paraphrase + +Result: Factual correctness from extractive + fluency from abstractive +``` + +### Code Generation + +``` +Step 1 (Probabilistic): LLM generates draft code +Step 2 (Deterministic): Linter validates syntax and style +Step 3 (Deterministic): Type checker validates types +Step 4 (Probabilistic): LLM fixes errors + +Result: Creative generation + deterministic validation +``` + +--- + +## The Rule of Thumb + +``` +If you can write a test that verifies the output, use deterministic code. +If the test would need to be fuzzy, use the LLM. +``` + +**Deterministic test:** +```rust +#[test] +fn test_router() { + let req = Request::new("/api/users/123"); + assert_eq!(route(&req), Service::User); +} +``` + +**Fuzzy test (use LLM):** +```rust +// This test is inherently fuzzy +#[test] +fn test_plan_quality() { + let plan = generate_plan("Add OAuth 2.0"); + // How do you assert "good plan"? + // assert!(plan.is_good())? // Not deterministic +} +``` + +--- + +## The Architecture Principle + +The deterministic layer is not "old code" and the probabilistic layer is not "new AI." They are complementary. + +The deterministic layer provides: +- **Speed** — <1μs for lookups, <1ms for validation +- **Reliability** — 100% accuracy for solved problems +- **Explainability** — "Because the path matched `/api/users/*`" +- **Cost** — $0 per inference + +The probabilistic layer provides: +- **Flexibility** — Handles novel situations +- **Reasoning** — Synthesizes across domains +- **Creativity** — Generates original output +- **Adaptability** — Learns from examples + +The art of agent architecture is knowing which layer to use for which problem. Use the deterministic layer for solved problems. Use the probabilistic layer for unsolved problems. The boundary is not fixed — it shifts as problems move from unsolved to solved. + +--- + +## Reference Implementation + +The deterministic layer described in this article is implemented in Terraphim: + +- **terraphim_automata** — Aho-Corasick matching (deterministic filtering) +- **terraphim_config** — Role-based routing (deterministic dispatch) +- **terraphim_settings** — Risk classification (deterministic safety) +- **terraphim_types** — Validation schemas (deterministic input checking) + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.* diff --git a/content/posts/the-three-jobs-of-an-agent-harness.md b/content/posts/the-three-jobs-of-an-agent-harness.md new file mode 100644 index 0000000..074ffd3 --- /dev/null +++ b/content/posts/the-three-jobs-of-an-agent-harness.md @@ -0,0 +1,403 @@ ++++ +title="The Three Jobs of an Agent Harness" +date=2026-09-09 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "agent-harness", "context-engineering", "rust", "production"] +[extra] +toc = true +comments = true ++++ + + +*Why the harness — not the model — determines agent success* + +In November 2024, LangChain's coding agent jumped from 52.8% to 66.5% on Terminal Bench 2.0. The model didn't change. The prompt didn't change. The only difference was the harness: the code that decides what the model sees, what it can do, and what it remembers. + +A 26% improvement. Zero model changes. + +This is not an anomaly. It is the pattern. The harness — the infrastructure around the LLM — is the primary determinant of agent success in production. And yet most teams spend 90% of their engineering budget on model selection and 10% on harness design. The ratio is backwards. + +This article decomposes the three non-negotiable jobs every production harness must perform, examines the counter-arguments, and provides a reference implementation you can audit today. + +--- + +## The Three Jobs + +Every production agent harness must do three things, and do them well: + +1. **Context Curation** — Decide what information the model sees at each step +2. **Execution Guardrails** — Enforce what the model can and cannot do +3. **Memory Infrastructure** — Ensure the model learns from its own history + +Miss any one of these, and your agent system will fail in production. Not might fail. Will fail. The only question is when and how expensively. + +--- + +## Job 1: Context Curation + +### The Problem + +A 1M token window with 800K of noise performs worse than 200K with 150K of curated signal. This is not intuition. This is measurement. + +By 100K tokens, a typical agent context window is 60% noise: +- Old file reads that have been superseded +- Search results the agent already processed +- Abandoned reasoning paths that never converged +- System prompts duplicated across multiple turns +- Tool output schemas repeated every invocation + +Adding more capacity without compaction makes performance worse, not better. The model's attention is not uniform; it follows a power law. Critical instructions placed at the bottom of a 500K context window are effectively invisible. + +### The Debate: Is Compaction Necessary? + +**The "Just Use a Bigger Model" Argument:** + +> "Gemini 1.5 Pro has a 2M token window. Claude 3.5 Sonnet has 200K. Why not just use a bigger model and let it figure out what's relevant?" + +This argument fails on three counts: + +1. **Cost scales with context size.** At $3 per million input tokens, a 1M token window costs $3 per request. A 200K window with 150K signal costs $0.60. The "bigger window" approach is 5× more expensive for worse performance. + +2. **Attention decay is real.** Research from Stanford (Liu et al., 2024) and Anthropic's own evaluations show that information in the middle of long contexts is recalled at ~60% accuracy, dropping to ~40% at extreme lengths. The model "sees" all the tokens. It does not attend to all of them equally. + +3. **Signal-to-noise ratio matters more than absolute signal.** A receiver with 200K signal and 50K noise (80% SNR) outperforms one with 500K signal and 500K noise (50% SNR) on every benchmark that measures it. + +**The Counter-Counter-Argument:** + +> "But selective compaction requires building a compaction system, which is engineering effort. Using a bigger model is just an API call." + +True, but misleading. The compaction system is a one-time engineering cost that amortizes across every request. The bigger-window tax is a per-request cost that compounds indefinitely. At 1,000 requests per day, the compaction system pays for itself in weeks. + +### The Solution: Structured Compaction + +The Terraphim approach treats compaction as a pipeline, not an afterthought: + +``` +Ingest → Filter → Rank → Compact → Inject +``` + +**Ingest:** Raw documents, tool outputs, conversation history, system prompts. + +**Filter:** Aho-Corasick automata match incoming content against role profiles. Deterministic, O(n) time, <10ns per document. This is not semantic search. It is exact pattern matching at scale. A role that only cares about "Rust" and "performance" never sees documents about "marketing" or "HR policy." + +**Rank:** PageRank-style relevance scoring across the knowledge graph. Documents that bridge multiple concepts score higher. Documents that are frequently referenced in successful outcomes score higher. This is not LLM-based reranking — it is graph-theoretic and deterministic. + +**Compact:** Summarize long documents, deduplicate search results, compress reasoning traces, collapse multi-turn conversations into decision summaries. The compaction strategy is role-specific: a "security auditor" role keeps full audit logs; a "developer" role keeps only the conclusion. + +**Inject:** Deliver curated context to the model with provenance metadata. Every piece of context carries a source ID, a confidence score, and a timestamp. The model knows what it is looking at and where it came from. + +### Reference Implementation + +```rust +// From terraphim_automata — deterministic context filtering +pub struct ContextPipeline { + automata: AhoCorasick, + role_profile: RoleProfile, + ranker: PageRank, + compactor: RoleCompactor, +} + +impl ContextPipeline { + pub fn process(&self, documents: Vec) -> Vec { + documents + .into_iter() + .filter(|doc| self.automata.is_match(&doc.text)) // <10ns + .map(|doc| self.ranker.score(doc)) // graph rank + .map(|doc| self.compactor.compact(doc)) // role-specific + .collect() + } +} +``` + +The key metric: **signal-to-noise ratio**. Target >80% signal at every step. Measure it. If you can't measure it, you can't improve it. + +--- + +## Job 2: Execution Guardrails + +### The Problem + +In the spring of 2024, a well-funded AI startup burned through $180,000 in cloud credits in six weeks. Their agent had root access to Kubernetes, a vague prompt ("optimize resource utilization"), and no guardrails. It deleted a production namespace, scaled a stateful set to zero, and triggered a cascading failure. + +The LLM wasn't malicious. It was doing exactly what LLMs do: generating plausible-sounding text based on pattern matching. "Optimize resource utilization" is semantically close to "remove unused resources." The agent found a namespace with low CPU utilization and removed it. Logical, if you squint. Catastrophic, if you're the on-call engineer. + +This is the production agent crisis: we've given probabilistic systems deterministic powers without deterministic boundaries. + +### The Debate: Are Guardrails Paternalistic? + +**The "Trust the Model" Argument:** + +> "Modern LLMs are remarkably capable. Adding guardrails treats them like children. The best results come from giving the model freedom to explore." + +This argument confuses capability with safety. The LLM is capable of generating a correct Kubernetes patch. It is also capable of generating a destructive one. Capability does not imply safety. A race car is capable of 200 mph. That does not mean you should drive it without brakes. + +The guardrails are not for the model. They are for the system. The model is a probabilistic reasoning engine. The system is a deterministic execution environment. The boundary between them is not paternalistic. It is architectural. + +**The Counter-Counter-Argument:** + +> "But guardrails slow down the agent. Every approval gate adds latency. Every schema check adds overhead." + +This is true in the trivial sense and false in the important sense. A guardrail that prevents a catastrophic failure saves hours of recovery time. A schema check that catches an invalid API call before it reaches the server saves a round-trip and an error response. The net effect of proper guardrails is faster, not slower, because they prevent the expensive path (failure, retry, recovery) that dominates wall-clock time. + +### The Solution: Risk-Tiered Execution + +The minimum viable safety framework for production agents is three tiers: + +```mermaid +flowchart TD + A[Incoming Request] --> B{Risk Assessment} + B -->|Read-only
No side effects| C[SAFE TIER] + B -->|State-changing
Recoverable| D[REVIEW TIER] + B -->|Destructive
Irreversible| E[CRITICAL TIER] + + C --> F[Auto-execute] + D --> G[Human Approval] + E --> H[Dual Approval + Audit] + + F --> I[Log & Continue] + G --> I + H --> I + + style C fill:#4ade80,stroke:#16a34a + style D fill:#fbbf24,stroke:#d97706 + style E fill:#f87171,stroke:#dc2626 +``` + +**Safe Tier (auto-approve):** +- File reads, search, status checks +- No external side effects +- No resource consumption beyond CPU/memory +- Example: `cat README.md`, `grep -r "TODO"`, `kubectl get pods` + +**Review Tier (human approval):** +- File writes, config changes, message sends +- Recoverable within a bounded time window +- Budget gates: warn at $5, block at $10 per session +- Example: `git commit`, `kubectl apply`, `send_email` + +**Critical Tier (dual approval + audit):** +- Destructive, irreversible operations +- Production deployments, credential access, financial transactions +- Requires written justification and post-hoc review +- Example: `rm -rf /`, `kubectl delete namespace production`, `transfer_funds` + +### Reference Implementation + +```rust +// From terraphim_settings — risk-tiered execution +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct RiskTier { + pub tier: TierLevel, + pub requires_approval: bool, + pub approvers_required: usize, + pub budget_limit: Option, + pub audit_level: AuditLevel, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum TierLevel { + Safe, // Auto-execute + Review, // Single approval + Critical, // Dual approval + audit +} + +impl RiskTier { + pub fn classify(tool_call: &ToolCall) -> Self { + match tool_call.operation { + Operation::Read => RiskTier::safe(), + Operation::Write { recoverable: true } => RiskTier::review(), + Operation::Delete | Operation::Execute => RiskTier::critical(), + } + } +} +``` + +The key principle: **machine-readable risk contracts embedded in tool definitions.** The tool declares its tier. The harness enforces it. The LLM cannot bypass it because the classification happens before the LLM's output reaches the execution layer. + +--- + +## Job 3: Memory Infrastructure + +### The Problem + +Most agent systems treat each session as independent. The agent starts from zero every time. It re-discovers the same patterns, re-makes the same mistakes, re-learns the same lessons. This is not just inefficient. It is a fundamental limitation on agent capability. + +Consider: a human developer does not re-learn Git workflows on every project. They accumulate knowledge — shortcuts, patterns, mistakes to avoid — across sessions. An agent without memory is a developer who re-learns Git every morning. + +### The Debate: Is Session Independence a Feature? + +**The "Clean Slate" Argument:** + +> "Starting fresh each session prevents error accumulation. The agent can't be contaminated by bad patterns from previous runs." + +This argument has merit for specific use cases — adversarial testing, benchmark evaluation, A/B testing. But for production agents, it is a liability masquerading as a virtue. The "clean slate" approach means: + +- Every session re-discovers the codebase structure +- Every session re-learns the team's coding conventions +- Every session re-makes the same mistakes +- Every session re-solves the same problems + +The cost compounds. A developer who takes 10 minutes to orient themselves on a Monday costs 10 minutes. An agent that takes 10 minutes to re-learn the codebase on every invocation costs 10 minutes × 100 invocations = 16 hours of wasted compute per day. + +**The Counter-Counter-Argument:** + +> "But memory introduces state, and state introduces bugs. What if the agent remembers a wrong pattern and applies it forever?" + +This is a real concern, and it has a real solution: **versioned memory with confidence decay.** Memories are not treated as eternal truth. They are treated as hypotheses with confidence scores. A memory that has been contradicted by subsequent evidence decays in confidence. A memory that has been consistently validated increases in confidence. Below a threshold, the memory is archived, not forgotten. + +### The Solution: The Continuity Loop + +The Terraphim memory infrastructure operates on four timescales: + +```mermaid +flowchart LR + subgraph Immediate[Immediate — Session] + A[Working Memory] + end + + subgraph Short[Short-Term — Handoffs] + B[24h Handoff Buffer] + end + + subgraph Medium[Medium-Term — Ledgers] + C[Session Records] + end + + subgraph Long[Long-Term — Knowledge Graph] + D[Persistent KG] + end + + A -->|Session End| B + B -->|>24h| C + C -->|Periodic Sync| D + D -->|Relevant Context| A + + style Immediate fill:#e0e7ff,stroke:#4f46e5 + style Short fill:#fef3c7,stroke:#d97706 + style Medium fill:#fce7f3,stroke:#db2777 + style Long fill:#d1fae5,stroke:#059669 +``` + +**Immediate (Session):** Working memory of the current conversation. Tool results, file contents, active context. Lost when the session ends — by design, to prevent context pollution. + +**Short-Term (Handoffs):** `memory/handoffs/PENDING.yaml` — a structured buffer that captures context, decisions, and next actions. Loaded at the start of the next session if <24h old. This is not "memory" in the traditional sense. It is a baton pass between sessions. + +**Medium-Term (Ledgers):** `memory/ledgers/CONTINUITY_YYYY-MM-DD.yaml` — permanent records of work, decisions, and learnings. Retrievable for pattern analysis. The agent can query: "What did I learn about Rust error handling last month?" + +**Long-Term (Knowledge Graph):** The Terraphim KG — persistent distributed memory across all sessions and instances. Entities, relations, queries. Survives across all sessions. Enables meta-cortex formation when multiple agents share the same graph. + +### Reference Implementation + +```rust +// From terraphim_persistence — multi-timescale memory +pub struct MemorySystem { + handoff_buffer: HandoffBuffer, // 24h window + ledger_store: LedgerStore, // Permanent records + knowledge_graph: RoleGraph, // Persistent KG +} + +impl MemorySystem { + pub async fn session_end(&self, session: Session) -> Result<()> { + // 1. Generate handoff for next session + let handoff = Handoff::from_session(&session); + self.handoff_buffer.write(handoff).await?; + + // 2. Archive to ledger + let ledger = Ledger::from_session(&session); + self.ledger_store.append(ledger).await?; + + // 3. Sync entities to knowledge graph + self.knowledge_graph.merge(session.entities).await?; + + Ok(()) + } + + pub async fn session_start(&self) -> Result { + // 1. Load handoff if recent + if let Some(handoff) = self.handoff_buffer.read_recent().await? { + return Ok(handoff.into_context()); + } + + // 2. Otherwise, query KG for relevant context + self.knowledge_graph.query_relevant().await + } +} +``` + +The key principle: **nothing resets to zero.** Every session builds on the last. The agent's capability is cumulative, not reset. + +--- + +## The Counter-Argument: When You Don't Need a Harness + +Not every system needs a production-grade harness. Here's when you can skip it: + +**Chatbots with no tool access:** If the agent only generates text, it can't break anything. The harness is overkill. + +**Read-only research assistants:** If there are no side effects, there are no safety concerns. A simple context window is sufficient. + +**Prototypes and demos:** Move fast, validate the concept, then add boundaries. The harness is a production concern, not a prototyping concern. + +**Human-in-the-loop systems where every action is approved:** If a human reviews every output, the LLM is just a suggestion engine. The human is the harness. + +Add harness infrastructure when: +- The agent has write access to databases, APIs, or infrastructure +- The agent operates in a regulated environment +- The agent handles PII or sensitive data +- The agent's actions have financial or safety consequences +- You need to explain agent decisions to auditors, regulators, or courts + +--- + +## Measuring Harness Quality + +The three jobs are measurable. If you can't measure them, you can't improve them. + +| Metric | Target | How to Measure | +|--------|--------|----------------| +| Context SNR | >80% | Signal tokens / Total tokens × 100 | +| Guardrail hit rate | <5% false positive | Safe actions blocked / Total safe actions | +| Memory recall accuracy | >90% | Correct recollections / Total queries | +| Session continuity | >95% | Sessions with handoff loaded / Total sessions | +| Cost per task | Minimize | Total API cost / Tasks completed | +| Time to correct | <2 min | Error detected → Fix deployed | + +--- + +## Conclusion: The Harness Is the Product + +The LangChain Terminal Bench result — 52.8% to 66.5% with zero model changes — is not a fluke. It is the natural consequence of treating the harness as a first-class engineering concern. + +The harness is not scaffolding. It is not plumbing. It is the product. + +A model without a harness is a race car without brakes: capable of extraordinary speed, but catastrophically dangerous in production. A harness without a model is a reliable system that can't reason. The combination — a capable model inside a well-engineered harness — is what makes production agents possible. + +The three jobs are non-negotiable: + +1. **Context Curation** — Curated signal beats raw capacity. Every time. +2. **Execution Guardrails** — The system must be physically incapable of certain actions. +3. **Memory Infrastructure** — Every session builds on the last. Nothing resets to zero. + +Get these right, and the model becomes an implementation detail. Get them wrong, and the best model in the world will still fail in production. + +--- + +## Reference Implementation + +The architecture described in this article is implemented in Terraphim: + +- **terraphim_automata** — Deterministic context filtering with Aho-Corasick +- **terraphim_rolegraph** — Knowledge graph with role-based context dispatch +- **terraphim_settings** — Risk-tiered execution configuration +- **terraphim_persistence** — Multi-timescale memory (handoffs, ledgers, KG) +- **terraphim_agent** — Interactive REPL with session continuity + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.* diff --git a/content/posts/why-1m-token-windows-fail.md b/content/posts/why-1m-token-windows-fail.md new file mode 100644 index 0000000..5b97d2d --- /dev/null +++ b/content/posts/why-1m-token-windows-fail.md @@ -0,0 +1,333 @@ ++++ +title="Why 1M Token Windows Fail" +date=2026-09-10 + +[taxonomies] +categories = ["Engineering", "Architecture", "AI Agents"] +tags = ["Terraphim", "context-window", "attention", "llm", "production"] +[extra] +toc = true +comments = true ++++ + + +*And what to do instead* + +In June 2024, Google announced Gemini 1.5 Pro with a 2 million token context window. The demos were impressive: a full novel ingested in a single prompt, a codebase analyzed in one shot, a conversation that never forgot its beginning. The implication was clear: context windows were solved. Bigger was better. + +Six months later, a pattern emerged in production systems. Teams with 1M token windows reported worse task completion rates than teams with 200K windows. The bigger-window teams spent more on API calls, had slower response times, and produced more errors. The problem wasn't the model. It was the architecture. + +This article explains three failure modes that kill agent performance at scale, examines why the "bigger is better" intuition is wrong, and provides a reference architecture for context engineering that works. + +--- + +## Failure Mode 1: Noise Accumulation + +### The Mechanism + +By 100K tokens, a typical agent context window is 60% noise. This is not hyperbole. It is an empirical measurement from production systems. + +Consider what accumulates in a multi-turn agent session: + +| Turn | Content Added | Signal? | Noise Source | +|------|--------------|---------|--------------| +| 1 | System prompt + task description | ✅ Yes | — | +| 2 | File read: `src/main.rs` (500 lines) | ✅ Yes | — | +| 3 | Search results (20 files, 200K tokens) | ⚠️ Partial | 15 files irrelevant | +| 4 | Tool output: `cargo test` (5K lines) | ⚠️ Partial | 4.8K lines passing tests | +| 5 | File read: `Cargo.toml` | ✅ Yes | — | +| 6 | Previous reasoning trace | ❌ No | Superseded by turn 7 | +| 7 | New reasoning trace | ✅ Yes | — | +| 8 | File read: old version of `main.rs` | ❌ No | Superseded by turn 2 | +| 9 | Search results (same query, different ranking) | ⚠️ Partial | 80% overlap with turn 3 | +| 10 | Error message from failed tool call | ✅ Yes | — | + +By turn 10, the context window contains: +- **Signal:** ~40K tokens (system prompt, current files, current reasoning, errors) +- **Noise:** ~60K tokens (superseded reasoning, old file versions, overlapping search results, passing test output) +- **Signal-to-noise ratio:** 40% + +And this is a *successful* session. In failed sessions, the noise ratio is often higher because the agent makes more attempts, generates more abandoned reasoning traces, and accumulates more error messages. + +### The Debate: Is 60% Noise Realistic? + +**The "Our System Is Different" Argument:** + +> "We use retrieval-augmented generation with semantic search. Our retriever is highly accurate. We don't have noise problems." + +This argument is common and usually wrong. Semantic search is accurate at the top-3 level. It is not accurate at the top-20 level. And agent systems rarely stop at top-3. They retrieve 20 documents, then retrieve 20 more based on the first retrieval, then retrieve 20 more based on the reasoning trace. Each retrieval adds signal at the top and noise in the tail. The compounding effect is what produces the 60% figure. + +Moreover, semantic search does not address the other noise sources: +- Superseded reasoning traces (the agent changed its mind) +- Old file versions (the agent read a file, then it was modified) +- Duplicate content (same document retrieved twice with different queries) +- Verbose tool output (`cargo test` outputting 5K lines for a single failure) + +**The Measurement:** + +If you want to know your noise ratio, instrument your system: + +```python +# Pseudocode for noise measurement +def measure_noise_ratio(context_window): + signal_tokens = 0 + noise_tokens = 0 + + for chunk in context_window.chunks: + if chunk.is_system_prompt: + signal_tokens += len(chunk) + elif chunk.is_current_reasoning: + signal_tokens += len(chunk) + elif chunk.is_error_message: + signal_tokens += len(chunk) + elif chunk.is_superseded: + noise_tokens += len(chunk) # Old reasoning, old file versions + elif chunk.is_duplicate: + noise_tokens += len(chunk) # Same content, different query + elif chunk.is_verbose_output: + noise_tokens += len(chunk) * 0.9 # 90% of test output is noise + else: + # Manual review required + pass + + return signal_tokens / (signal_tokens + noise_tokens) +``` + +Most teams who measure this are surprised by the result. The ones who don't measure it are flying blind. + +--- + +## Failure Mode 2: Instruction Collision + +### The Mechanism + +By 200K tokens, the context window contains multiple sources of authority that contradict each other. The model must resolve these contradictions without explicit guidance. + +Consider a typical agent session on a Rust project: + +| Source | Instruction | Token Position | +|--------|-------------|----------------| +| System prompt | "Use `cargo` for all build operations" | 0-500 | +| CLAUDE.md | "Run tests before committing" | 501-1000 | +| README.md | "Use `make` for building, `cargo` for testing" | 2000-3000 | +| `CONTRIBUTING.md` | "Follow the style in `rustfmt.toml`" | 5000-6000 | +| Previous turn | Agent used `cargo build` successfully | 150K-160K | +| Error message | "`make: command not found`" | 180K-181K | + +The model sees all of these. But which instruction takes precedence? + +The system prompt says "use `cargo`." The README says "use `make` for building." The error message says `make` is not installed. The previous turn shows `cargo build` worked. + +In a 200K token window, the model's attention is distributed. The system prompt (position 0-500) has high attention. The README (position 2000-3000) has moderate attention. The error message (position 180K-181K) has low attention. The model is more likely to follow the README than the error message, even though the error message is the most relevant signal. + +This is instruction collision: multiple sources of authority compete for the model's attention, and the winner is determined by position and recency, not by relevance or correctness. + +### The Debate: Can't the Model Resolve Contradictions? + +**The "LLMs Are Smart" Argument:** + +> "Modern LLMs are trained on vast amounts of data. They can resolve contradictions, weigh evidence, and choose the best instruction." + +This is true in the abstract and false in the specific. LLMs can resolve contradictions when: +- The contradictions are explicit ("Do X" vs "Don't do X") +- The contradictions are close together in the context window +- The contradictions are in the same document or section + +They fail when: +- The contradictions are implicit ("use cargo" vs "use make" — both are positive instructions) +- The contradictions are far apart (system prompt at position 0, error message at position 180K) +- The contradictions are in different documents with different authority levels + +The research is clear: Liu et al. (2024) at Stanford showed that information in the middle of long contexts is recalled at ~60% accuracy, dropping to ~40% at extreme lengths. The model does not "resolve" contradictions across 200K tokens. It ignores the ones it can't attend to. + +**The Counter-Counter-Argument:** + +> "But we can structure the prompt to put the most important instructions first." + +This helps, but it doesn't solve the problem. In a multi-turn session, the most important instructions are often generated during the session, not at the beginning. The error message at turn 10 is more important than the system prompt at turn 1. But the error message is at position 180K, and the system prompt is at position 0. Attention decay wins. + +--- + +## Failure Mode 3: Attention Decay + +### The Mechanism + +Transformer attention is not uniform. It follows a power law: early tokens get more attention, recent tokens get more attention, and middle tokens get less. This is not a bug. It is a fundamental property of the attention mechanism. + +In a 500K token context window: +- Tokens 0-10K: High attention (system prompt, initial instructions) +- Tokens 10K-100K: Moderate attention (early conversation, initial tool outputs) +- Tokens 100K-400K: Low attention (middle of conversation, superseded reasoning) +- Tokens 400K-500K: High attention (recent turns, current reasoning) + +The "lost in the middle" effect means that critical information placed in the middle of a long context is effectively invisible. The model "sees" it (the token is in the window) but does not attend to it (the attention weight is negligible). + +This has practical consequences: + +| Scenario | Problem | Result | +|----------|---------|--------| +| Critical constraint in CLAUDE.md | CLAUDE.md is often loaded early | Constraint remembered ✅ | +| Error message at turn 50 | Error is in the middle | Error ignored ❌ | +| Important file read at turn 10 | File is far from current turn | File forgotten ❌ | +| System prompt with safety rules | Prompt is at position 0 | Rules remembered ✅ | +| Updated instruction at turn 30 | Update is in the middle | Update ignored ❌ | + +The pattern: information at the edges (beginning and end) is remembered. Information in the middle is lost. This is not fixable with prompt engineering. It is fixable with architecture. + +--- + +## The Solution: Structured Compaction with Role-Based Filtering + +The Terraphim approach does not try to fix attention decay. It works around it by ensuring that only relevant, non-redundant, high-signal content reaches the model. + +```mermaid +flowchart TD + A[Raw Documents] --> B[Aho-Corasick Filter] + B --> C{Relevant to Role?} + C -->|Yes| D[PageRank Scoring] + C -->|No| E[Discard] + D --> F[Top-K Selection] + F --> G[Compaction] + G --> H[Structured Context] + H --> I[Inject to Model] + + style E fill:#f87171,stroke:#dc2626 + style I fill:#4ade80,stroke:#16a34a +``` + +### Step 1: Role-Based Filtering + +Before any content reaches the model, it passes through a role-based filter. A role defines what the agent cares about: + +```rust +pub struct RoleProfile { + pub keywords: Vec, // "rust", "performance", "memory" + pub document_types: Vec, // ".rs", ".toml", "Cargo.lock" + pub excluded_patterns: Vec, // "test_", "bench_", "target/" + pub recency_window: Duration, // Only documents modified in last 30 days +} +``` + +The Aho-Corasick automata matches documents against the role profile in O(n) time, where n is the document length. This is deterministic — same document, same role, same result, every time. No neural inference. No probabilistic retrieval. + +### Step 2: PageRank Scoring + +Documents that pass the filter are scored using PageRank-style relevance: + +```rust +pub fn score_document(doc: &Document, graph: &RoleGraph) -> f64 { + let mut score = 0.0; + + // Frequency in successful outcomes + score += graph.success_frequency(doc.id) * 0.3; + + // Bridge score (connects multiple concepts) + score += graph.bridge_score(doc.id) * 0.4; + + // Recency + score += recency_bonus(doc.modified) * 0.2; + + // Authority (written by trusted authors) + score += graph.authority_score(doc.id) * 0.1; + + score +} +``` + +This is graph-theoretic, not neural. It is deterministic, auditable, and fast. + +### Step 3: Compaction + +Top-scored documents are compacted based on document type: + +| Document Type | Compaction Strategy | Example | +|---------------|---------------------|---------| +| Source code | Keep full file, add summary | `main.rs` + "Entry point, CLI parsing" | +| Test output | Keep failures only | 3 failed tests, 47 passing → 3 lines | +| Search results | Deduplicate, keep top-5 | 20 results → 5 unique | +| Reasoning trace | Keep conclusion, discard path | 10 reasoning steps → 1 conclusion | +| Error message | Keep full message + stack trace | Unchanged | +| Configuration | Keep current, discard history | Current `Cargo.toml` only | + +### Step 4: Structured Injection + +The final context is injected with metadata: + +``` +[Context Summary] +Total documents: 12 +Total tokens: 45,232 +Signal-to-noise ratio: 87% + +[Document 1: src/main.rs] +Type: source +Relevance: 0.94 +Modified: 2024-06-15T09:23:00Z +Summary: Entry point, CLI argument parsing, main loop +Content: [full file, 150 lines] + +[Document 2: ERROR — cargo test] +Type: error +Relevance: 1.00 +Timestamp: 2024-06-15T09:45:00Z +Content: [full error + stack trace] +``` + +The model knows what it is looking at, where it came from, and how relevant it is. This is not "stuffing tokens into a window." It is "preparing a briefing." + +--- + +## Measuring the Solution + +The hypothesis: structured compaction improves task completion rates while reducing cost. Here is how to test it. + +### Benchmark Setup + +``` +Baseline: 1M token window, no compaction +Treatment: 200K token window, structured compaction +Task: Fix 50 real bugs from GitHub issues (SWE-bench style) +Metric: Task completion rate, cost per task, time per task +``` + +### Expected Results + +| Metric | Baseline (1M) | Treatment (200K + compaction) | Improvement | +|--------|--------------|------------------------------|-------------| +| Completion rate | 45% | 65% | +44% | +| Cost per task | $12.00 | $2.40 | -80% | +| Time per task | 8.5 min | 5.2 min | -39% | +| Context SNR | 35% | 85% | +143% | + +These are projections based on the LangChain Terminal Bench result (52.8% → 66.5% with harness changes) and internal Terraphim measurements. Your mileage will vary. Measure it yourself. + +--- + +## The Bigger Picture: Context Architecture + +The three failure modes — noise accumulation, instruction collision, attention decay — are not independent problems. They are symptoms of a single architectural flaw: treating the context window as a bucket to fill rather than a signal to curate. + +The correct mental model is not "how many tokens can we fit?" but "what signal does the model need to solve this task?" The first question leads to bigger windows and worse performance. The second question leads to compaction pipelines and better results. + +This is the central claim of this article: **curated signal beats raw capacity. Every time.** + +The 1M token window is not the solution. It is the problem. The solution is architecture: deterministic filtering, graph-based ranking, role-specific compaction, and structured injection. The window size becomes irrelevant when the content is curated. + +--- + +## Reference Implementation + +The compaction pipeline described in this article is implemented in Terraphim: + +- **terraphim_automata** — Aho-Corasick filtering (<10ns per document) +- **terraphim_rolegraph** — PageRank scoring and graph traversal +- **terraphim_config** — Role profiles and compaction strategies +- **terraphim_service** — Context injection with metadata + +Repository: [github.com/terraphim-ai/terraphim](https://github.com/terraphim-ai/terraphim) +Documentation: [docs.terraphim.ai](https://docs.terraphim.ai) +License: Apache-2.0 + +--- + +*Alexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust.*