Most teams fixate on the model pick. Harness engineering points elsewhere. In LangChain’s Terminal-Bench, swapping only the harness — with the same model — lifted a coding agent from roughly 30th place into the top 5. That reframes the problem. If the harness decides quality, then loop control is architecture, not deployment detail. Paul Iusztin’s course Building a Coding Agent From Scratch presents Decode with three modes, each tuned for a different latency profile and a different inference provider.
Is the harness really more important than the model?
Yes. Terminal-Bench showed that replacing only the harness, with the model unchanged, can sharply improve an agent. The question stops being “which model” and becomes “how do we run the loop.” That is practical, not theoretical.
This changes your design lens. You focus less on parameters and more on mechanics: when the model is called, how tools are gated, how context is stored and compacted. Stability and throughput live here. The model is a component; the harness is the coordinate system.
Decode builds on that idea. It separates run modes because different interactions demand different latency. Live chat cannot wait; offline batches need throughput; async splits the difference. The same LLM behaves like different agents under different harnesses.
None of this diminishes the model. It highlights that identical LLMs in different harnesses act differently in quality and behavior. Your true lever is the loop architecture.
One headless core, three shapes
The system’s center is a headless harness with no interface of its own. Inside runs the shared agent loop: the LLM picks an action, a tool executes, the observation feeds back. Everything reads and writes to the context window. Interfaces plug on top without changing the core.
The agent is small. In Decode it is a ~20-line Pydantic AI definition composing the model, tools, and output type. In Claude Code’s leaked source, the core loop is roughly 150 lines. Everything else — memory, skills, sandbox, permissions, LSP feedback, compaction — is harness. It provides discipline, not “magic.”
This composition lets you expose three shapes without branching the core logic. You toggle interface adapters, queues, and runtimes. The loop body stays the same, which simplifies debugging and result repeatability.
Cleanly separating interfaces from the core also locks in invariants: which events can steer, when to inject user input, where to interrupt without corrupting history. That is critical for correctness.
Mode 1: Interactive, online — how to steer safely
Interactive mode wires a terminal UI to one live in-memory session in the same process. Events stream back through async generators as tokens arrive. The hard part is mid-flight steering: typing while a tool call is running corrupts the turn. Decode’s answer is a steering queue plus a priority gate at safe boundaries.
Input is buffered and injected only at safe points. The loop exposes two: MODEL_REQUEST, before the next model call, and WOULD_STOP, when the turn would end. You do not “break” the turn; you slot into the next one.
Three input modes map to those boundaries. Plain Enter steers within the turn. Alt+Enter queues a follow-up until the turn stops. Esc triggers a cooperative abort at the next boundary, clearing both queues so history stays intact. You control tempo without damaging context.
A human reads every token, so latency is the constraint. That is why this mode belongs on a low-latency hosted API. You pay per token, but you get immediate feedback, as live work demands.
Modes 2 and 3: Remote offline and async online
Remote mode keeps the harness headless and runs it on a server through an agent runtime. Decode uses Kitaru, ZenML’s agent runtime, deployed to GCP, with agents executing on Modal. Nobody is watching the stream, so a backlog fans out to N harnesses in parallel, each producing its own PR.
The runtime records progress step by step. If a sandbox dies mid-task, it resumes from the last recorded step instead of restarting. If a run pauses for human input, it freezes and consumes no compute while it waits. That saves cost and reduces fragility.
Tools execute inside Modal Sandboxes remotely, Docker locally. The metric that matters is throughput per dollar, not time-to-first-token. You trade “responsiveness” for “output,” which fits batch work.
Async online sits between the two. A live session hands work to a job queue and returns immediately. Background workflows fan out LLM calls and post results later. The run outlives the client that started it. This pattern powers Slack-triggered agents and background PR review. It bills like batch, not like chat.
Why provider choice shifts by mode: cost math and capacity
The cost model follows the latency requirement, and the gap is large. Take 1,000 documents at 30,000 input tokens each and roughly 500 output tokens per document. At frontier API rates of $3 per million input and $15 per million output, the arithmetic lands near $97. Prompt caching does not help, because every document is a different prefix.
Batched on a serverless GPU at around 3,000 tokens per second, the same work fits in under three GPU hours — roughly $13. The reverse case is just as sharp. Decode’s default test model, Qwen3.6 35B, runs on a single H200. Modal’s published pricing lists H200 SXM at $0.001261 per second, or about $4.54 per hour. Leave an interactive agent idle overnight awaiting a “y,” and ten idle hours add roughly $45 to the bill.
That is the whole argument. Interactive work pays per token because a human is waiting. Offline and async work pays per GPU-hour because throughput is the objective and idle time is the enemy.
There is a second axis: serverless versus reserved capacity. Modal’s pricing analysis reduces it to one comparison: reservations charge the peak rate for the whole contract; serverless follows the demand curve. When the peak-to-average ratio exceeds the reservation discount, serverless is cheaper. Modal reports typical discounts of 2–5× against peak-to-average ratios of 5–10× for inference, training, and agentic development. Industry surveys it cites put reservation utilization below 30%, often under 10%.
Key takeaways: harness beats model; interactive mode steers via a queue draining at MODEL_REQUEST and WOULD_STOP; remote and async modes are throughput-bound, so GPU-hour billing wins at volume; 1,000 documents cost about $97 at frontier API rates versus about $13 of batched GPU time; serverless wins whenever peak-to-average demand exceeds the reservation discount, typically 5–10× against 2–5×.
Based on the source material.
