Agent loops usually look sequential: generate a tool call, wait for the tool, put the result into context, then generate again. That model is easy to reason about, but it leaves time on the table. If a tool call is already obvious from the tokens being generated, waiting for the entire program or turn to finish is unnecessary.
Alex Zhang’s Speculative Programmatic Tool Calling (sPTC) frames this as a harness problem: overlap tool computation with code generation, then reuse the result if the final program actually makes that call.[1] The important word is if. Speculation must be cheap to cancel and safe to discard.
Where the latency comes from
Consider a code-based agent that generates a small program. The root model may spend seconds producing the program, while a sub-agent or search call also takes seconds. A synchronous harness puts these costs one after another:
There are two useful overlap points. First, launch calls during token streaming, while the main context is still being generated. Second, inspect the completed program like a simple just-in-time compiler and run independent calls concurrently. Zhang describes both opportunities for programmatic tool calling.[1]
The speculative idea
The harness maintains two views of the program. A shadow REPL parses partial output and looks for calls that can be launched. The real REPL remains authoritative and executes the final program. When the real program reaches a call, it checks whether a matching future already exists.
# while the model is still streaming
partial = receive_tokens()
for call in parse_safe_calls(partial, shadow_namespace):
speculation[call_id(call)] = launch_async(call)
# after generation completes
result = execute(final_program, real_namespace,
cached_calls=speculation)Literal arguments are the easy case. If the stream contains llm_query("Summarize this document"), the harness can recognize the complete call and start it. Dependencies are harder: a later call may need the output of an earlier call, or a branch may never execute. The shadow environment exists to compute safe dependencies without mutating the real execution state.[1]
How to design it safely
1. Define a speculation contract
Tools should declare whether they are eligible for speculation. A useful first contract is speculatable plus pure: the tool can run early, and running it has no irreversible side effects. Search, retrieval, and sub-LLM calls are natural candidates. Sending an email, charging a card, deleting a file, or mutating a production database should not be launched from partial code.
2. Give every call an identity
Match a future using the tool name, normalized inputs, and an occurrence number when identical calls are non-deterministic. Without an occurrence number, three parallel votes over the same prompt might accidentally share one result. Deterministic calls can be deduplicated more aggressively; non-deterministic calls need separate identities.[1]
3. Track dependencies, not just text
A call using a literal can be launched immediately. A call using "Summarize: " + first_result should wait for its dependency. This is a small dependency graph: nodes are tool futures, and edges describe values that must resolve first. The graph lets independent branches run in parallel while preserving program order where it matters.
4. Keep the real runtime authoritative
Partial code can be syntactically incomplete or logically wrong. The shadow runtime should not become the source of truth. It should only produce candidates and futures; the completed program decides which results are actually consumed. If a speculative call is not used, cancel it when possible or record the wasted work.
5. Bound the cost
Speculation can overload the tool backend. Use an allowlist, concurrency limits, cancellation, timeouts, and a budget for speculative work. The source implementation uses a shadow copy of the REPL and blocks unsafe dependencies; those controls are as important as the early launch itself.[1]
How it relates to research
sPTC sits at the intersection of several lines of work. They share the goal of overlapping computation, but they speculate at different layers and make different guarantees:
| Work | What is speculated | Main idea |
|---|---|---|
| Speculative decoding | Output tokens | A smaller draft model proposes tokens that a larger model verifies in parallel; the method can preserve the target distribution.[5] |
| Conveyor | Partial code/tool work | Parse execution opportunities during decoding so a line of code can run before the whole response is complete.[2] |
| AsyncFC | Function-call futures | Separate decoding from function execution and overlap independent calls without changing the model or tool implementation.[3] |
| Speculative Interaction Agents | Tool calls and interaction I/O | Use asynchronous I/O and speculative calls for real-time agents, including settings where later user input may change the best action.[4] |
| sPTC | Programmatic tool calls | Use a shadow REPL to inspect partial programs, launch safe calls, and route matching real calls to cached futures.[1] |
The distinction matters. Speculative decoding is primarily a model-inference technique with a formal output-preservation story. Tool speculation is an execution-system technique: it must reason about branches, side effects, dependencies, cancellation, and whether a call will be used at all. The guarantees are therefore different.
What to measure
A faster first token is not enough. Compare a baseline harness and a speculative harness on the same tasks and serving load. Report:
- End-to-end latency: time to first useful output and time to task completion.
- Overlap: how much tool runtime occurs while the root model is decoding.
- Speculation precision: the fraction of launched calls that the final program consumes.
- Wasted work: cancelled, abandoned, or duplicate calls and their cost.
- Correctness: task success, tool arguments, and side-effect safety compared with baseline.
- Load behavior: queue depth, backend saturation, and tail latency under concurrency.
Results are highly workload-dependent. Zhang reports roughly 1–1.2× speedups in the RLM experiments discussed in the post, while also noting that larger gains appear in more deterministic programs where overlap is obvious.[1] Related work on asynchronous tool execution reports larger gains in its own settings, but those numbers should not be transferred across harnesses without reproducing the workload.[3], [4]
The practical takeaway
Start with the narrowest safe version: speculate only pure, idempotent calls with literal inputs; cap concurrency; keep the real interpreter authoritative; and log every hit and miss. Then add dependency-aware speculation when measurements show enough predictable work to justify it.
The deeper lesson is that an agent harness is an execution engine. Once an agent writes programs instead of isolated JSON calls, there are compiler-like opportunities—parsing, dependency analysis, scheduling, caching, and rollback—that can reduce latency without asking the model to think less.
References
- Alex L. Zhang, Speculative Programmatic Tool Calling, 2026.
- Xu et al., Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution, 2024.
- Feng et al., Concurrency without Model Changes: Future-based Asynchronous Function Calling for LLMs, 2026.
- Hooper et al., Building Interactive Real-Time Agents with Asynchronous I/O and Speculative Tool Calling, 2026.
- Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2023.