On this page
Two places to put control
In the path. A gateway or proxy sits between your app and the model provider. Every request goes through it, so it can count tokens, block a request, swap the model or apply a rate limit before the call reaches the provider. It usually holds the provider API keys.
Beside the agent. A library inside your app wraps each agent run. Model calls go straight from your app to the provider, with your own keys. The library reports what happened (status, tokens, model, timings, errors) in the background, sends a heartbeat, and checks the agent's state, running, paused or stopped, before each run it wraps. Commands arrive as signed messages, or with the next heartbeat reply.
Both can show you every run and both can stop an agent. They differ in what they cost on every call and in how they fail.
The trade-offs
| Question | In the path (proxy) | Beside the agent (SDK) |
|---|---|---|
| Latency per call | One more network hop on every model call | None on the call; reports are buffered and sent in the background |
| If the control layer is down | Calls fail, unless you build a bypass | Agents keep working; reporting and new commands resume later |
| Provider API keys | Usually stored in the proxy | Stay in your app; the control layer never needs them |
| Enforcement | Can refuse any single request | Refuses the next wrapped run; a request already sent finishes |
| What it sees | Model calls that pass through it | Whole runs you wrap, including your own tool steps and errors |
| Adoption | Change the base URL of every client | Install a library and wrap the agent entry point |
Neither column wins everywhere. A proxy is the right tool when every request must be filtered by policy, or when keys must be managed centrally. A library beside the agent is the right tool when the agents already run in production with their own keys, latency matters, and nobody wants a new single point of failure in front of the model provider.
What fail-open means in practice
Fail-open means a problem in the control layer never becomes an outage in your product. In the Agent Control Panel SDKs that rule shapes every default:
- Run reports are buffered and flushed every five seconds or after fifty runs, with retries; a full buffer drops the oldest reports, never your agent's work. The default buffer holds 200 runs.
- Each request to Agent Control Panel has a 10-second timeout and is never retried in the agent's own call path.
- Nothing the SDK does throws into your code, except the documented "agent is paused" error when you ask it to wrap a run while the agent is paused or stopped.
- Large content is cut to 100,000 characters per field and flagged, so one huge answer cannot block a batch.
The cost of fail-open is honest to state: if Agent Control Panel is unreachable at the moment you press pause, the command reaches the agent only when the connection returns. For most teams that is the right trade, because the alternative is that the control layer can take production down.
A kill switch must survive a restart
A control layer beside the agent has one subtle failure mode. When your process restarts, after a deploy, a crash or a new replica, the library starts with no knowledge of the agent's state until the first heartbeat reply arrives. If the agent runs immediately, a paused agent does one run after every restart.
We found exactly this on our own demo agent: it was paused, a deploy restarted it, and it ran once. Since Node SDK 0.3.3 and Python SDK 0.1.3, the first run after start waits for the first heartbeat reply, for at most three seconds by default. If the reply says paused, the run is refused. If Agent Control Panel does not answer in time, the run goes ahead, which keeps the fail-open promise, and later runs never wait. The state itself is held by Agent Control Panel, so it also applies to new replicas.
The general lesson: decide what the library assumes when it does not know yet, and make that assumption explicit and bounded.
Commands you can trust
A control channel that can stop agents is also an attack surface. Commands from Agent Control Panel are HMAC-SHA256 signed, carry a timestamp and a single-use nonce, and are accepted only within five minutes of that timestamp and only once. Unsigned or replayed commands are rejected, and a previous secret can be accepted during rotation. Each app also has a control scope enforced by the server: a reporting-only app cannot receive commands at all, and an app can be limited to pause, resume, stop, restart and model swaps.
Because webhooks can be missed, for example when an app has no public URL, the state is also carried in every heartbeat reply. The result is that control works for workers and cron jobs too, and a missed message only delays a command by one heartbeat interval (30 seconds by default).
How to choose
- If you must block or rewrite individual model requests by policy, you need something in the path.
- If your agents already run in production and the priority is to see them and stop or steer them without adding risk, put control beside the agent and make it fail open.
- If you need both, they combine: a gateway for request policy, and a control panel for per-agent state, cost, alerts and the people who operate them.
Further reading: SDK lifecycle and reporting limits, webhook security, controlling agents and what an agent control plane is.
Frequently asked questions
What does fail-open mean for an AI agent control plane?
If the control layer is slow or down, the agents keep running and serving users; reporting and new commands resume when the connection returns.
Does Agent Control Panel need my model provider API keys?
No. The SDK runs inside your app, your agents call the provider with your own keys, and the SDK reports tokens and the model used.
Can a pause stop a request that is already running?
No. A pause refuses the next wrapped run; a model request that was already sent finishes. A proxy in the request path can refuse individual requests.
What happens to a paused agent when its process restarts?
From Node SDK 0.3.3 and Python SDK 0.1.3, the first run waits up to three seconds for the state held by Agent Control Panel, so the pause holds across restarts.