Architecture

Fail-open vs in-path: designing a control plane for AI agents

Any system that can pause, stop or re-route an AI agent has to sit somewhere. There are two places: in the request path, as a proxy that every model call passes through, or beside the agent, as a small library inside the app that reports out of band and receives commands. The choice decides what happens to your agents on the day the control layer itself has a problem. This essay explains both designs, the trade-offs, and why Agent Control Panel is built the second way.

By the Agent Control Panel teamPublished 5 min read

On this page
  1. Two places to put control
  2. The trade-offs
  3. What fail-open means in practice
  4. A kill switch must survive a restart
  5. Commands you can trust
  6. How to choose
  7. Frequently asked questions

Two places to put control

In the path. A gateway or proxy sits between your app and the model provider. Every request goes through it, so it can count tokens, block a request, swap the model or apply a rate limit before the call reaches the provider. It usually holds the provider API keys.

Beside the agent. A library inside your app wraps each agent run. Model calls go straight from your app to the provider, with your own keys. The library reports what happened (status, tokens, model, timings, errors) in the background, sends a heartbeat, and checks the agent's state, running, paused or stopped, before each run it wraps. Commands arrive as signed messages, or with the next heartbeat reply.

Both can show you every run and both can stop an agent. They differ in what they cost on every call and in how they fail.

The trade-offs

QuestionIn the path (proxy)Beside the agent (SDK)
Latency per callOne more network hop on every model callNone on the call; reports are buffered and sent in the background
If the control layer is downCalls fail, unless you build a bypassAgents keep working; reporting and new commands resume later
Provider API keysUsually stored in the proxyStay in your app; the control layer never needs them
EnforcementCan refuse any single requestRefuses the next wrapped run; a request already sent finishes
What it seesModel calls that pass through itWhole runs you wrap, including your own tool steps and errors
AdoptionChange the base URL of every clientInstall a library and wrap the agent entry point

Neither column wins everywhere. A proxy is the right tool when every request must be filtered by policy, or when keys must be managed centrally. A library beside the agent is the right tool when the agents already run in production with their own keys, latency matters, and nobody wants a new single point of failure in front of the model provider.

What fail-open means in practice

Fail-open means a problem in the control layer never becomes an outage in your product. In the Agent Control Panel SDKs that rule shapes every default:

  • Run reports are buffered and flushed every five seconds or after fifty runs, with retries; a full buffer drops the oldest reports, never your agent's work. The default buffer holds 200 runs.
  • Each request to Agent Control Panel has a 10-second timeout and is never retried in the agent's own call path.
  • Nothing the SDK does throws into your code, except the documented "agent is paused" error when you ask it to wrap a run while the agent is paused or stopped.
  • Large content is cut to 100,000 characters per field and flagged, so one huge answer cannot block a batch.

The cost of fail-open is honest to state: if Agent Control Panel is unreachable at the moment you press pause, the command reaches the agent only when the connection returns. For most teams that is the right trade, because the alternative is that the control layer can take production down.

A kill switch must survive a restart

A control layer beside the agent has one subtle failure mode. When your process restarts, after a deploy, a crash or a new replica, the library starts with no knowledge of the agent's state until the first heartbeat reply arrives. If the agent runs immediately, a paused agent does one run after every restart.

We found exactly this on our own demo agent: it was paused, a deploy restarted it, and it ran once. Since Node SDK 0.3.3 and Python SDK 0.1.3, the first run after start waits for the first heartbeat reply, for at most three seconds by default. If the reply says paused, the run is refused. If Agent Control Panel does not answer in time, the run goes ahead, which keeps the fail-open promise, and later runs never wait. The state itself is held by Agent Control Panel, so it also applies to new replicas.

The general lesson: decide what the library assumes when it does not know yet, and make that assumption explicit and bounded.

Commands you can trust

A control channel that can stop agents is also an attack surface. Commands from Agent Control Panel are HMAC-SHA256 signed, carry a timestamp and a single-use nonce, and are accepted only within five minutes of that timestamp and only once. Unsigned or replayed commands are rejected, and a previous secret can be accepted during rotation. Each app also has a control scope enforced by the server: a reporting-only app cannot receive commands at all, and an app can be limited to pause, resume, stop, restart and model swaps.

Because webhooks can be missed, for example when an app has no public URL, the state is also carried in every heartbeat reply. The result is that control works for workers and cron jobs too, and a missed message only delays a command by one heartbeat interval (30 seconds by default).

How to choose

  1. If you must block or rewrite individual model requests by policy, you need something in the path.
  2. If your agents already run in production and the priority is to see them and stop or steer them without adding risk, put control beside the agent and make it fail open.
  3. If you need both, they combine: a gateway for request policy, and a control panel for per-agent state, cost, alerts and the people who operate them.

Further reading: SDK lifecycle and reporting limits, webhook security, controlling agents and what an agent control plane is.

Frequently asked questions

What does fail-open mean for an AI agent control plane?

If the control layer is slow or down, the agents keep running and serving users; reporting and new commands resume when the connection returns.

Does Agent Control Panel need my model provider API keys?

No. The SDK runs inside your app, your agents call the provider with your own keys, and the SDK reports tokens and the model used.

Can a pause stop a request that is already running?

No. A pause refuses the next wrapped run; a model request that was already sent finishes. A proxy in the request path can refuse individual requests.

What happens to a paused agent when its process restarts?

From Node SDK 0.3.3 and Python SDK 0.1.3, the first run waits up to three seconds for the state held by Agent Control Panel, so the pause holds across restarts.