On this page
What changes when you run many agents
The signals per agent stay the same: liveness, errors, latency, tokens and cost, and answer quality (see how to monitor AI agents in production). What changes is the work around them:
- Finding the problem. Scrolling through every agent stops working long before a hundred. You need the unhealthy ones on top.
- Knowing what an agent belongs to. An agent called
summarizermeans nothing at 3 a.m. unless you can see its product, its role and who owns it. - Setting rules once. A spend cap or an error-rate alert configured agent by agent drifts. Policies belong at the level of an app.
- Acting at scale. When a provider degrades or a release goes wrong, you need to stop a whole product's agents, not click through them one at a time.
1. Organise the fleet by app
In Agent Control Panel, every agent belongs to an app: one product or service that you connect with its own API key and webhook secret. Group agents the way your incidents are scoped, usually one app per product or per deployable service.
- Each app has its own control scope (
monitor,controlorfull), enforced by the server. A reporting-only app can never receive a command; a production app can be limited to pause, resume, stop, restart and model swaps. - Each agent can have a type (orchestrator, worker, tool) and a parent agent in the same app. The Agents screen shows them as a list or as a hierarchy, so an orchestrator and the workers it calls stay together.
- Names should say what the agent does for whom, for example
support-triagerather thanagent-3. Search and filters on the Agents screen work on these names.
2. Triage: look at the worst first
A fleet view is only useful if it answers "what needs attention right now?" before anything else.
- The Dashboard opens with problems, money (token spend) and recent activity across all apps, and says so plainly when nothing needs attention.
- The Live Monitor sorts agents by worst health, highest token use or most recent activity, filters by status (running, idle, paused, error) and refreshes on its own.
- The Apps screen shows each app's status, health, last heartbeat, agent count and tokens today, which is where a whole product going quiet shows up first.
The health score of an agent is the share of its recent runs that finished without an error. With fewer than five recent runs there is no score at all, instead of a misleading 0% or 100%, so a new or rarely used agent never hides a real problem at the top of the list.
3. Set alert policies once, per app
Alert rules can target an app or a single agent. For a fleet, start at the app level and add agent-level rules only where an agent is different (an expensive model, a critical path).
| Rule | Catches | Suggested start |
|---|---|---|
| Heartbeat lost | A process or deployment that stopped reporting | Every app, notify |
| Error rate | A failing provider, a broken tool, a bad release (needs at least five runs) | Every app, notify |
| Daily spend cap | Runaway loops and unexpected traffic | Per app after a few days of real numbers, automatic pause |
| Spend anomaly | A sudden jump against the agent's own baseline (needs twenty baseline runs) | Agents on expensive models, notify |
Each rule either notifies or attempts an automatic pause, within the app's control scope, with a cooldown so one incident does not produce a flood. Details: alerts and automated response and daily spend caps and automatic pause.
4. Act on many agents at once
On the Apps screen, Pause sends a real pause command to every agent in that app, and Resume releases them. Use it when a provider degrades, a release misbehaves, or spend climbs across a whole product. For a single agent, the same commands, plus stop, restart and a model swap, are on its page.
- A pause takes effect before the agent's next wrapped call. It does not cancel a model request that is already in flight.
- The pause is held by Agent Control Panel and re-applied when the agent's process restarts or another replica starts, so a deploy does not quietly undo it (Node SDK 0.3.3 and Python SDK 0.1.3 or later).
- Every command and who sent it is recorded in the audit history.
If Agent Control Panel itself is unreachable, the SDK fails open: your agents keep working and reporting resumes later. See controlling agents and the AI agent kill switch guide.
5. Cost across the fleet
The Tokens screen breaks usage down by agent, app and model over any date range, so a cost increase can be traced to the agent and model that caused it. Cost is an estimate from the tokens your agents report and the model prices available to Agent Control Panel; your provider invoice remains the source of truth. To move an expensive agent to a cheaper model without a deploy, see switching an agent's model.
A weekly routine for a fleet
- Open the Live Monitor sorted by worst health and look at the five lowest agents.
- Check the Apps screen for any app whose last heartbeat is old or whose tokens today look unusual.
- Read the alert history: an alert that fired and needed no action is a threshold to adjust.
- Compare cost by app against last week on the Tokens screen.
- Make sure every new agent has a clear name, the right app and, where it matters, a type and parent.
Frequently asked questions
How do I monitor many AI agents at once?
Group agents by app, use a view sorted by worst health so problems come first, set alert rules per app, and keep a way to pause a whole app in one action.
Can I pause all agents of a product at the same time?
Yes. In Agent Control Panel, pausing an app sends a pause command to every agent in it; resuming releases them. Every command is recorded in the audit history.
Does a pause survive a restart or a new deployment?
Yes. The pause is held by Agent Control Panel, and from Node SDK 0.3.3 and Python SDK 0.1.3 an agent that restarts waits briefly for that state before its first run.
Do I have to configure alerts for every agent?
No. Alert rules can target a whole app; add rules for individual agents only where one needs a different threshold.