On this page
The signals that matter
| Signal | What it tells you | Typical action |
|---|---|---|
| Liveness (heartbeats) | The agent’s process is running and reporting | Check the deployment; a missing heartbeat means stale data, not proof the agent stopped |
| Error rate | Runs failing: provider errors, timeouts, exceptions in your code | Investigate the failing runs; switch model or pause if a provider is failing |
| Latency | Slow answers, often a provider issue or a growing prompt | Compare by model; shorten context; switch model |
| Tokens and cost | Spend per agent and app; loops show up as sudden jumps | Daily spend cap with automatic pause; cheaper model |
| Quality | Whether answers are correct and useful | User feedback, AI reviews of real runs, tested prompt changes |
| Changes | Which prompt and model each run used, and who changed them | Audit log; roll back the last change |
Why ordinary monitoring is not enough
Uptime checks and server metrics see a healthy process even while every answer is wrong or a loop is calling the model thousands of times. Agent failures are mostly logical, not infrastructural: a bad prompt, a degraded provider, a tool that returns nothing. So you have to measure at the level of each agent run, from inside the app.
Observability tools record traces of each run in detail, which is the right tool for debugging a single conversation. On its own, though, a trace does not stop the next run. Production monitoring is complete when every alert has an action attached, either for a person (switch the model, roll back the prompt) or automatic (pause the agent).
How Agent Control Panel monitors agents
- Heartbeats every 30 seconds from the SDK inside your app, so a silent agent is visible.
- Every run with duration, tokens, model, input and output, and the error when it failed. The SDK reads tokens and answers from OpenAI, Anthropic and Google Gemini responses.
- A health score per agent: the share of recent runs without errors. With fewer than five runs there is no score, rather than a misleading 0%.
- Alert rules for a daily spend cap, error rate (from five runs), spend anomaly (from twenty baseline runs) and a lost heartbeat. Each rule can notify or attempt an automatic pause, within the app’s control scope, and Discord delivery can be set up for the organization.
- Quality: feedback attached to an agent, and AI quality reviews of real runs.
- Action from the same screen: pause, stop, resume, switch model or roll out an approved prompt, with every command in the audit log.
The SDK fails open: if ACP is unreachable, your agents keep running and reporting resumes later. Start here: connect an existing app, then dashboard and live monitoring, runs and errors, tokens and cost and alerts.
A minimal setup for the first week
- Report every run from each agent, with tokens and model.
- Add a heartbeat-lost alert and an error-rate alert, set to notify.
- Add a daily spend cap per agent, set to pause automatically, after a few days of real numbers.
- Review the ten most recent failed runs every day until the error rate is stable.
- Write down who may pause an agent, and how, before you need to.
Frequently asked questions
What should I monitor for an AI agent in production?
Liveness, error rate, latency, tokens and cost, and answer quality for each agent, plus a record of every prompt and model change.
How is monitoring an agent different from monitoring a server?
Agent failures are mostly logical (bad prompts, degraded providers, loops), so they show up in per-run data such as errors, tokens and answers, not in server metrics.
Can an AI agent be paused automatically when it misbehaves?
Yes. In Agent Control Panel, alert rules on a daily spend cap, error rate, spend anomaly or lost heartbeat can attempt an automatic pause, within the app’s control scope.
Do I still need a tracing tool?
For deep debugging of single runs, a tracing tool is useful. ACP focuses on per-agent health, cost and the ability to act, and can run alongside one.