Appearance
How status works
The agent only reports facts. It never decides whether something is healthy. The server reads each report, gives it a status of online, warning or error with a one-line reason, stores it, and decides whether to raise an alert. This page explains those rules so you can predict when you'll be paged.
Reports, services and systems
Every plugin report arrives at PUT /api/clientstate/update with three headers: the system key, a type such as docker, and the plugin's uniqueId. The server keeps one row per system and uniqueId. That row is a service in the dashboard. Each new report replaces the row and adds a sample to its history.
Some plugins send several reports per cycle. docker sends one per compose project, web one per endpoint, lxc one per host that the server splits into one service per container.
From report to status
Each type has its own rules. The main ones:
| Type | error when | warning when |
|---|---|---|
os | CPU above 90%, or memory above 90% used | |
pm2 | the process status is errored | the process is in any other non-online state |
docker, lxc | a container is unhealthy, or stopped with a non-zero exit code | a container is stopped with exit code 0 |
web | the site is unreachable, the check failed, a forbidden string matched, or an expected string is missing | the origin check failed while the public URL worked, or the response took longer than 5000 ms |
wordpress | any posture finding, such as a new administrator or a crontab for the web user | a collector failed, so the posture is unknown |
logwatch | a finding the plugin marked as an error | a finding the plugin marked as a warning |
patroni | the node's API is unreachable, its state isn't running or streaming, the cluster has no leader, or lag has stayed over lagCritBytes for 5 minutes | Patroni is paused, lag has stayed over lagWarnBytes for 5 minutes, another member isn't running, or a restart is pending. See Clusters |
The reason names the specific thing, for example api exited (code 137), 4 restarts rather than 1/13 containers unhealthy. It shows on the service in the dashboard and in the alert text.
The dashboard maps these to its own levels. online is OK, warning is WARN and error is CRIT. Two more levels come from the dashboard, not from the server.
Offline
A service that stops reporting goes OFFLINE in the dashboard after 5 minutes. wordpress and Cloudflare services get 15 minutes, adsense and ga4 get 30, because they poll slowly.
Offline is a display state only. The server raises no alert when an agent goes quiet. A host that loses power stops reporting, and nothing pages. If you need to know when a host disappears, watch it from a second machine with a web or netscan check. A cluster rule with placement all also warns once a member has been silent for 15 minutes.
Standby
A host in a cluster that is correctly not running a clustered service shows STANDBY, not OFFLINE. The cluster card carries the real verdict.
When an alert is raised
A bad status turns the dashboard amber or red immediately. Raising an alert follows different rules depending on the kind of problem.
Single alerts wait for a second report
Most types, including os, pm2, docker, patroni and the others not listed below, raise one alert per service. The server holds that alert until the problem shows up in two consecutive reports. A container that goes unhealthy for 40 seconds and heals itself never pages. A real outage pages one interval later, which for a 60-second plugin costs a minute.
Findings alert at once, one per finding
web, wordpress and logwatch report security findings. One WordPress site can have a rogue administrator and a trashed login page at the same time, and each matters on its own. These types raise one alert per finding, with no second-report wait. Each finding has a stable key, such as the site plus the matched strings, so a new forbidden string on the same site alerts again even while the first alert is open.
Network devices
netscan, snmp and tplink reports that name the same IP address merge into one device service. The device takes the worst status across the plugins that watch it, and it alerts on the first bad report.
Clusters
A service declared in a cluster is judged across all members instead of per host, and waits for the cluster's grace period. See Clusters.
LXC containers
LXC containers show their status in the dashboard. They do not raise alerts.
One open alert at a time
An alert stays open until the condition clears. While it is open, the same problem on the same service raises nothing new. The server checks this on every report and enforces it with a unique index, so two reports racing each other can't open two alerts.
Reading an alert does not close it. Opening it in the inbox to go and look at the system marks it read and nothing else. Only recovery re-arms alerting, so investigating a fault never causes it to page again.
Auto-resolve on recovery
When a service reports online again, the server closes every open alert for that service. The alert stays in your history with the time it resolved. It just stops claiming to be current.
Findings close a little differently. On each report, the server closes any finding from that service that the report no longer contains. A forbidden string that disappears from the page closes its finding on the next check.
Re-alert cooldown
If an alert resolves and the same problem returns within 30 minutes, the server records the new event but does not send a push notification or a Slack message. The recurrence shows in the inbox and the event history. This stops a value hovering around a threshold from paging every few minutes. After 30 minutes, a recurrence pages normally.
Where to see what happened
Each alert is an event that belongs to its system. The system page lists unresolved events as active issues and keeps a paged history of the rest. Every event has its own page at /systems/<id>/events/<event id>, which names the finding, shows any before and after detail, and lists recurrences. Slack messages link straight to it.
History retention
Every report is stored as a history sample. A daily job at 02:00 UTC thins it out.
- The last 24 hours keep full resolution, up to 2000 samples per service.
- Older days keep 24 samples per service per day, roughly one an hour.
- Anything older than 30 days is deleted.
Charts for 7 and 30 days therefore show hourly points. adsense and ga4 are not sampled this way, because each report already carries its own 30-day series.
Separately, an hourly job deletes services that have stopped appearing while their agent keeps reporting. If the system has checked in within 15 minutes and a service on it has been silent for 2 hours, the service is removed. Removed containers, retired web checks and stopped pm2 processes leave the fleet on their own. If the whole system goes quiet, nothing is removed, so you can still see what the machine was running when it went down.