Sentinel — Remote Alerting

Sentinel pushes an alert to the people who run an installation the moment the engine stops or falls into a crash loop. It is designed for permanent installations, digital signage, and unattended attractions that run for months or years without an operator present.

Where Auto-Heal & Watchdog recovers the system locally, Sentinel notifies you remotely. The two are complementary: Auto-Heal fixes what it can on the box, and Sentinel tells you about anything that needs a human.

How It Works

Exaplay already collects the signals that matter — crash events in the Watchdog, FPS/CPU/RAM/VRAM in Auto-Heal, machine load in the service. Sentinel watches the liveness signals and, when the engine stops or keeps crashing, sends a notification to your VIOSO Cloud account.

Alerts come from one place:

• The Watchdog service raises liveness alerts (engine down, crash loop). It is the only component guaranteed to be running when the engine itself has crashed, so it owns the alerts that matter most.

The engine used to raise a set of soft alerts as well (low FPS, low disk, missed schedule). All three are retired — see Retired alerts.

Alert Events

Event	Severity	Raised when
engine.down	🔴 Critical	The Watchdog detects the engine process has stopped
engine.crashloop	🔴 Critical	3 crashes within 60 s — automatic restart is disabled

Each event type can be enabled or disabled independently. Both are owned by the watchdog service, so they keep working when the engine is gone and they do not depend on Auto-Heal being enabled.

There is no engine.restored alert: recovery is derived from the next heartbeat instead. engine.running = true closes the open engine.down / engine.crashloop incident, and operators subscribed to Resolved Alerts on the Cloud get that notification — one resolution path rather than a second alert Exaplay has to wire up and throttle separately.

Delivery

There is one transport: every alert goes to your VIOSO Cloud account over the machine's pairing. Sentinel has no channels of its own — no webhook URL, no mail server, no recipient list on the machine.

Where the notifications reach you is a Cloud account setting, not an Exaplay setting. Exaplay's job ends at delivering the event. Who gets told, through which medium, and which subscriptions apply (for example Resolved Alerts) is configured once in your Cloud account and covers every paired machine, instead of being re-entered on each box.

Sentinel therefore requires an active pairing. Without one there is nowhere for an alert to go: it is queued on disk and delivered if and when the machine is paired. See VIOSO Cloud → Pairing a machine.

If the network is down

Alerts are not lost when the link is down. A failed send lands in an on-disk retry queue:

Situation	What happens
No route / DNS / TLS failure, 5xx, 429	Retried with a backoff of 30 s → 60 s → 2 min → 5 min → 15 min, then every 15 min
Token invalid, revoked, or instance mismatch (401 / 403 / 409)	The queue parks for an hour instead of hammering a connection that cannot work — reconnect the machine to resume
Malformed payload (422)	Dropped, and logged. Retrying a client bug forever helps nobody
Queue limits	1000 items max; anything older than 7 days is discarded

Two consequences worth planning around: an alert can arrive later than the event it describes (the timestamp in the envelope is when it happened, not when it was delivered), and a machine that was offline will flush its backlog on reconnect. The heartbeat works the other way — it is never queued, because a stale liveness signal is worse than a missing one; a failed heartbeat is dropped and the next interval tries again.

Turning the Sentinel master switch off holds queued alerts on disk rather than discarding them; they flush once it is back on. The heartbeat is independent of that switch (see the note under Configuration).

Pairing comes first

Sentinel is not configured with a destination — it inherits one. Linking a machine to the Cloud is a one-time login that is separate from Sentinel, because the same connection also carries heartbeat monitoring, remote access and the Exaplay Cloud media library. It therefore lives behind the account icon to the left of the Header menu rather than inside the Sentinel dialog. Once paired, that icon shows the site initials.

→ VIOSO Cloud — pairing a machine covers the flow, the connection states, and how to unpair.

The Sentinel dialog shows a read-only connection badge with a Manage connection shortcut to that dialog, so you can tell at a glance whether alerts can currently be delivered.

Setting Up Alerts

1. Pair the machine with the Cloud if it is not paired yet.
2. Open the Sentinel Alerts entry in the Header menu.
3. Toggle Sentinel on.
4. Choose which events you want to be alerted about, and set the cooldown.
5. Click Save preferences, then use Send test alert in the account dialog to verify, on-site, that the message actually arrives before you leave.

Configuration

Setting	Description	Default
Enabled	Master on/off for all alerting	Off
Cooldown	Minimum seconds between repeats of the same event	300
Site name	Label every alert is signed with (e.g. Main Lobby — Acme HQ). Entered when pairing, not here — it is required before a machine can pair	(none; falls back to Exaplay Installation)
Planned-stop grace (sentinel.planned-stop-grace-sec)	Window after a service-initiated engine stop/restart in which an engine exit is treated as planned rather than as a crash (10–600 s)	90

Enabled and Cooldown live in the Sentinel Alerts dialog in the Header menu. The site name belongs to the pairing dialog behind the account icon — see VIOSO Cloud.

Note: the Enabled switch governs alert delivery only. The Cloud heartbeat — the periodic liveness signal the watchdog service sends once an instance is paired — is independent of this switch and cannot be turned off on its own. As soon as a pairing is active the service sends an immediate heartbeat (carrying engine info) and then keeps heartbeating in the normal rhythm for the life of the pairing; unpairing the instance is what stops it. The same applies to the engine-side half: the engine refreshes its status bridge (engine-status.json, see below) for every paired instance, whether or not alerting is on. Gating the writer on this switch used to freeze the engine-side heartbeat fields while the heartbeat kept resending the last snapshot.

Throttling

To prevent alert storms (for example, an engine that crashes repeatedly), Sentinel applies a cooldown per event type: once an event fires, the same event will not fire again until the cooldown elapses.

Planned Restarts & Reboots

engine.down is meant for unexpected exits. Planned downtime is recognised and does not alert:

• System shutdown / reboot / logoff — including the scheduled reboot — kills the engine before it can write its graceful-exit marker. The watchdog detects the ending session and treats the engine exit as planned: no engine.down, no crash counting, no restart attempt. (Previously this produced a false down → resolved pair on the Cloud after every machine restart, because the queued alert was delivered right after boot and the first heartbeat then resolved it.)
• Service-initiated stop/restart (tray menu, /wdog commands, multi-client Start/Restart Exaplay) arms the planned-stop grace window. Any engine exit inside that window — even one that needed the force-terminate fallback — is treated as planned. The window is consumed as soon as the engine is seen running again, so a genuine crash of the relaunched instance still alerts normally. The default of 90 s covers the worst-case teardown (20 s close wait plus the engine's 60 s shutdown failsafe); extend it via sentinel.planned-stop-grace-sec if your shows tear down slower.

The Cloud heartbeat pauses for the duration of a reboot and resumes immediately once the service is back (the first post-boot heartbeat is sent right away). Whether that gap raises an instance offline incident is decided by the Cloud's own offline threshold, which should be set comfortably above a normal reboot duration.

Sentinel, Auto-Heal & the Scheduler

Sentinel observes the other unattended-operation features rather than replacing them:

• When the Watchdog restarts the engine, Sentinel sends engine.down; the next heartbeat resolves it.
• When a crash loop disables auto-restart, Sentinel sends engine.crashloop so a human knows to intervene.
• The Scheduler and the Auto-Heal health checks no longer feed Sentinel — see Retired alerts for what they used to send and where that information lives now.

See Auto-Heal & Watchdog, the Scheduler, and the Permanent Installations workflow for the complete unattended-operation picture.