Auto-Heal & Watchdog

Exaplay 3 includes a built-in crash recovery system designed for unattended operation in permanent installations, trade-show booths, and touring productions that run 24/7.

How It Works

There are two halves, and they are switched on separately:

1. Saving and restoring state. Exaplay continuously writes its playback state to disk. When it starts with -autoheal-restore, it resumes playback at approximately the position where it stopped. This half is on by default (Auto-Heal).
2. Restarting the engine. The Watchdog can relaunch the engine after a crash or a hang, and it is the Watchdog that starts it with -autoheal-restore. This half is off by default. Turn on Watchdog restart to use it.

Both switches are in Menu → Auto-Heal & Watchdog.

Restarting is opt-in

Out of the box the Watchdog observes but does not restart Exaplay, and with it off a detected hang is logged and left alone rather than the engine being terminated. Only the Watchdog starts the engine with -autoheal-restore, so with restarting off a saved state is not resumed when you start Exaplay yourself. For an unattended installation that should recover on its own, turn Watchdog restart on.

The restore resumes playback inside the project that loads at startup (the startup project or the last-project autoload). If a different project loads, or the saved state is older than Max state age, nothing is restored and the normal startup behaviour (for example auto-play) applies.

The Auto-Heal & Watchdog dialog shows the same two switches, and they are just as independent there: switching one never moves the other. With Auto-Heal on and Watchdog Restart off, the dialog says so — the state is saved, but only a start with -autoheal-restore resumes it.

Skipping the restore for a session

Launch Exaplay 3 with the -fresh flag — note that -fresh is a full safe mode: besides ignoring auto-heal data it also skips the startup project and the last-project autoload, so the engine comes up with an empty project.

Watchdog

The Watchdog is part of Exaplay 3 Service (exaplay3-service.exe), a background program with its own tray icon. It is not a Windows service: the installer puts a shortcut into the Windows Startup folder, so it starts when the user logs on. An unattended machine therefore needs automatic Windows logon. You can also start it from the Start menu (Exaplay 3 Service).

When restarts are enabled (Watchdog restart), repeated crashes escalate instead of looping: the first restart restores playback (-autoheal-restore), a second short-lived run restarts in safe mode (-fresh, empty project), and after a third the Watchdog stops restarting and raises a critical alert so a person can investigate. A run counts as part of a crash loop when the engine died within 5 minutes of starting, however long booting the show took. A run that stays up longer, or a clean exit, ends the episode. The tray menu also offers Start Exaplay (safe mode, no project) for starting safe mode manually. On a multi-client client, safe mode is reported, not repaired: the client tells its master it is in safe mode, the Network tab shows it as in safe mode (Loaded show: none — safe mode) and the fleet as not in sync, and Auto Sync does not load the master's show on it — if that show is what crashed the client, loading it again would be the third crash, after which the Watchdog stops restarting. Find the cause, then load the show on that client by hand or restart Exaplay there; Auto Sync resumes once it runs a show again.

The Watchdog also detects two kinds of hang:

• No heartbeat. The engine writes a health heartbeat every few seconds. If it is older than autoheal.hang-timeout (30 s by default, 10–300 s), the engine counts as hung. Opening media during a project load keeps the heartbeat alive, so a long load is not mistaken for a hang.
• No picture. The heartbeat keeps coming but the renderer has not finished a frame for autoheal.render-hang-timeout (60 s by default, 20–600 s; 0 turns this check off). Project loads are excluded.

On a hang the Watchdog writes a memory dump of the engine to Documents\Exaplay3\logs\crash\ (exaplay3-hang-*.dmp). With Watchdog restart on it then terminates the engine and restarts it; with it off it leaves the engine running.

To check that the Watchdog is running, look for its tray icon or for exaplay3-service.exe in Task Manager → Details. To restart it, right-click the tray icon, choose Quit Exaplay Service, and start Exaplay 3 Service from the Start menu again.

Auto-Heal Configuration

In Menu → Auto-Heal & Watchdog (Save Interval, Diagnostics Interval, Max State Age, Verbose Logging and Notify Output Changes show while Auto-Heal is on; Restart Wait while Watchdog restart is on):

Setting	Description	Default
Watchdog Restart	Let the Watchdog relaunch the engine after a crash or hang	Off
Restart Wait	Seconds to wait before relaunching (0–120)	5
Auto-Heal	Save playback state, and resume it after a Watchdog restart	On
Save Interval	How often playback state is written to disk (s)	2
Health Interval	How often the engine writes its health heartbeat (s)	5
Diagnostics Interval	How often the engine checks its displays, missing media and frame rate (s, at least 5). Runs only while Auto-Heal is on	10
Max State Age	Ignore a saved state older than this (s)	3600
Verbose Logging	Detailed auto-heal logs	Off
Notify Output Changes	When the number of connected outputs changes, write it to the engine log and show it in the Monitor as a Display event while the Monitor is recording. Off: the change is not reported. Checked on the Diagnostics Interval, while Auto-Heal is on	On

Keep Health Interval well below the hang timeout (30 s), or the Watchdog reads a healthy engine as hung.

The two hang timeouts are not in the dialog. Set them as autoheal.hang-timeout and autoheal.render-hang-timeout in Documents\Exaplay3\config\config_system.json.

Scheduled Reboot

Menu → Auto-Heal & Watchdog → Scheduled Reboot restarts Windows once a day at Reboot time (hh:mm, 24-hour, default 03:00; off by default). The Watchdog stops the engine first, and the reboot does not raise a Sentinel alert.

Manual Recovery

If the system does not recover automatically:

1. Check that Exaplay 3 Service is running (see above).
2. Look at the log files in Documents\Exaplay3\logs\ for the error.
3. Start Exaplay 3. A normal start does not resume the saved state; to resume it, start exaplay3.exe -autoheal-restore.

Skipping Auto-Heal Restore

To start with a clean state (ignoring any saved auto-heal data), launch Exaplay 3 with:

exaplay3.exe -fresh

This is useful after a major project change or when debugging.

Not to be confused with

• Auto Sync in multi-client setups repairs media that is missing or different on a client. See Multi-Client → File sync.
• Main / Backup failover puts a backup machine on air when the master is lost. See Multi-Client → Main / Backup Failover.