Auto-Heal & Watchdog

Exaplay 3 includes a built-in crash recovery system designed for unattended operation in permanent installations, trade-show booths, and touring productions that run 24/7.

How It Works

There are two halves, and they are switched on separately:

1. Saving and restoring state. Exaplay continuously writes its playback state to disk. When it starts with -autoheal-restore, it reopens the show and picks playback up at approximately the position where it stopped. This half is on by default (Auto-heal in Settings → Autoheal).
2. Restarting the engine. The Watchdog service can relaunch the engine after a crash or a hang. This half is off by default — turn on Watchdog restart in Settings → System.

Restarting is opt-in

Out of the box the Watchdog observes and alerts but does not restart Exaplay, and with it off a detected hang is reported and left alone rather than the engine being terminated. For an unattended installation that should recover on its own, turn Watchdog restart on — otherwise only the state restore applies, on the next start you perform.

Skipping the restore for a session

Launch Exaplay 3 with the -fresh flag — note that -fresh is a full safe mode: besides ignoring auto-heal data it also skips the startup project and the last-project autoload, so the engine comes up with an empty project.

Watchdog Service

The Watchdog is installed as a Windows Service and starts automatically with Windows. You do not need to interact with it directly.

When restarts are enabled (Watchdog restart in the System settings), repeated crashes escalate instead of looping: the first restart restores playback (-autoheal-restore), a second short-lived run restarts in safe mode (-fresh, empty project), and after a third the Watchdog stops restarting and raises a critical alert so a person can investigate. A run counts as part of a crash loop when the engine died within a few minutes of starting — however long booting the show took. The tray menu also offers Start Exaplay (safe mode, no project) for starting safe mode manually.

To check its status:

Get-Service -Name "Exaplay3Watchdog"

To restart it manually:

Restart-Service -Name "Exaplay3Watchdog"

Auto-Heal Configuration

In Menu → Settings → Autoheal:

Setting	Description	Default
Auto-heal	Crash recovery and playback resume	On
State save interval	How often playback state is written to disk (s)	2
Health check interval	How often the engine writes its health heartbeat (s)	5
Diagnostics interval	How often the Watchdog reads that heartbeat (s)	10
Max state age	Ignore a saved state older than this (s)	3600
Verbose logging	Detailed auto-heal logs	Off
Notify on display changes	Warn when the monitor topology changes	On

In Menu → Settings → System:

Setting	Description	Default
Watchdog restart	Let the Watchdog relaunch the engine after a crash or hang	Off
Watchdog restart wait	Seconds to wait before relaunching (0–120)	5

The hang timeout — how stale the engine's heartbeat must get before the Watchdog calls it hung — is autoheal.hang-timeout in config_system.json, 30 s by default (clamped to 10–300).

Manual Recovery

If the system does not recover automatically:

1. Check the Watchdog Service status (see above).
2. Look at the log file in Documents\Exaplay3\logs\ for the error.
3. Start Exaplay 3 manually; Auto-Heal will still restore the last saved state.

Skipping Auto-Heal Restore

To start with a clean state (ignoring any saved auto-heal data), launch Exaplay 3 with:

exaplay3.exe -fresh

This is useful after a major project change or when debugging.