systemd timer persistent — the missed 3am run

Why didn't my systemd timer run while the box was off?

Because a timer only fires when it is alive to see the moment. The claim sentence: a OnCalendar timer checks the clock at each elapse; a moment when the machine is powered off is a moment the timer never sees. Nothing is queued, nothing is journaled. The run doesn't fail — it never existed.

That's the part mills skip: there is no error. The 03:00 backup window passed while the box was off, and at 09:00 everything looks fine. The journal for the timer shows only the runs that happened. The shift notes are blank on purpose.

Systemd does keep one piece of paper: the stamp file. Verify the door on your own box:

ls -l /var/lib/systemd/timers/
# stamp-bench.timer  — one file per timer with Persistent=true
systemctl list-timers --all

verify: ls prints one stamp-<unit> per persistent timer; list-timers prints the next elapse for every timer.

What does Persistent=true change?

Exactly one thing, and it's mechanical: with Persistent=true, the stamp file records the last scheduled window the timer actually serviced; at activation systemd compares the stamp against the calendar and fires immediately if a window was missed. Without it, no stamp, no comparison, no catch-up.

The receipt from the bench, phase by phase — a one-shot logger service on a one-minute OnCalendar timer, then a window pinned to a chosen minute with the box powered off across it:

# the miss, armed: OnCalendar=2026-11-09 06:03:57 (UTC)
# persistent=true — poweroff 20 s before the window, dark through it
$FIRE $(date +%F_%T) >> /tmp/bench-fires.log

verify: /tmp/bench-fires.log held one line stamped 06:05:44 — the boot bootstrapped the catch-up, 107 s after the window the box slept through.

Run B was identical except Persistent=false: window pinned to 06:26:37, box dark through it, same boot dance. The stamp file did not move — mtime still 06:05:44, read 06:28:36. No fire, no journal line, no record anywhere. Total journal lines about the miss in both runs: zero. The stamp is the only witness either way — it updates when a window is serviced, and it stays silent when one is not.

"A plain timer doesn't skip and tell. Only the persistent one keeps receipts."

Does cron catch up missed runs?

No — and this is where the September piece (Cron vs systemd timers: cron runs lines, timers run units) matters again: plain cron drops a run that lands while the box is down; there is no queue, no catch-up, no record. The classic mitigations each carry a cost:

door	missed run while off	the cost
systemd timer, default	dropped, silently	no stamp, no catch-up
systemd timer, Persistent=true	fires at next boot	a stamp file per timer
cron	dropped, silently	nothing written down
cron + anacron	runs late, on boot	second scheduler to feed; daily granularity
cron @reboot	runs at every boot	not a catch-up — runs even when nothing was missed

anacron is the honest runner-up — many distros already wire cron.daily through it. But anacron speaks days, not minutes: it can run your daily job late, it cannot replay a missed hourly window.

How do I make a systemd timer catch up?

One line in the [Timer] section. The full cure for a nightly job, runnable as-is:

# /etc/systemd/system/backup.timer
[Unit]
Description=nightly backup
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
[Install]
WantedBy=timers.target

verify: systemctl daemon-reload && systemctl enable --now backup.timer && systemctl list-timers backup.timer prints the next 03:00 elapse.

• Persistent=true in [Timer] — the stamp file is what makes boot a catch-up
• WantedBy=timers.target in [Install] + systemctl enable --now — a timer nobody enabled catches up for nobody
• systemctl list-timers --all — every unit should print a next elapse; a dead timer prints nothing
• one service, one timer — the service stays Type=oneshot, the timer owns the schedule
• after the next boot, check the journal: the catch-up run sits at the top of journalctl -u <service>, stamped seconds after boot

How do I know a run was missed?

Three reads, cheapest first: the stamp file's mtime is the last window the timer serviced; list-timers prints the last and next elapse; the journal shows only the runs that fired. If the stamp says 02:5x and the calendar says the window was 03:00 — the window is gone and nothing ran. That's the whole forensic kit; the confession was never written anywhere else.

Catch-up is immediate but it is not instant-restore magic: the service runs once, not once per missed window. A timer that missed six hourly windows fires one catch-up, not six. If every window matters, that's a queue's job, not a timer's.

Two traps the bench itself stepped on, worth their own receipts. First, the silent name-binding: a bench.timer looks for bench.service — not bench-log.service — and refuses to start with a journal line that names the missing unit: Refusing to start, unit bench.service to trigger not loaded. Second, systemctl enable refuses any timer with no [Install] section — WantedBy=timers.target is not optional. Both failures printed to the journal, not to you.

what the bench did not cover

Measured: catch-up timing, the silence of the default, the stamp witness, the one-minute baseline — on one VM, systemd 262, UTC clock, windows at the minute. Not measured: AccuracySec jitter under load, RandomizedDelaySec, timezone-edge windows (DST), and wake-from-suspend (timers can wake the box via WakeSystem=true — a different door with its own firmware caveats). The stamp mechanism itself is documented in systemd.timer(5).

HUMAN-VERIFIED BENCH · systemd 262 · fresh Arch VM · last tested 2026-11-09

If your 3am job matters, say so in the unit file. ■