a stop job is running — the 90-second fuse nobody lit

"A stop job is running for X" means the stop of unit X already began and is waiting: SIGTERM went out, nothing came back, and the job waiter is counting down a fuse. Read the fuse: systemctl show <unit> -p TimeoutStopUSec → 90s. Fix it at the source — handle SIGTERM (measured 0.03s) — or shrink the fuse with a drop-in: TimeoutStopSec=10 (measured 10.10s). At the end of the fuse the escalation is automatic: SIGKILL, Failed with result 'timeout'.

Tested on a throwaway Arch Linux VM, cloud image, systemd 262, last tested 2026-11-08. Every number below is lifted from that run, unedited — one stubborn service, stopped three ways, stopwatch on each stop.

Two siblings from this desk covered the other fuses: systemd service restart on failure — the hidden breaker (the rate limit that stops a crashing unit from restarting too often) and systemd unit is masked — the dev-null execution (the start that never happens and never logs). This is the third fuse — the stop side, the one you watch count to 1:30 on a shutdown that went badly.

What does "a stop job is running" actually mean?

Stopping a unit is a negotiation with a deadline. The moment you (or shutdown) ask, systemd sends SIGTERM — signal 15 — to every process in the unit's control group, and starts counting. The line on your screen is that count. From the same bench run, taken six seconds into the stop:

$ systemctl list-jobs
   JOB UNIT             TYPE STATE
2558 stubborn.service stop  running

$ systemctl is-active stubborn.service
deactivating

stop + running in the jobs table is the machine's own copy of the line you are watching. Nothing is broken in this state. Everything is waiting — the difference matters, because the fix is never "press Ctrl+C harder". The waiter counts to the unit's fuse, and on a default install that fuse is the same for every unit you did not explicitly set one for.

Why does it always take 1min 30s?

Because the default fuse is 90 seconds, and almost nobody has ever set it. One command prints it for any unit:

$ systemctl show stubborn.service -p TimeoutStopUSec -p KillSignal -p SendSIGKILL
TimeoutStopUSec=90s
KillSignal=15
SendSIGKILL=yes

The bench service — a loop that sets SIGTERM to ignore, as databases and JVMs legitimately do mid-write — was stopped with the stopwatch on. Result: 90.20s. The fuse, to the second, plus scheduling dust. The journal keeps the whole escalation as four lines:

$ journalctl -u stubborn.service --no-pager | tail -4
systemd[1]: stubborn.service: State 'stop-sigterm' timed out. Killing.
systemd[1]: stubborn.service: Killing process 894 (python3) with signal SIGKILL.
systemd[1]: stubborn.service: Main process exited, code=killed, status=9/KILL
systemd[1]: stubborn.service: Failed with result 'timeout'.

verify: journalctl -u <unit> after any slow stop — status=9/KILL plus result 'timeout' is a fuse burning to its end, straight from the bench log.

KillSignal 15 is the polite ask; SIGKILL is the end of the negotiation. SIGTERM is a question, and ninety seconds later systemd stops asking.

Which unit is the line stuck on?

The line names the unit, and the search bar's eight variants of this query are the same waiter over different tenants. What the fuse is actually waiting on, per tenant:

the line says	the unit behind it	what the fuse is waiting on
for session / user manager	`[email protected]`	a session process that ignores the ask — often a hung terminal or a background job
for unattended-upgrades	`unattended-upgrades.service`	an upgrade mid-flight holding the package lock — interrupting it is the worse outcome
for snap daemon	`snapd.service`	shutdown hooks across many installed snaps
for zabbix / a custom service	your unit	an `ExecStop` that never finishes — or no handler at all
for anaconda	the installer's teardown	its own shutdown script, on the way out

The fuse belongs to the unit, not to the machine: systemctl show <that-unit> -p TimeoutStopUSec names the exact number the line is counting down for your case.

Why does a service ignore the polite ask?

Three ordinary reasons. The process deliberately set SIGTERM to ignore — a database guarding a checkpoint, a JVM mid-compaction. The process never installed any handler and its startup is slow, so the ask lands in a blind window. Or the unit has an ExecStop that hangs, and the fuse times the hang. systemd does not read minds: KillSignal=15 goes out to the whole control group, nothing comes back, the fuse burns. The mid-stop state you saw above — deactivating, one stop job running — is exactly this: healthy mechanism, deaf tenant.

How do I make the stop instant?

Cure A: answer the door. The bench's second service is the same loop with one handler added:

import signal, sys

def bye(signum, frame):
    sys.exit(0)

signal.signal(signal.SIGTERM, bye)

Stopped with the same stopwatch: 0.03s. The journal reads like the unit was never asked at all:

systemd[1]: Stopping Gracious (handles SIGTERM)...
systemd[1]: gracious.service: Deactivated successfully.
systemd[1]: Stopped Gracious (handles SIGTERM).

Three hundredths of a second versus ninety seconds — the whole 3000× gap is one function call that returns. For your own services, the handler should checkpoint and exit; for units you do not own, this cure is a patch to their code, which is why the second cure exists.

How do I change the 90 seconds?

Cure B: set a fuse you chose, as a drop-in — never by editing the unit file itself:

$ sudo systemctl edit stubborn.service
# in the editor, under [Service]:
TimeoutStopSec=10

$ systemctl show stubborn.service -p TimeoutStopUSec
TimeoutStopUSec=10s

Same service, same refusal to answer, chosen fuse: 10.10s measured, the same four escalation lines printed at the 10-second mark. The number to pick: 5–15s for units that normally answer, the 90s default is only right for units with a long, legitimate teardown, and TimeoutStopSec=infinity is legal and almost always a mistake you will rediscover during a datacenter power test.

verify: systemctl show <unit> -p TimeoutStopUSec before and after the drop-in — the live fuse, printed twice.

The 60-second cure, in order

Reproduce the count: sudo systemctl stop <unit> — let it hang, watch the line.
Read the fuse: systemctl show <unit> -p TimeoutStopUSec — 90s means nobody ever set it.
Name the deaf tenant: journalctl -u <unit> --since '-5 min' — status=9/KILL + result 'timeout'.
Answer the door if you own the code: SIGTERM handler, checkpoint, exit — measured 0.03s.
Else shrink the fuse: sudo systemctl edit <unit>, TimeoutStopSec=10 — measured 10.10s.

verify: the next stop is sub-second or fuse-length — on the bench machine, 90.20s never happened twice.

One thing the bench cannot print: the counting line itself lives on the console, not in the journal — the journal's copy is the four escalation lines above. If you arrived from a screenshot of your own boot screen, the receipts above are that line, timed to the second. The line was never a hang. It is a fuse you can read, and now you can also name who lit it.