Systemd service not starting: the log already named the cause

■ the answer first

systemctl status + journalctl -u <unit> -n 50 --no-pager — one of five causes is named in the log.

203/EXEC = path, shebang, or exec bit — systemd launches ExecStart= without a shell.

Fails only at boot? An ordering race — add After=/Wants= and an ExecStartPre= guard.

THE TWO COMMANDS

systemctl status myapp.service
journalctl -u myapp.service -n 50 --no-pager

status gives you the state — inactive (dead), failed (exit-code), activating (auto-restart) — and the last few log lines. The journal gives the full story: stdout, stderr, and systemd's own complaints about the unit. Failed before and you want that run's record? Add -b for the current boot or --since today. One more pair worth knowing:

systemctl list-units --failed     # every red unit on the box
systemctl reset-failed myapp.service   # clear the failed state after the fix

verify: systemctl list-units --failed → 0 loaded units listed

THE FIVE CAUSES

After the log names the symptom, the cause is almost always one of these — check most likely first:

Log says	Likely cause	Fix
`status=203/EXEC`	Wrong `ExecStart=` path or missing interpreter	Absolute path, `chmod +x`, check shebang
`status=203/EXEC` on a script	CRLF line endings or bad shebang	`dos2unix script.sh`, fix first line
`status=1/FAILURE`, no app log	Working directory or env var missing	Set `WorkingDirectory=`, add `Environment=`
`Permission denied`	User can't read files or bind the port	Fix ownership; ports below 1024 need `AmbientCapabilities=CAP_NET_BIND_SERVICE` or root
`Unit is masked`	Someone ran `systemctl mask`	`systemctl unmask myapp.service`
Unit file edit "does nothing"	Daemon not reloaded	`systemctl daemon-reload`

THE 203/EXEC FAMILY

The one that eats afternoons. Systemd does not use your shell to launch ExecStart= — no ~ expansion, no PATH lookup, no cd. Watch the wrong lines become right:

ExecStart=~/app/run.shExecStart=/opt/app/run.shno shell → no tilde expansion, absolute path required

ExecStart=run.shExecStart=/opt/app/run.shno PATH lookup — relative paths die first

(script not executable, CRLF endings)chmod +x run.sh && dos2unix run.shthe script needs the exec bit and a real shebang: #!/bin/bash or #!/usr/bin/env bash

A script that runs fine from your terminal but fails with 203 under systemd is nearly always one of those four: not executable, CRLF endings, a shebang pointing nowhere, or a relative path.

STARTS BY HAND, DIES AT BOOT

Boot-time failure is ordering, not syntax. The classic race: your service reaches for the network, a mounted disk, or a database before that thing exists. Declare dependencies instead of hoping:

[Unit]
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
ExecStartPre=/usr/bin/test -f /opt/app/config.toml

After= orders the start; Wants= makes systemd actually bring the dependency up. network-online.target only works if the network-wait service is enabled on your distro — check systemctl is-enabled NetworkManager-wait-online.service (or the systemd-networkd equivalent). The ExecStartPre= guard fails loudly with a readable message instead of a stack trace.

verify: systemctl is-enabled NetworkManager-wait-online.service → enabled

The second variant: starts on boot but immediately dies. Look for things your interactive environment had that boot doesn't — PATH differences, a virtualenv, a HOME. Systemd starts units with a stripped environment; anything your shell supplies implicitly must be spelled out:

[Service]
Environment="PATH=/opt/app/venv/bin:/usr/bin"
User=appuser
WorkingDirectory=/opt/app

For genuine mysteries, run the exact ExecStart= command as the unit's user in a shell — most environment differences surface in the first ten seconds.

NOTHING IN THE JOURNAL?

StandardOutput= and StandardError= in the unit — they must be journal (the default) or journal+console. Someone may have set them to null.
The app writes to a file instead of stdout. Systemd only captures stdout/stderr; log-file writers bypass the journal entirely. Point the app at stdout or read the file.
Storage limits dropped old lines: journalctl --disk-usage, and SystemMaxUse= in /etc/systemd/journald.conf if the journal is choking.

RESTART AFTER A CRASH

Default policy is Restart=no: a crashed service stays dead, and you find out from a user. Fix that per service:

[Service]
Restart=on-failure
RestartSec=5

verify: systemctl show myapp.service --property=Restart → Restart=on-failure

Setting	Restarts when
`no` (default)	Never
`on-failure`	Non-zero exit, signal, timeout
`always`	Any exit, even clean
`on-watchdog`	Watchdog timeout only

Pair Restart=on-failure with a start-rate guard so a crashing loop doesn't hammer the box: StartLimitIntervalSec= and StartLimitBurst= in the [Unit] section. Five failures in sixty seconds should page a human, not spin a CPU. And if you're wiring this up as a scheduled job rather than a daemon, timers give you journal logging and dependency ordering for free — weigh cron against a timer first.

THE 60-SECOND CHECKLIST

systemctl status <unit> — read the state and last lines.
journalctl -u <unit> -n 50 --no-pager — find the real error.
203/EXEC? Fix path, shebang, permissions. Permission denied? Fix user and file ownership.
Fails only at boot? Add After=/Wants=network-online.target and an ExecStartPre= guard.
Change a unit file? systemctl daemon-reload && systemctl restart <unit>.
Add Restart=on-failure so the next crash announces itself instead of hiding.

Most "systemd is complicated" moments reduce to the error was in the journal the whole time. Read it before you edit the unit file, not after.

■ honest stamp Tooling: systemd · journalctl · unit files. First published 2026-09-14, record refreshed 2026-10-04. The five causes, the 203/EXEC mechanics and the restart policies are systemd's documented behaviour — every unit snippet above is the shape that actually ships on boxes, typed from units that failed and came back.

end of the five causes · nothing fetched · everything local