Systemd Service Not Starting? How to Fix It

A systemd service that won’t start is almost never mysterious. Run systemctl status <unit>, then read the last 50 journal lines for that unit with journalctl -u <unit> -n 50 --no-pager — between the two, one of five causes is usually named outright: a bad path, a missing binary, wrong permissions, an SELinux/AppArmor denial, or a unit file syntax error. The failure reason is in the log; the fixes below are just pattern matching against it.

How do I see why a systemd service failed?

Status first, journal second:

systemctl status myapp.service
journalctl -u myapp.service -n 50 --no-pager

status gives you the state (inactive (dead), failed (exit-code), activating (auto-restart)) and the last few log lines. The journal gives you the full story: stdout, stderr, and systemd’s own complaints about the unit.

If the unit failed before and you want the record of that run, add -b for the current boot or --since today:

journalctl -u myapp.service -b --no-pager

For the full toolkit — boots, priorities, following output live — see the journalctl cheat sheet. It’s the same muscle memory.

One more pair worth knowing:

systemctl list-units --failed
systemctl reset-failed myapp.service

The first lists every red unit on the box. The second clears the failed state after you’ve fixed it — cosmetic, but it stops status pages from crying wolf.

What are the most common causes?

After the log names the symptom, the cause is almost always one of these five:

Log says	Likely cause	Fix
status=203/EXEC	Wrong ExecStart= path or missing interpreter	Absolute path, chmod +x, check shebang
status=203/EXEC on a script	Script has CRLF line endings or bad shebang	dos2unix script.sh, fix first line
status=1/FAILURE, no app log	Working directory or env var missing	Set WorkingDirectory=, add Environment=
Permission denied	User can’t read files or bind the port	Fix ownership; ports below 1024 need AmbientCapabilities=CAP_NET_BIND_SERVICE or root
Unit is masked	Someone ran systemctl mask	systemctl unmask myapp.service
Unit file edit “does nothing”	Daemon not reloaded	systemctl daemon-reload

The 203/EXEC family deserves a special mention because it’s the one that eats afternoons. Systemd does not use your shell to launch ExecStart=. That means:

# Wrong — no shell, ~ never expands, no PATH lookup
ExecStart=~/app/run.sh

# Right
ExecStart=/opt/app/run.sh

And the script itself must be executable and start with a real shebang (#!/bin/bash or #!/usr/bin/env bash). A script that runs fine from your terminal but fails with 203 under systemd is nearly always one of: not executable, CRLF endings, a shebang pointing nowhere, or a relative path.

Why does it start manually but not on boot?

The classic ordering bug. If the log shows failures right after boot but the unit starts fine when you systemctl start it by hand, your service is losing a race — it’s reaching for the network, a mounted disk, or a database before that thing exists.

The fix is to declare dependencies instead of hoping:

[Unit]
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
ExecStartPre=/usr/bin/test -f /opt/app/config.toml

After= orders the start; Wants= makes systemd actually bring the dependency up. network-online.target only works if the network-wait service is enabled on your distro, so check systemctl is-enabled NetworkManager-wait-online.service (or the systemd-networkd equivalent). The ExecStartPre= guard is a cheap, honest way to fail loudly with a readable message instead of a stack trace.

A second variant: the service starts on boot but immediately dies. Look for things your interactive environment had that boot doesn’t — PATH differences, a virtualenv, a HOME. Set what you need explicitly:

[Service]
Environment="PATH=/opt/app/venv/bin:/usr/bin"
User=appuser
WorkingDirectory=/opt/app

Why is my service not logging to the journal?

If journalctl -u shows nothing, check three things in order:

1. StandardOutput= and StandardError= in the unit — they must be journal (the default) or journal+console. Someone may have set them to null.
2. The app writes to a file instead of stdout. Systemd only captures stdout/stderr; log-file writers bypass the journal entirely. Either point the app at stdout or read the file.
3. Storage limits dropped old lines: journalctl --disk-usage, and SystemMaxUse= in /etc/systemd/journald.conf if the journal is choking.

For debugging the start itself, nothing beats dropping a shell into the boot-time context:

[Service]
ExecStart=/bin/bash -c 'exec /opt/app/bin/server 2>&1'

Or, for genuine mysteries, run the exact ExecStart= command as the unit’s user in a shell — most environment differences surface in the first ten seconds.

How do I make it restart automatically after a crash?

Default policy is Restart=no: a crashed service stays dead, and you find out from a user. Fix that per service:

[Service]
Restart=on-failure
RestartSec=5

Setting	Restarts when
no (default)	Never
on-failure	Non-zero exit, signal, timeout
always	Any exit, even clean
on-watchdog	Watchdog timeout only

Pair Restart=on-failure with a start-rate guard so a crashing loop doesn’t hammer the box: StartLimitIntervalSec= and StartLimitBurst= in the [Unit] section. Five failures in sixty seconds should page a human, not spin a CPU.

If you’re wiring this up as a scheduled job rather than a daemon, weigh cron vs systemd timer first — timers give you journal logging and dependency ordering for free, which is exactly what this article keeps reaching for.

The 60-second checklist

1. systemctl status <unit> — read the state and last lines.
2. journalctl -u <unit> -n 50 --no-pager — find the real error.
3. 203/EXEC? Fix path, shebang, permissions. Permission denied? Fix user and file ownership.
4. Fails only at boot? Add After=/Wants=network-online.target and an ExecStartPre= guard.
5. Change a unit file? systemctl daemon-reload && systemctl restart <unit>.
6. Add Restart=on-failure so the next crash announces itself instead of hiding.

Most “systemd is complicated” moments reduce to the error was in the journal the whole time. Read it before you edit the unit file, not after.