systemd service status 203 — the exec that never ran

tl;dr

systemd service status 203 prints as status=203/EXEC: the kernel's execve call failed, and your program never ran a single line. It is not a crash — it is a refusal at the door. Measured on systemd 262: a wrong path, a missing execute bit, and a broken shebang — three different mistakes — landed the identical ExecMainStatus=203, seconds apart, on the same VM. The journal's second line names which of the three is yours.

203/EXEC always means one thing: systemd tried to hand your ExecStart command to the kernel and the kernel refused it before your code existed as a process. The journal line under the failure tells you which door refused: Unable to locate executable … No such file or directory = wrong path; Unable to locate executable … Permission denied = missing exec bit; Failed to execute … No such file or directory = the interpreter your shebang names doesn't exist. Fix that one thing — path, chmod +x, or an honest line 1 — and the unit runs. The bench cure took one chmod and one shebang edit: Result=success.

Last tested 2026-11-11 · systemd 262 (262.1-1-arch) · fresh Arch VM. Every line below is pasted from that run.

Fig 1 — the exec relay. status=203/EXEC is the baton dropping at door 1, 2 or 3; the box at the bottom never started.

What does status 203/EXEC actually mean?

The claim sentence: status=203/EXEC is systemd reporting that the execve of your ExecStart command failed at spawn time — your process never existed, so nothing about your program's logic, config, or code is implicated yet. On the bench box, one throwaway oneshot unit wearing each fault in turn printed the same verdict every time:

$ systemctl show bench203.service -p ExecMainCode -p ExecMainStatus -p Result
ExecMainCode=1
ExecMainStatus=203
Result=exit-code

verify: ExecMainCode=1 is CLD_EXITED and 203 is the exec-family exit status; the same unit cured reads ExecMainStatus=0 and Result=success (receipt below).

Three different mistakes, one number

Three ExecStart values, three starts on the same boot, seconds apart. systemctl status — the screen most people actually read — could not tell them apart:

Process: 703 ExecStart=/opt/bench203/no-such-binary (code=exited, status=203/EXEC)
Process: 778 ExecStart=/opt/bench203/worker.sh      (code=exited, status=203/EXEC)   # file exists, mode 0644
Process: 853 ExecStart=/opt/bench203/shebang.sh     (code=exited, status=203/EXEC)   # +x, interpreter missing

verify: three starts, three identical status=203/EXEC tails — the status screen shows the what, never the which.

The folk cure is reinstalling the program or restarting the unit in a loop. Restart loops are the lie here: the program was never the patient. The second line of the journal entry — the one under Starting… — is the diagnosis, and it differs all three times:

bench203.service: Unable to locate executable '/opt/bench203/no-such-binary': No such file or directory
bench203.service: Unable to locate executable '/opt/bench203/worker.sh': Permission denied
bench203.service: Failed to execute /opt/bench203/shebang.sh: No such file or directory

verify: pasted verbatim from journalctl -u bench203.service, one line per fault, PIDs 703 / 778 / 853.

Which of the three is yours?

Read the second line in two steps. First phrase: Unable to locate executable speaks about your path (fault A or B); Failed to execute means systemd got past your file and lost something it names inside it — the interpreter on line 1 (fault C). The errno after the colon: No such file or directory vs Permission denied splits A from B.

Fault B's phrasing deserves its own warning: Unable to locate executable … Permission denied accuses the file of not existing while the file sits right there — 28 bytes, mode 0644, minding its own business. systemd did locate it; it just can't say so in that phrase. The truth is the errno, never the verb.

Awrong path — Unable to locate … No such file or directory: the file genuinely isn't there

Bno exec bit — same Unable to locate phrase, errno Permission denied: the bit is the missing piece

Cbroken shebang — Failed to execute … No such file or directory: your file is fine, #!/usr/bin/bashh isn't

How do I fix status 203/EXEC?

The cure on the bench was one chmod for B's shape and an honest line 1 for C's shape — no reinstall, no restart loop:

sudo chmod 0755 /opt/bench203/worker.sh        # give back the exec bit
# and make line 1 of the script name a real interpreter:
#!/bin/bash
$ systemctl show bench203.service -p ExecMainStatus -p Result
ExecMainStatus=0
Result=success

verify: the cured run's journal reads worker.sh[931]: worker ran → Deactivated successfully. → Finished bench203 fault shapes. — the oneshot did its job and left.

September's systemd service not starting: the log already named the cause made the same promise for a quieter failure. 203 is that law at its loudest — the confession lands in the journal before you have even opened the unit file.

The cure, in order

systemctl status <unit> -l — confirm status=203/EXEC on the Main PID line
journalctl -u <unit> --no-pager — read the second line: Unable to locate (your path) vs Failed to execute (interpreter)
the errno after the colon — No such file or directory vs Permission denied
ls -l the path (the x bits) · head -1 the script (the interpreter named) · test -x that interpreter
fix the one thing, systemctl start again — show -p Result reads success

The three confessions at a glance

journal says	what's missing	the fix
Unable to locate executable … No such file or directory	the `ExecStart` path itself	correct the path — absolute, no typos
Unable to locate executable … Permission denied	the execute bit	`sudo chmod +x /path/to/script`
Failed to execute … No such file or directory	the shebang's interpreter	fix line 1 (`#!/bin/bash`) or install the interpreter

"203 isn't your program dying — it's systemd failing to hand it the keys. The journal names which key, every time."

what the bench did not cover

Measured: three fault shapes (missing path, mode 0644, broken shebang) on one fresh Arch VM, systemd 262 (262.1-1-arch), oneshot unit, seconds apart, plus the cured run. Not measured: a relative ExecStart path — systemd refuses those at parse time with its own message, not 203 (public record); SELinux/AppArmor denials, which surface as the same Permission denied errno — check your LSM before blaming the bit; chroot'd ExecStart paths. One box, one build: the reading rule is the portable part; your own journal line is the verdict.

HUMAN-VERIFIED BENCH · systemd 262 (262.1-1-arch) · fresh Arch VM · last tested 2026-11-11

One number, three doors, and a journal that names the door every time. Read the second line before you re-run anything. ■