SSH Permission Denied (publickey): The Real Fix

At 02:10 a deploy died on a line I have typed a thousand times: ssh deploy@staging → Permission denied (publickey). The key was fine. The offer wasn't.

$ ssh -T [email protected]
[email protected]: Permission denied (publickey).

The error is a verdict, not a hint: everything in the offer was refused.

The 30-second triage

• ssh -vvv user@host 2>&1 | grep -i offering — which keys were offered?
• journalctl -u ssh -n 50 | grep -i publickey — why were they refused?
• ls -ld ~/.ssh ~/.ssh/authorized_keys — 700 / 600?
• ssh-copy-id user@host — install with sane permissions
• Key installed — run ssh -o BatchMode=yes user@host 'echo ok' again to confirm

Why does SSH say "Permission denied (publickey)"?

Public-key auth is an offer-and-refuse negotiation. The client proposes every key it can find — identities from your agent, your ~/.ssh/config, default filenames (id_ed25519, id_rsa) and anything passed with -i. The server checks each offer against the target account's ~/.ssh/authorized_keys. When none match, and password auth is disabled or already exhausted, the client prints the one line every devops engineer has memorised.

Two facts matter for debugging. First, the message does not say which user you hit — half of all cases are a perfectly good key sitting in the wrong account's authorized_keys. Second, the server already told you why: sshd logs every refused offer. The client-side stack trace nobody asked for and everybody needs is -vvv.

How do I find which key SSH is actually offering?

ssh -vvv deploy@staging 2>&1 | grep -iE "offering|identity|denied|authentications"
# debug1: Offering public key: /home/you/.ssh/id_ed25519 RSA-SHA256 (explicit)
# debug1: Authentications that can continue: publickey
# deploy@staging: Permission denied (publickey).

Read three lines: Offering public key (what the client proposed), Authentications that can continue (what the server will still accept — if password is absent, no password prompt will ever come), and the final verdict. If your intended key never appears in the offer list, the problem is client-side: wrong -i path, an IdentityFile in ~/.ssh/config you forgot about, or an agent (ssh-add -l) holding a stale identity that answers first.

verify: your intended key appears in the Offering public key lines — if not, stay client-side

What are the four real causes?

Match the server's exact line to a row — the fix is in the third column.

Cause	Telltale	Fix
Wrong user	Key works for root@host, fails for deploy@host	Key must be in that user's ~/.ssh/authorized_keys
Key not installed	Server log shows Failed publickey for every offer	ssh-copy-id user@host
Permissions too open	Client: WARNING: UNPROTECTED PRIVATE KEY FILE — the key is skipped, never offered	chmod 700 ~/.ssh; chmod 600 ~/.ssh/*
Legacy ssh-rsa key	Server runs OpenSSH 8.8+; old RSA key never accepted	Re-generate as ed25519, or upgrade the key
Right key exists, wrong one arrives	-vvv offers an identity you don't recognise	ssh-add -l to list, ssh-add -d to drop the stale one
-i mykey ignored	A config IdentityFile overrides the offer order	Fix or remove the ~/.ssh/config block

The permissions cause deserves its own note, because OpenSSH enforces it hard: a private key that group or world can read is refused by the client itself and silently drops out of the offer. The fix is two commands:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519 ~/.ssh/authorized_keys

verify: ls -ld ~/.ssh → drwx------ · ls -l ~/.ssh/id_ed25519 → -rw-------

The ssh-rsa cause trips older CI images and Raspberry Pis: OpenSSH 8.8 (released 2021-10-01, openssh.com/txt/release-8.8) disabled ssh-rsa signatures — the SHA-1-based variant — by default. Per the release notes, "the ssh-rsa signature scheme is ... disabled by default". An RSA key generated in 2018 is not broken; its signature format is just no longer spoken. Generating an ed25519 key (ssh-keygen -t ed25519) is the durable answer.

Afraid of locking yourself out mid-fix?

Cloud consoles give you a serial console that does not depend on sshd — VPS providers (DigitalOcean, Hetzner, AWS EC2 Instance Connect) all have one. If password auth is already disabled and your fix touches authorized_keys, keep that console open before you restart anything.

How do I see why the server refused the key?

The server already logged why it refused you.

The server-side log is the honest witness. On Ubuntu (systemd), sshd logs per-offer verdicts:

journalctl -u ssh -n 50 --no-pager | grep -iE "publickey|denied|accepted"
# sshd[4421]: Failed publickey for deploy from 203.0.113.7 port 51422 ssh2: RSA SHA256:...
# sshd[4421]: Accepted publickey for deploy from 203.0.113.7 port 51422 ssh2: ED25519 SHA256:...

Failed publickey with a fingerprint you did not expect means the client offered a different key than you thought — back to -vvv. Failed publickey with your fingerprint means the key is right but not installed for that account, or the home-directory permissions on the server side are wrong (same 700/600 rule applies to the server's ~/.ssh and ~/.ssh/authorized_keys). If sshd refuses to start while you are editing configs, that is a different hunt — see systemd service not starting.

verify: journalctl -u ssh -n 50 | grep Accepted → a line with your fingerprint and user

How do I install my key correctly?

ssh-copy-id deploy@staging
# Number of key(s) added: 1
ssh -o BatchMode=yes deploy@staging 'echo ok'   # non-interactive proof

ssh-copy-id appends your public key to the target account's authorized_keys and sets sane permissions — it exists because hand-pasting into authorized_keys fails on a trailing newline or a missing directory exactly often enough. The BatchMode check is the real test: it disables all prompts, so success means the key alone carried the login. Once the key works, scp and rsync inherit the same auth for free — the choice between them is bandwidth, not credentials (rsync vs scp).

verify: ssh -o BatchMode=yes deploy@staging 'echo ok' → ok, no prompt

Permission denied (publickey) is a negotiation result, not a password prompt — the server refused every key in the offer, and its log already knows why.

Run the four commands in order and the failure stops being mysterious: one of them names the culprit every time. Mine was a stale agent identity answering before the right key — ssh-add -d, and the deploy went green at 02:31. For pushing code over that same key, fatal: not a git repository is the next wall, and the git recovery hub holds the rest.