systemd Services: Restarting Automatically When They Crash

Your site runs on the server — and then what? Close the SSH window and the service dies with it. The beginner's classic trio: & to background it, nohup to hang it up, tmux to keep a window alive. All three share three flaws: first, a reboot wipes them all — one midnight power cut and you're up manually recovering; second, nobody notices when the process dies, let alone restarts it; third, logs are unmanaged — output redirected to a file grows forever. The right way is systemd: Linux's service butler. You tell it how to start your site; it handles boot auto-start, restart-on-crash, and log archiving. This post writes one ready-to-use .service file, with every field explained in plain language.

First, a ready-to-use example

Say your site lives in /var/www/myapp, starts with node server.js, and runs as the deploy user. Create /etc/systemd/system/myapp.service with this content:

[Unit]
Description=my-site
After=network.target

[Service]
Type=simple
User=deploy
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=5
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

Every field, in plain language

• ·Description: what this service is — a name for yourself; write it clearly.
• ·After=network.target: wait for the network before starting — don't run at boot before the network is up.
• ·Type=simple: the start command IS the service (node server.js keeps running) — the most common choice.
• ·User=deploy: which user runs it. Not root — use the normal user created earlier.
• ·WorkingDirectory: cd into this directory before starting; relative-path configs depend on it.
• ·ExecStart: the real start command — must be an absolute path (/usr/bin/node; find it with which node).
• ·Restart=always: the core. Restarts however the process exits — pulls itself back up on crash. Note it restarts on ANY exit, including your manual systemctl stop — stop tells systemd to leave it alone, it won't fight you; only unexpected exits trigger it.
• ·RestartSec=5: wait 5 seconds before restarting — don't hammer the box with instant crash loops. If the service crashes on start (e.g. a config typo), systemd retries a few times at this interval — enough time for you to read the logs.
• ·Environment: environment variables, one per line — don't forget NODE_ENV=production in production. Too many variables? Don't pile them here; use EnvironmentFile=/var/www/myapp/.env pointing at a file — much cleaner. Keep that file unreadable by others; it may hold secrets.
• ·WantedBy=multi-user.target: enables boot auto-start — enable errors without this line. multi-user.target is the command-line multi-user mode; servers usually boot into it, and your service starts when boot reaches that stage.

Other Types in one sentence each

Type=simple is the most common, but you'll meet two others: Type=forking for old-style programs that start then retreat to the background (like nginx) — systemd finds them via a PID file; Type=oneshot for run-once tasks, paired with RemainAfterExit=yes. simple covers 90% of cases; don't panic when you see forking in someone else's service file — just know it exists.

Going live in four steps: drop it in, make it take effect

sudo cp myapp.service /etc/systemd/system/  # move it into the system directory
sudo systemctl daemon-reload              # tell systemd there is a new service
sudo systemctl enable --now myapp         # enable at boot + start right now
sudo systemctl status myapp               # check the status

status showing “Active: active (running)” means it's alive. Note step two, daemon-reload — run it every time you edit the .service file, or you're editing the old version. That's the most forgotten step for beginners: config changed but nothing took effect, and nine times out of ten this is why. Ask yourself first: did I reload? enable and start are different things: start runs it now, enable runs it at next boot too, --now does both. Confirm auto-start really took with systemctl is-enabled myapp — only “enabled” counts.

The systemctl commands worth memorizing

• ·systemctl status myapp: check status — active or failed; look at this first.
• ·systemctl start/stop/restart myapp: start, stop, restart — restart is the most used after config changes.
• ·systemctl enable/disable myapp: turn boot auto-start on/off — doesn't affect whether it's running now.
• ·systemctl is-enabled myapp: check whether boot auto-start really took — only “enabled” counts.
• ·systemctl daemon-reload: must run after editing the .service file, or you changed nothing.
• ·systemctl cat myapp: see the complete effective config — for tracing which file actually applies.

Memorize these six and daily ops are covered. A word on systemctl cat: it shows the merged config systemd actually read — when you suspect a config didn't take effect, cat it before guessing; much faster. Tab completion works on service names too — systemctl status my + tab completes; no fear of half-remembered names, far less typing.

Where service files go: don't mix up the two directories

The system has two places for .service files: /lib/systemd/system holds package-shipped ones (like nginx.service); /etc/systemd/system holds your own. Yours go under /etc — system upgrades won't overwrite them, and they take precedence. When both sides have the same filename, /etc wins. Remember this rule so you know which file actually applies when troubleshooting — the “Loaded” line in systemctl status states exactly which path it uses.

Reading logs: journalctl

systemd collects the service's output (those console.logs) for you — no manual redirecting to files. Two lines cover log reading:

journalctl -u myapp -n 50 --no-pager   # show the last 50 lines
journalctl -u myapp -f                  # follow live, Ctrl+C to exit

When a service won't start, read the logs before restarting. Nine times out of ten it's a wrong path, a taken port, or unconfigured environment variables — the logs state them plainly. journalctl has handy combos: --since '1 hour ago' for just the last hour, -p err for error-level only — stack them. Logs collected by systemd live in /var/log/journal, capped at 10% of disk by default — no fear of it eating your disk. Note: if /var/log/journal doesn't exist, logs live only in memory and die on reboot. Run sudo mkdir -p /var/log/journal and restart the service so logs persist — worth checking on small boxes.

The three most common errors, matched to their fixes

• ·status shows failed, log says “permission denied”: eight times out of ten User=deploy but the files belong to root and deploy can't read them. Hand the directory back with chown -R deploy:deploy /var/www/myapp, or check the EnvironmentFile's permissions.
• ·203/EXEC error: ExecStart's path is wrong — systemd can't find that executable. Look up the full path with which node before filling it in; don't write /usr/local/bin/node from memory — version-manager-installed node paths are all over the place.
• ·EADDRINUSE port taken: another process already uses this port. Find it with ss -tlnp | grep PORT — either kill the old one or change the new service's port. The usual cause: an old nohup-started process still alive, two fighting over one port.

When NOT to use Restart=always

Restart=always suits services that run forever (sites, APIs). But one kind doesn't: run-once-and-exit scheduled jobs, like a backup script run once daily at dawn. Those get systemd timers or cron; forcing Restart=always turns into an infinite loop — finish, exit, systemd pulls it up, finish again, pulled up again — your backup runs thousands of times a day. And don't enable always during debugging either: use Restart=no first, switch to always only after the service runs stably — otherwise it restarts repeatedly while you edit config and the logs become unreadable.

Graceful shutdown: don't lose data on restart

Restart=always handles crashes, but normal restarts and shutdowns deserve dignity too. When systemd stops a service it sends SIGTERM by default, giving the process time to finish up (close connections, flush caches); only after TimeoutStopSec (default 90 seconds) without exiting does it SIGKILL. Your program should handle SIGTERM — flush what must be flushed; in Node listen on process.on('SIGTERM') for cleanup, in Python use the signal module. If the program needs a special stop command (like notifying the main process first), add ExecStop=/usr/bin/kill -TERM $MAINPID — though most Node/Python programs are fine with the default. Remember: kill -9 is the nuclear option, only for zombie processes; daily restarts go through systemctl restart and end with dignity.

Resource limits: don't let one service kill the whole box

On a 1-CPU 2GB small box, one memory-leaking service can drag the whole machine into OOM, SSH unreachable. systemd can cap services:

[Service]
MemoryMax=512M
CPUQuota=80%

MemoryMax=512M kills the service when it exceeds 512M (then Restart=always pulls it back up); CPUQuota=80% caps it at 0.8 cores. A kill leaves a record in the logs — you can see it's the culprit instead of the whole box mysteriously freezing with SSH unreachable. Cap every service on small boxes — two minutes to avoid being woken at night; worth it.

Verify auto-restart really works

Don't trust the docs — verify by hand once: find the process with ps, kill it, wait a few seconds, then systemctl status — Main PID changed and the start time updated. That's a true self-restart. Only then can you sleep at night. One detail: if the service crashes repeatedly, status shows failed; after fixing the config, run systemctl reset-failed myapp to clear the failure count, then start — otherwise it still remembers the failed state.

ps aux | grep "node server.js"   # find the PID, e.g. 812
kill 812                          # simulate a crash
sleep 8
sudo systemctl status myapp       # PID PID changed and timestamp updated = auto-restart works

If the service doesn't come back after the kill, don't panic: check whether status is failed; if so, journalctl -u myapp -n 30 for the cause. Usually RestartSec hasn't elapsed yet (wait a few more seconds), or systemd throttled it after too many consecutive crashes — that's a protection mechanism; fix the root cause, then reset-failed. Don't fight it with hard restarts.

That's it — your site now has a butler: starts itself on boot, restarts itself on crash, logs collected, resource caps as the backstop, dignified shutdowns, sleep soundly at night. If you skipped server initialization, go back to “10 Things to Do First on a New VPS”; for reverse proxy and HTTPS see “Caddy Reverse Proxy: HTTPS for Your Site in 5 Minutes.” For systemd commands you can't remember (systemctl, journalctl), this site's shell command cheat sheet (the /shell page) has them all in the log-troubleshooting group — copy and paste.