You run systemctl status nginx and get a wall of green text. Everything looks fine. Then a colleague asks why the service restarted at 3 a.m., and you realize you've been reading that output for years without actually understanding half of what it says. Sound familiar?
Most Linux administrators learn systemd the same way they learn a lot of things: by copy-pasting unit files from Stack Overflow and hoping. It works, until it doesn't. And when a systemd service fails silently at boot, or a restart loop eats your CPU, or a hardening directive breaks an app in production, you need to understand the machinery underneath.
This is what I want to walk you through. Not the history of SysVinit. Not a man page transcript. The practical anatomy of systemd services, the parts that bite, and the runbook I actually use when I create or debug a unit. I've made most of the mistakes below myself, sometimes more than once.
Key Takeaways
- A systemd service is a unit file (usually
/etc/systemd/system/*.service) that describes how to run a process, not the process itself. Type=determines how systemd decides your service is "started". Choosing the wrong one causes hangs or false failures.systemctl daemon-reloadis mandatory after editing a unit file. Skipping it is the most common source of confusion.journalctl -u service-nameis your primary debugging tool, notsystemctl status.- Hardening directives like
ProtectSystem=andNoNewPrivileges=are cheap wins, but they will break poorly-written applications.
Systemd services explained: what you're really managing
A service unit is a text file. That's it. It contains a [Unit] section, a [Service] section, and an [Install] section. systemd reads it, parses the directives, and decides how to spawn and supervise your process.
The confusion starts because people conflate three things: the unit file, the running process, and the unit's state in systemd's internal registry. When you edit a file and nothing changes, it's because systemd is still holding the old version in memory. Editing ≠ reloading. That single misunderstanding probably accounts for a third of "systemd is broken" complaints I've heard, including from myself.
Where unit files actually live
Three directories matter, and the priority order trips people up:
/usr/lib/systemd/system/— distribution packages drop their units here. Don't edit these; package updates will overwrite you./etc/systemd/system/— your overrides and custom units. Takes precedence over the above./run/systemd/system/— runtime units, disappear on reboot.
Need to change one line in a packaged unit? Use systemctl edit nginx, which creates a drop-in at /etc/systemd/system/nginx.service.d/override.conf. It survives package upgrades and keeps your diff readable. I wish someone had told me this in my first year instead of me copying entire unit files into /etc and losing track of which was authoritative.
The five directives that do the real work
You can write a functional unit in six lines:
[Unit]
Description=My app
After=network.target
[Service]
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.toml
Restart=on-failure
[Install]
WantedBy=multi-user.target
That's a working systemd unit file example. After= orders startup but does not guarantee the dependency exists—that's Requires= or Wants=. WantedBy=multi-user.target is what makes systemctl enable actually link the service into a boot target. Omit it and your service will never start at boot, no matter how many times you enable it.
Restart= deserves a warning. on-failure is sane. always combined with a broken ExecStart creates an infinite crash loop that will quietly hammer your CPU and fill the journal. I did this once on a monitoring agent and only noticed when the box's load average hit 40.
Systemd service types: choosing the right mode
This is where analysts and admins disagree, and where I'll plant a flag: most people use Type=simple when they should, and Type=forking when they absolutely shouldn't. Here's the breakdown.
| Type | How systemd detects "started" | Use when |
|---|---|---|
simple | Immediately after fork | Your process stays in foreground. Default and usually correct. |
forking | Parent process exits | Legacy daemons that daemonize themselves. |
notify | Process calls sd_notify() | App is systemd-aware and wants to signal readiness. |
oneshot | Process exits | Run-once scripts, migrations, setup tasks. |
exec | After execve() succeeds | Like simple, but catches permission errors before marking started. |
idle | After jobs finish | Rare. Console output during boot. |
The trap with forking: if the daemon doesn't actually fork, systemd waits for a parent exit that never comes, then times out. With simple and a forking daemon, systemd marks the service started, the daemon keeps running, and you get a lifecycle that looks fine until you try to stop it cleanly.
When to use a oneshot
Use Type=oneshot for anything that runs once and exits: a database migration, a bootstrap script, a certificate renewal. Pair it with RemainAfterExit=yes if you want systemctl status to still report the service as active after it finishes. Skip that directive and the service will show as inactive (dead) even though it succeeded—which confuses everyone who inspects it later.
Creating a service: the runbook that works
Here's the sequence I use when I add any new service. Each step has a verification, because skipping verification is how you end up chasing ghosts.
- Write the unit at
/etc/systemd/system/myapp.service. systemctl daemon-reload— required every time you touch the file.systemctl enable --now myapp— enables at boot and starts immediately.systemctl status myapp --no-pager— check that it'sactive (running), notactivating.journalctl -u myapp -n 50 --no-pager— read the last 50 lines of logs before you declare victory.
The --now flag on enable is one of those small things I resisted for too long. It saves a step and prevents the classic "I enabled it but forgot to start it" mistake.
Debugging a failing service
When a service lands in failed state, don't start guessing. Open systemctl status and read the Result= field in the main output block. It tells you directly:
Result=exit-code— the process exited non-zero. Check theExitCode=field and the journal.Result=signal— killed by a signal.Status=shows which one. Signal 9 means something OOM-killed it.Result=timeout— startup exceededTimeoutStartSec=. Usually aType=mismatch.Result=resources— hit a resource limit (LimitNOFILE=,MemoryMax=, etc.).
Nine times out of ten, the journal gives you the actual error in plain English. The remaining tenth is usually a permissions issue—the service runs as a different user than you tested with. Add User=appuser and Group=appuser to the unit and re-test; the behavior will change.
Syntax gotchas nobody mentions
A few things that have burned me:
- Spaces around
=in directives will cause a parse error.ExecStart = /bin/foofails;ExecStart=/bin/fooworks. - Directives are case-sensitive.
restart=on-failureis silently ignored. No warning. - Environment variables in
ExecStart=need$VAR, not${VAR}unless you're using explicit set syntax. And they must be declared withEnvironment=orEnvironmentFile=. - Comments require
#at the start of a line—no inline comments.
Actually, wait—I need to correct myself on that last one. As of systemd 240-ish, inline comments after a value are still forbidden for some directives but tolerated for others depending on the parser version. Don't rely on it. Put comments on their own lines and stay safe across distros.
Hardening directives worth using
Adding a handful of security directives to a unit costs nothing and closes real attack surface. You don't need to go full systemd-analyze security on day one. Start here:
NoNewPrivileges=yes— blocks setuid escalation. Almost free.PrivateTmp=yes— gives the service its own/tmp. Breaks very few things.ProtectSystem=strict— makes/usr,/boot, and/etcread-only from the service's view. There'sfullandyestoo, butstrictis stronger and still workable for most apps.ProtectHome=yes— hides/homeand/root.User=andGroup=— run as a dedicated unprivileged user, not root.
One caveat I learned the hard way: ProtectSystem=strict will break any app that writes to its own config directory under /etc. If you hit that, either move the config to /var/lib/myapp (with StateDirectory=) or add ReadWritePaths=/etc/myapp to punch a hole. The latter is a compromise, but a documented one.
What to remember when the pager closes
systemd services are not complicated once you stop treating them as magic. They're files describing processes, with a handful of directives that determine lifecycle, ordering, and privilege. The complexity lives entirely in the failure modes: the wrong Type=, the missing daemon-reload, the restart loop nobody noticed.
The next time a service misbehaves, skip systemctl restart. Read the Result= field. Watch the journal. Look at the unit file. Nine times out of ten, the answer is right there in a directive you copied from someone else's blog post in 2019—and never quite fit.
And when you do find that directive, change it in a drop-in, not the original file. Future you, doing an upgrade six months from now, will thank present you.