A systemd WTF

tldr; There is nothing obviously wrong with systemd. This makes it so difficult to learn. Check journalctl after every systemctl operation to avoid unexpected failures. Systemd is ignoring what it calls warnings and behaves as if nothing happened, which results in unrelated errors.

This one is my personal WTF, which might just be my basic misunderstanding of the topic.

Running a web application on a far too old Debian, I am trying to upgrade the distribution to the latest stable version — I have postponed this for far too long fearing the systemd.

So I wrote a unit file: /etc/systemd/system/web.service.

Unit]
Description=web.service
 
Service]
WorkingDirectory=/opt/web
ExecStart=/opt/bin/web -c /opt/web/config

Linked it into the /etc/systemd/system/multi-user.target.wants, and tried to start it:

# systemctl start web.service
# systemctl status web.service
[…]
’/opt/web/config’: No such file or directory

The web service needs a config directory in /opt/web/config, or it won’t start. So I decided to add it.

Unit]
Description=web.service
 
Service]
WorkingDirectory=/opt/web
ExecStartPre=mkdir -p /opt/web/config
ExecStart=/opt/bin/web -c /opt/web/config

Running systemctl restart web.service results in Warning: web.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.

OK then, systemctl daemon-reload it is.

Trying to see if it worked again:

# systemctl restart web.service
# systemctl status web.service
[…]
 Active: inactive (dead) since Thu 2016–12–29 16:10:15 UTC; 5s ago
[…]
’/opt/web/config’: No such file or directory

Right. Investigating.

1. Trying /opt/bin/web -c /opt/web/config by hand.
   Result: ’/opt/web/config’: No such file or directory
2. # ls -la /opt/web/config
ls: cannot access ‘/opt/web/config’: No such file or directory

WTF?

There are no warning with daemon-reload or restart, See anything wrong with the Unit file? Turns out, there was an error on daemon-reload. But systemd decided to ignore it and let me start web.service as if nothing happened.

# journalctl | grep web.service
Dec 29 16:07:25 systemd[1]: [/etc/systemd/system/web.service:6] Executable path is not absolute, ignoring: mkdir -p /opt/web/config

I can easily imagine other applications depending on ExecStartPre and failing in completely unexpected ways during run time.

This is not the first problem I have encountered, but it was trivial enough to understand and write about. It seems that the “UI” choices the systemd developers made, are not so consistent.

I wonder why are simple things so difficult here. Is there an error in my thinking? Do I use the tool in entirely different way as it was indended to be used?

I really wish to see it developing towards “There is obviously nothing wrong” with systemd as opposed to the current situation.

Edit: fixed formatting of the code parts.

Edit 2: Apparently the solution would be to run
systemctl edit web.service && systemd-analyze verify web.service
I wonder if systectl daemon-reload could just spit out the warnings and errors by default and would not implicitly defer it to journalctl