Systemd service not starting – 'Failed at step EXEC spawning… Permission denied'

Baron Mingus asked:

I’ve read other answers for similar issues (Failed at step EXEC spawning… Permission denied) and none seem to apply.

After I upgraded my bitcoin node to v0.20.0 it is no longer starting on boot nor manually with systemctl start bitcoind.service.

Journalctl output:-

$ journalctl -xe
...
Jul 12 15:58:22 $HOSTNAME systemd[572]: bitcoind.service: Failed to execute command: Permission denied
Jul 12 15:58:22 $HOSTNAME systemd[572]: bitcoind.service: Failed at step EXEC spawning /usr/bin/bitcoind: Permission denied
-- Subject: Process /usr/bin/bitcoind could not be executed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The process /usr/bin/bitcoind could not be executed and failed.

I’m using the default init script (https://github.com/bitcoin/bitcoin/blob/master/contrib/init/bitcoind.service) softlinked from my home directory to /etc/systemd/system/.

The executable is softlinked from my home directory into /usr/local/bin, so previous answer I’ve linked above suggesting an SELinux issue could apply but I don’t believe it is enabled: (-bash: getenforce: command not found). I am running Debian 10.4 and believe it is disabled by default.

$ ls -al /usr/bin/bitcoind 
lrwxrwxrwx 1 root root 34 May  5  2019 /usr/bin/bitcoind -> /home/$MY_NAME/src/bitcoin/src/bitcoind
$ ls -al /home/$MY_NAME/src/bitcoin/src/bitcoind
-rwxr-xr-x 1 bitcoin bitcoin 175058584 Jul  7 20:40 /home/$MY_NAME/src/bitcoin/src/bitcoind

Everything is set to run under the ‘bitcoin’ nologin user:

$ id bitcoin
uid=999(bitcoin) gid=999(bitcoin) groups=999(bitcoin)
$ cat /etc/passwd
...
bitcoin:x:999:999::/home/bitcoin:/bin/false
...

Here are the permissions on the relevant directories:

$ ls -al /etc/bitcoin
total 12
drwx--x---  2 bitcoin bitcoin 4096 Jul 11 22:30 .
drwxr-xr-x 87 root    root    4096 Jul 12 15:58 ..
-rw-rw----  1 bitcoin bitcoin  601 Jul 11 22:30 bitcoin.conf
$ ls -al /var/lib/bitcoind
total 21896
drwx--x---  5 bitcoin bitcoin     4096 Jul  9 11:55 .
drwxr-xr-x 32 root    root        4096 Dec  5  2019 ..
...
$ ls -al /run
total 24
drwxr-xr-x 18 root        root         560 Jul 12 15:59 .
drwxr-xr-x 22 root        root        4096 Jun  2 17:46 ..
...

I’m stumped – any input greatly appreciated.

Edit – permissions on path to the binary:

$ ls -al /home/$MY_NAME
total 160
drwxr-xr-x 11 XXX  XXX   4096 Jul 12 15:58 .
drwxr-xr-x  3 root root  4096 May  4  2019 ..
...
drwxr-xr-x  6 XXX  XXX   4096 Oct 25  2019 src
...
$ ls -al /home/$MY_NAME/src
total 24
drwxr-xr-x  6 XXX XXX 4096 Oct 25  2019 .
drwxr-xr-x 11 XXX XXX 4096 Jul 12 15:58 ..
drwxr-xr-x 15 XXX XXX 4096 Jul  7 20:26 bitcoin
...
$ ls -al /home/$MY_NAME/src/bitcoin
total 1976
drwxr-xr-x 15 XXX XXX    4096 Jul  7 20:26 .
drwxr-xr-x  6 XXX XXX    4096 Oct 25  2019 ..
...
drwxr-xr-x 28 XXX XXX   12288 Jul  7 21:01 src
...
$ ls -al /home/$MY_NAME/src/bitcoin/src
total 936796
drwxr-xr-x 28 XXX     XXX         12288 Jul  7 21:01 .
drwxr-xr-x 15 XXX     XXX          4096 Jul  7 20:26 ..
...
-rwxr-xr-x  1 bitcoin bitcoin 175058584 Jul  7 20:40 bitcoind
...

My answer:


Install the binary properly in /usr/bin, as it should have been installed, and try again.

The systemd unit does a lot of confining of the service, and one thing that is denied is access to user home directories. It’s not expected that it will need to access a user home directory when running as a system service.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.