systemctl –user cannot start docker containers on Ubuntu 20.04

ovidiu asked:

I am in the process of migrating some services from Ubuntu 18.04 to 20.04. In 18.04 I run these services under a non-root user. All these services start a docker container, and they’re working just fine. Under Ubuntu 20.04 these services no longer start.

To illustrate, here’s a very simple ~/.config/systemd/user/hello-world.service that works fine on Ubuntu 18.04:

# -*-systemd-*-
[Unit]
Description=Hello world
After=network.service
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
TimeoutStartSec=0

ExecStartPre=/bin/echo user = $USER
ExecStartPre=/usr/bin/docker pull hello-world
ExecStart=/usr/bin/docker run \
  --name hello-world \
  --rm -a STDIN -a STDOUT -a STDERR \
  hello-world

ExecStop=/usr/bin/docker stop -t 2 %n

[Install]
WantedBy=default.target

I run the container in the shell directly as the non-root user and it runs fine, both on the 18.04 machine, as well as on the 20.04 machine:

/usr/bin/docker pull hello-world
/usr/bin/docker run \
  --name hello-world \
  --rm -a STDIN -a STDOUT -a STDERR \
  hello-world

For systemd I run the following:

systemctl --user enable hello-world.service
systemctl --user start hello-world.service

On Ubuntu 18.04 everything runs as expected when I investigate the out with journalctl -xe -f.

On Ubuntu 20.04 I get the dreaded:

Sep 15 14:56:26 m4 docker[107614]: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/create?fromImage=hello-world&tag=latest: dial unix /var/run/docker.sock: connect: permission denied

I checked the permissions, groups and everything seems to be correct. Again, if I run the docker directly in the command line while logged in as username, docker runs just fine.

[email protected]:/etc/apt> ll /var/run/docker.sock 
srw-rw---- 1 root docker 0 Sep 15 14:08 /var/run/docker.sock=
[email protected]:/etc/apt> grep docker /etc/group
docker:x:998:docker,username

The only thing that’s different is that on 18.04 systemd is at version 237, while on 20.04 is at version 245.

Docker is the same on both machines:

Docker version 19.03.12, build 48a66213fe

Both versions of systemd show the user echoed in ExecStartPre as being my non-root user.

It looks like systemd 245 is starting the docker process under the wrong user and/or group. Any thoughts?

Update

As @larsks suggested, I replaced $USER with /usr/bin/id. Here’s the output I received:

Sep 15 21:36:09 m4 id[122143]: uid=1001(username) gid=1001(username) groups=1001(username)
Sep 15 21:36:09 m4 docker[122144]: Using default tag: latest
Sep 15 21:36:09 m4 docker[122144]: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/create?fromImage=hello-world&tag=latest: dial unix /var/run/docker.sock: connect: permission denied

username is part of the docker group, as shown above.

My answer:


Your systemd user unit doesn’t specify a Group=, thus the user’s default group is used. Since docker is not the default group, systemd doesn’t start the process with this group.

Set Group=docker in the [Service] section of the unit.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.