our nginx server is stopping/crashing on its own

Sayantan Das asked:

our nginx server crashes on its own and it did like this for couple of times randomly. I can’t figure out why its happening. This is what it shows when I check nginx status.

‚óŹ nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2018-11-01 00:48:16 IST; 9h ago
  Process: 16654 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCE
  Process: 16702 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
  Process: 16699 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 1353 (code=exited, status=0/SUCCESS)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

and nginx error.log

2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:80 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to [::]:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: bind() to 0.0.0.0:443 failed (98: Address already in use)
2018/11/01 00:48:13 [emerg] 16702#16702: still could not bind()
2018/11/01 00:48:16 [alert] 16665#16665: unlink() "/run/nginx.pid" failed (2: No such file or directory)

EDIT

sudo netstat -nlp
=====================================================================================================
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      1036/mongod     
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      30941/mysqld    
tcp        0      0 127.0.0.1:587           0.0.0.0:*               LISTEN      1382/sendmail: MTA:
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      1147/redis-server 1
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      29550/nginx -g daem
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1153/sshd       
tcp        0      0 127.0.0.1:3000          0.0.0.0:*               LISTEN      6988/node       
tcp        0      0 127.0.0.1:8088          0.0.0.0:*               LISTEN      1042/influxd    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1382/sendmail: MTA:
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      29550/nginx -g daem
tcp6       0      0 127.0.0.1:7983          :::*                    LISTEN      1691/java       
tcp6       0      0 :::80                   :::*                    LISTEN      29550/nginx -g daem
tcp6       0      0 :::8086                 :::*                    LISTEN      1042/influxd    
tcp6       0      0 :::22                   :::*                    LISTEN      1153/sshd       
tcp6       0      0 :::8983                 :::*                    LISTEN      1691/java       
tcp6       0      0 :::8888                 :::*                    LISTEN      1117/chronograf 
tcp6       0      0 :::443                  :::*                    LISTEN      29550/nginx -g daem
udp        0      0 0.0.0.0:68              0.0.0.0:*                           825/dhclient    
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     8775     1/init              /run/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     16995534 25536/systemd       /run/user/1000/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     17834    1326/systemd        /run/user/120/systemd/private
unix  2      [ ACC ]     SEQPACKET  LISTENING     8779     1/init              /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     8790     1/init              /run/systemd/journal/stdout
unix  2      [ ACC ]     STREAM     LISTENING     8793     1/init              /run/lvm/lvmpolld.socket
unix  2      [ ACC ]     STREAM     LISTENING     8794     1/init              /run/lvm/lvmetad.socket
unix  2      [ ACC ]     STREAM     LISTENING     13115    1/init              /var/lib/lxd/unix.socket
unix  2      [ ACC ]     STREAM     LISTENING     17902    1036/mongod         /tmp/mongodb-27017.sock
unix  2      [ ACC ]     STREAM     LISTENING     24330    1053/node           /home/ubuntu/.pm2/pub.sock
unix  2      [ ACC ]     STREAM     LISTENING     17115993 6389/git-credential /home/ubuntu/.git-credential-cache/socket
unix  2      [ ACC ]     STREAM     LISTENING     24408    1053/node           /home/ubuntu/.pm2/rpc.sock
unix  2      [ ACC ]     STREAM     LISTENING     13112    1/init              /run/snapd.socket
unix  2      [ ACC ]     STREAM     LISTENING     13113    1/init              /run/snapd-snap.socket
unix  2      [ ACC ]     STREAM     LISTENING     13114    1/init              /run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     13118    1/init              /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     23857    1114/python         /var/run/supervisor.sock.1114
unix  2      [ ACC ]     STREAM     LISTENING     17939    1382/sendmail: MTA: /var/run/sendmail/mta/smcontrol
unix  2      [ ACC ]     STREAM     LISTENING     13111    1/init              /run/uuidd/request
unix  2      [ ACC ]     STREAM     LISTENING     13271    1034/iscsid         @ISCSIADM_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     18633    1386/php-fpm.conf)  /run/php/php7.0-fpm.sock
unix  2      [ ACC ]     STREAM     LISTENING     9990652  30941/mysqld        /var/run/mysqld/mysqld.sock

And is there any way to restart nginx automatically if anything happens like this?

My answer:


Your netstat output shows that nginx is already running at the time you are trying to start it. Was it started manually outside of systemd? That’s usually what happens in this case. Try manually killing the nginx process, then restarting it within systemd.

killall nginx
# and wait until netstat no longer shows it, or use kill -9
systemctl start nginx

It could also be that systemd lost track of a running nginx process because whoever wrote that systemd unit doesn’t know what they’re doing. It actually tries to use the ancient and now obsolete start-stop-daemon to send signals to nginx, when systemd is perfectly capable of doing this itself! This is guaranteed to cause misery eventually. Try updating to a current version of nginx and/or Ubuntu, where that service might be fixed. Or just remove the erroneous ExecStop= line.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.