Not able to start apache in Wildfly 10 domain mode cluster with Apache mod_cluster, due to SELinux issue

abhishek asked:

I am trying to setup a Wildfly 10 cluster in domain mode with Apache mod_cluster.

In my Centos 7 webserver node I have installed Apache(2.4.6) using:

# yum install httpd

Then copied the following .so files to the /etc/httpd/modules directory

mod_cluster_slotmem.so
mod_manager.so
mod_proxy_cluster.so
mod_advertise.so

and appended the following in the httpd.conf file

# LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
.
.
.

LoadModule cluster_slotmem_module modules/mod_cluster_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so

<IfModule manager_module>
    Listen 192.168.56.105:10001
    ManagerBalancerName other-server-group
<VirtualHost 192.168.56.105:10001>
    <Location />
       Require all granted
    </Location>
    <Location /mod_cluster-manager>
       SetHandler mod_cluster-manager
       Require all granted
    </Location>
</VirtualHost>
</IfModule>

Now when I am trying to start the httpd, it is throwing some error:

# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Mar 25 17:39:03 webserver01.internal setroubleshoot[2772]: SELinux is preventing /usr/sbin/httpd from write access on the file /var/log/httpd/manager.node.nodes. For co
Mar 25 17:39:03 webserver01.internal python[2772]: SELinux is preventing /usr/sbin/httpd from write access on the file /var/log/httpd/manager.node.nodes.

                                               *****  Plugin catchall (100. confidence) suggests   **************************

                                               If you believe that httpd should be allowed write access on the manager.node.nodes file by default.
                                               Then you should report this as a bug.
                                               You can generate a local policy module to allow this access.
                                               Do
                                               allow this access for now by executing:
                                               # ausearch -c 'httpd' --raw | audit2allow -M my-httpd
                                               # semodule -i my-httpd.pp

Detailed log:

type=AVC msg=audit(1521962682.292:313): avc:  denied  { write } for  pid=3891 comm="httpd" path="/var/log/httpd/manager.node.nodes.lock" dev="dm-0" ino=656345 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521962682.292:313): arch=c000003e syscall=2 success=no exit=-13 a0=5583cf525ce0 a1=80041 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=3891 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521962682.292:313): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964160.534:399): avc:  denied  { write } for  pid=4580 comm="httpd" path="/var/log/httpd/manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521964160.534:399): arch=c000003e syscall=2 success=no exit=-13 a0=560012130cb8 a1=800c1 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=4580 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964160.534:399): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964202.459:432): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964202.459:432): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=7ffc42786620 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964202.459:432): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964203.462:433): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964203.462:433): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964203.462:433): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964204.463:434): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964204.463:434): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964204.463:434): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964206.467:436): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964206.467:436): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964206.467:436): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964205.465:435): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964205.465:435): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964205.465:435): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521974913.642:174): avc:  denied  { remove_name } for  pid=2738 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521974913.642:174): arch=c000003e syscall=87 success=yes exit=0 a0=55fac7b30cb8 a1=55fac7bd3598 a2=180 a3=7ffedd5736e0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521974913.642:174): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521974913.642:175): avc:  denied  { write } for  pid=2738 comm="httpd" path="/var/log/httpd/manager.node.nodes" dev="dm-0" ino=656322 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521974913.642:175): arch=c000003e syscall=2 success=yes exit=18 a0=55fac7b30cb8 a1=800c1 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521974913.642:175): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521974913.643:176): avc:  denied  { name_bind } for  pid=2738 comm="httpd" src=23364 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=udp_socket
type=SYSCALL msg=audit(1521974913.643:176): arch=c000003e syscall=49 success=yes exit=0 a0=16 a1=55fac7b31140 a2=10 a3=0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521974913.643:176): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521975145.939:226): avc:  denied  { write } for  pid=2738 comm="httpd" path="/var/log/httpd/manager.node.nodes.slotmem" dev="dm-0" ino=656174 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521975145.939:226): arch=c000003e syscall=2 success=yes exit=3 a0=55fac7bd44d8 a1=80042 a2=1b6 a3=0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521975145.939:226): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521975145.940:227): avc:  denied  { remove_name } for  pid=2738 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656322 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=AVC msg=audit(1521975145.940:227): avc:  denied  { unlink } for  pid=2738 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656322 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521975145.940:227): arch=c000003e syscall=87 success=yes exit=0 a0=55fac7bd3960 a1=55fac7bd3598 a2=0 a3=7ffedd5738a0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521975145.940:227): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521979740.681:199): avc:  denied  { write } for  pid=2761 comm="httpd" path="/var/log/httpd/manager.node.nodes" dev="dm-0" ino=655447 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521979740.681:199): arch=c000003e syscall=2 success=no exit=-13 a0=5598961f3cb8 a1=800c1 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=2761 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521979740.681:199): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44

When I ran SELinux in permissive mode then httpd started properly and I was able to access the cluster.

As running SELinux in permissive mode is not recommended please help me to find out the root cause of this behavior and how to fix it?

——-UPDATE————

As suggested by Tom H, the output of inspection using audit2allow:

# audit2allow -i /var/log/audit/audit.log -m my-httpd

module my-httpd 1.0;

require {
    type gssproxy_t;
    type httpd_log_t;
    type httpd_t;
    type fs_t;
    type unreserved_port_t;
    class udp_socket name_bind;
    class file { unlink write };
    class dir remove_name;
    class filesystem getattr;
}

#============= gssproxy_t ==============

#!!!! This avc is allowed in the current policy
allow gssproxy_t fs_t:filesystem getattr;

#============= httpd_t ==============
allow httpd_t httpd_log_t:dir remove_name;

#!!!! The file '/var/log/httpd/manager.node.nodes.lock' is mislabeled on your system.
#!!!! Fix with $ restorecon -R -v /var/log/httpd/manager.node.nodes.lock
allow httpd_t httpd_log_t:file { unlink write };

#!!!! This avc can be allowed using the boolean 'nis_enabled'
allow httpd_t unreserved_port_t:udp_socket name_bind;

Other o/p:

# sesearch -s httpd_t -t httpd_log_t --allow
Found 6 semantic av rules:
   allow daemon logfile : file { ioctl getattr lock append } ;
   allow httpd_t httpd_log_t : lnk_file { read getattr } ;
   allow httpd_t httpd_log_t : file { ioctl read write create getattr setattr lock append unlink open } ;
   allow httpd_t file_type : filesystem getattr ;
   allow httpd_t file_type : dir { getattr search open } ;
   allow httpd_t httpd_log_t : dir { ioctl write create getattr setattr lock add_name remove_name search open } ;

# rpm -qa | egrep 'httpd|selinux'
libselinux-2.5-11.el7.x86_64
httpd-2.4.6-67.el7.centos.6.x86_64
pcp-selinux-3.11.8-7.el7.x86_64
selinux-policy-3.13.1-166.el7_4.9.noarch
httpd-manual-2.4.6-67.el7.centos.6.noarch
libselinux-python-2.5-11.el7.x86_64
libselinux-utils-2.5-11.el7.x86_64
selinux-policy-targeted-3.13.1-166.el7_4.9.noarch
httpd-tools-2.4.6-67.el7.centos.6.x86_64
libselinux-2.5-11.el7.i686

My answer:


Your audit2allow output contains an interesting comment, which you should have read:

#!!!! The file '/var/log/httpd/manager.node.nodes.lock' is mislabeled on your system.
#!!!! Fix with $ restorecon -R -v /var/log/httpd/manager.node.nodes.lock

I would also guess that you have more mislabeled files on your system. I’d fix the labels recursively for the whole directory myself.

restorecon -R -v /var/log/httpd

You should also ensure that the system is up to date and in particular has received SELinux policy updates.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.