Making Puppet Enterprise 3.7 work with IPv6

If you’re running Puppet in a dual stack or IPv6-only environment and tried to install or upgrade to 3.7, you’ll have noticed that some things are a bit broken.

To begin with, the new web-based installer doesn’t listen on IPv6, making it impossible to use in an IPv6-only environment. If you get past that, you’ll notice that the PE console isn’t accessible via IPv6.

This is a work log of things I had to do to get a PE 3.7.2 puppetmaster working in an IPv6-only or dual stack environment.

Puppet Enterprise Installer

As I mentioned before, the PE web-based installer only listens on IPv4, making it impossible to use in an IPv6-only environment, and a real pain in a dual-stack environment. Fortunately, making this work required only a single one-line change to the puppet-enterprise-installer script:

--- puppet-enterprise-installer.orig    2015-02-04 23:09:56.000000000 +0000
+++ puppet-enterprise-installer 2015-03-31 03:48:30.024000000 +0000
@@ -1225,7 +1225,7 @@
                     # This will block
                     pushd /opt/puppet/share/installer &>/dev/null
 
-                    run_suppress_output "RACK_ENV=production /opt/puppet/bin/bundle exec thin start --debug -p ${INSTALLER_PORT} -a 0.0.0.0 --ssl --ssl-disable-verify"
+                    run_suppress_output "RACK_ENV=production /opt/puppet/bin/bundle exec thin start --debug -p ${INSTALLER_PORT} -a :: --ssl --ssl-disable-verify"
 
                     popd &>/dev/null
 

(Use patch -l to apply this patch if you cut and paste it.)

On my CentOS 7 test system, this makes the web-based installer available via both IPv4 and IPv6. Once patched, we can successfully generate a Puppet Enterprise answer file or install Puppet Enterprise via the web-based interface.

Puppet Enterprise Console Services

On my dual stack test system, the pe-console-services service failed to start, giving a “No route to host” exception while trying to connect to PostgreSQL. However, it started correctly on my IPv6-only test system.

On digging into it, I discovered that the cause was that my dual stack machine had a global IPv6 address, but an RFC 1918 IPv4 address which was 1-1 NATted to a global IPv4 address, and Java was trying to connect to the global IPv4 address. The global IPv4 address does not have Hairpin NAT, causing the traffic to be unroutable. Even if it did have Hairpin NAT, this would cause the traffic to exit and re-enter the system and thus hit the host’s firewall. And even if it weren’t firewalled, this would be a performance hit.

While this still needs a permanent solution, my workaround was to put the global IPv6 and private IPv4 addresses of the puppetmaster’s FQDN in /etc/hosts, causing the traffic to remain local. After this, pe-console-services started successfully. A permanent solution would involve causing Java to stop mistakenly preferring IPv4 over IPv6 (unfortunately a very common error).

At this point, though, I had to roll back the entire installation using puppet-enterprise-uninstaller -d -p and start from the beginning, as the installer had failed to classify itself and the master was hopelessly broken. The next run did complete successfully. If you’re installing fresh or upgrading, check to see if you need this workaround before you start.

Puppet Enterprise Console

The last thing that was broken is that Puppet’s Apache server was not listening on port 443 for IPv6 connections. The PE console was therefore available only on IPv4. This makes the console unusable in an IPv6-only network. It also causes difficulty in a dual stack network, as browsers try IPv6 first, find the connection refused, and don’t bother to attempt the IPv4 address. As the console worked on IPv6 in 3.3, this is quite the regression.

In Puppet 3.7, Apache’s configuration is closely controlled by some of Puppet Enterprise’s internal modules. Inspecting their templates, I found out that the module has a variable, ssl_listen_address, which was set to 0.0.0.0 and then used to configure Apache’s Listen and VirtualHost directives for the console. In my first attempt to deal with this, I tried setting this variable to [::] in the console, but this resulted in an unusable environment, as the variable is used elsewhere as well.

Finally I gave up and yanked the configuration out of the Puppet module templates entirely, giving the port 443 virtual host a hard-coded listener.

--- opt/puppet/share/puppet/modules/puppet_enterprise/templates/profile/console-services/puppetproxy.conf.erb.orig   2015-01-28 23:00:17.000000000 +0000
+++ opt/puppet/share/puppet/modules/puppet_enterprise/templates/profile/console-services/puppetproxy.conf.erb        2015-03-31 04:45:12.737000000 +0000
@@ -1,5 +1,5 @@
-Listen <%= @ssl_listen_address -%>:<%= @ssl_listen_port %>
-<VirtualHost <%= @ssl_listen_address -%>:<%= @ssl_listen_port %>>
+Listen <%= @ssl_listen_port %>
+<VirtualHost _default_:<%= @ssl_listen_port %>>
 
     # Enables SSL Encryption
     SSLEngine on

I applied this patch to the puppet template and then kicked off a puppet run with puppet agent --test. At that point the console was finally accessible on IPv6.

Conclusion

It’s clear that Puppet Labs is not sufficiently testing their enterprise product. These showstopper issues should have arisen during integration testing, at the latest. Unfortunately it’s not common for developers to set up IPv6-enabled dual stack or IPv6-only test environments, despite that people will be using their code with IPv6 and will expect it to work.

The Apache configuration needs more thought put into it with respect to dual-stack and IPv6-only environments. Simply having a single listen address that’s expected to apply to everything doesn’t work when there are a few components of the system which are still speaking only IPv4 (albeit only locally) and some components must listen on IPv6. Better yet it ought to be scrapped and replaced with a better performing and easier to manage web server, that is, nginx. This is possibly my personal bias, though; the more I’ve used nginx the more I’ve grown to dislike Apache.

2 Comments

  1. Thanks for the helpful feedback. I’ll advise the developers as IPv6 is a concern of mine too.

  2. […] the puppetmaster should be listening on a dual stack IPv6/IPv4 socket by default. Though PE has some other IPv6-related brokenness (my site) you’ll have to work […]

Leave a Comment

Your email address will not be published. Required fields are marked *