How do you manage updates without a staging environment: CentOS 6.3

user160910 asked:

I am managing about 20 servers, many of them virtual. They are almost all different purpose, and none are clustered. I have a distributed LAMP stack, a few application servers, some build servers, a few KVM hosts. They are CentOS 6.3 mostly with a few Ubuntu (unfortunately). I don’t have the resources to setup a staging environment where I can have duplicates of my machines and test updates before rolling them out. I am taking file backups. What I want to know is how you are approaching backing up your Linux systems. I assume you don’t just do yum update, but then how are you choosing the packages worthy of updating? When (if ever) are you updating the kernel, etc.. How do you test updates without a staging environment? Snapshot and hope for the best?

My answer:

This is pretty common with servers that are pets, not livestock.

If you really can’t test updates, then you:

  1. Have backups in place. Remember that you don’t really have backups unless restores work.
  2. Read the description of the updates to see what they change.
  3. Do updates during off-hours. Schedule a maintenance window even if you end up not needing it.
  4. Apply the updates. Reboot if the kernel was updated. Test the affected services.
  5. Wait for the users to start yelling.
  6. If necessary, roll the updates back (using yum history undo).

My guess is that you didn’t know you could revert updates with a single command. Check the yum man page and read its history section to see what else you can do with it. For instance, you don’t have to revert updates in the order you applied them.

And stop worrying so much. Most updates fix problems that you need to have fixed; introducing new problems is far less common (though it can and does happen).

View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.