There are dozens of theories and black box solutions out there which claim to manage your OS patching. I don’t have all the answers but I do have experience which may be of some use in the real world. I’ll discuss what I’ve found to work and what usually does not work at an enterprise scale. This article maybe helpful, in providing requirements, to those of you looking to purchase and design a solution so that you are better prepared to ask the right questions.
- Enterprise patching is all about communicating with your stakeholders. Communicate early and often. If possible, communicate in person and get visual confirmation that they understand the scope and what is required of them. Some things you’ll want to communicate:
- Deployment and downtime schedule.
- Actual downtime.
- If you need them to perform and application related health check, call this out.
- Don’t assume your system is healthy before the patching. Always perform a health check prior to patching or you’ll find yourself chasing a red herring if something doesn’t work after the patching is complete.
- Automate your health check utilizing enterprise tools or simplify repeatable checks using a script. This will save you from having to repeat keystrokes over and over and will provide a more uniform health check. There is nothing more frustrating than finding out your system won’t come back up due to an old hardware failure. Using WMI, SSH or psexec there are some things you’ll want to check for:
- System logs for errors. (Application, System, Syslog, /var/adm/messages)
- Automatic services which should be running but are not.
- Hardware specific logs and online diagnostic utilities.
- An application health check (i.e. can my website connect to the database and does it behave as expected). Leverage scripting like Perl, Vbscript or Powershell to automate these tests.
- Pre-task everything. Do as much as you can before the deployment schedule.
- Health checks
- Patch package pre-staging. Have the patch packages copied ahead of time to your systems and don’t rely on fileshare, webservers or “Relay” servers during the deployment. These servers tend to get saturated and perform poorly during mass deployments. I cannot stress enough how CRITICAL this is! This accounts for the majority of the deployment wait time.
- Disk cleanup to free up space prior to patching.
- Checklists: Document all your deployments in the form of a checklist. There is nothing worse than having to read through a wordy document while you are working in production.
- Test, test and then when you are done, test again. If possible run mockups in a test environment. If that is not an option, be very careful to test any standby, passive or secondary nodes before proceeding. Fail the application over and validate functionality. If you can’t afford a test environment, look into P2V technologies to virtually replicate your systems and test on a virtual instance.
- Never assume that your deployment schedule will proceed on time. Always allow for unexpected health check results, hardware issues and unknowns.
- If your systems are virtualized, look into creating a clone connected to an isolated VLAN, patch the clone, then on the day of the deployment, simply incrementally sync any data and switch over to the patched system.
- Finally always have a backup of your system state and data. Leverage bare-metal-restore (BMR) solutions to help get you back to your previous state if something fails. If possible prepare a virtual failover system, ahead of time for your critical applications, to temporarily restore service while you are working on until you are able to restore the physical.