So I was asked the other day how do you patch a server that is behind on patching?
What's the best action plan?
1. Take a backup. Ensure you have a DR plan first and foremost. While 99.999999999% of the time throughout the entire world we patch and reboot just fine, it's that .00000000001% of the time that I guarantee you will be when you don't have time to deal with it. So having a DR strategy first and foremost ensures that you can roll back.
2. Finding about known issues. Get asked about this one a lot as well. See the goal of Microsoft, and any vendor that releases patches is that they shouldn't disrupt your business. The vast majority of patches and updates occur without issue. When there are some issues Microsoft typically documents them in the upper section of a Security bulletin. That "Known issues" section that is up there documents the known issues. By now the majority of the known issues are known and documented. Many of the known issues section are more about install issues that interactions with a vendor software. And most interactions with vendor software by now are remedied by ensuring that the updates from the vendors are installed. And remember (well with the exception of the recent botched Win7 sp1) you can uninstall patches. So even if there is some funky interaction that isn't known because you have the wacky client with the unusual line of business app, most updates are uninstallable so you can roll back. And these days in light of the Win7 sp1 blow up you might think about imaging/backing up some of the key workstations.
3. But I don't have time to read each bulletin or paying attention to what patches get offered up on my system, you say. Okay. So then if you don't have time, then your DR plan should be even more robust. Because sometimes there's reasons to not deploy patches and service packs right away.
Here's categories of patches that I wait a little bit on before deploying.
a. SQL. Read the bulletin or review in the window if the update for SQL is a service pack. Unless the method of attack is from some remote attacker, there's typically no rush needed to patch sql. And you REALLY need to make sure you take a backup of the system first before applying either a security patch or a SQL update. The older your SQL instance (as in 2000, 2005) the more you may have issues patching it. While I've not updated 2008 or 2008R2's too much, they've made increases and fixes in the updating so that it's not as horrific as it was in the SQL slammer era when you had to practically read a white paper to update SQL.
b. .NET. Me and .net hate each other. A lot. Again review the description of the update. Again if the manner of attack is not "can worm there way in" and is instead "local attacker must log in and..." then the risk is much lower in your network. Supposedly .net 4 was supposed to be better. But it's had patches already for .net 4 and caused such install issues that I'd recommend that unless the server has .net 4 (like SBS 2011) don't install it unless your line of business app puts it on there. Also try to keep older .net's off of Win7 and Server 2008 r2 (more on this later ). Try to keep the .net version in the age range of the server. 1.1 and 2 is just fine for SBS 2003. 2 and 3 is just fine for SBS 2008. 2, 3 and 4 is fine for SBS 2011.
c. SharePoint. There are times that SharePoint and .NET are fighting to see who will be on my hated updates list. Make sure you have a backup of the SharePoint first before installing, don't freak if it doesn't properly run the psconfig command.
d. Service packs of any kind, but especially Exchange or any other one that says it's uninstallable. Because Exchange is a database, it's service packs are not uninstallable. You want to roll back to SP2? You reinstall Exchange ...which on a SBS box is not what I'd recommend. Always always always have a backup of your system before doing any service packs. It's only been recently that SQL now allows for uninstallable service packs. As we've seen lately install them ALL BY THEMSELVES.
e. XML updates. For a while there we used to get this patched a lot and it sometimes either got stuck and reoffered over and over again, or just wouldn't install. Havent had a problem in a while, nor gotten a patch in a while.
So now that we have our problem categories identified... let's identify how we'd tackle this beast.
1. Look at the server and see if it needs a major service pack and thus will leapfrog you past the umpteen plus security updates. ALWAYS install these separately and consider downloading the full file and install that rather than going to MU or using WSUS. Once you have the major service packs out of the way, then chunk the patches down into categories -- choose the windows updates and install them. Chose the Exchange and install them. For anything that is database-y, I'll typically do those one at a time just because I feel more comfy doing so.
2. Don't patch over RWW. I guarantee that when you update that server web services will be turned off. So don't plan on using RWW to update. In fact I'd recommend that you think about some other patching tool in addition to WSUS if that's what's on the server. The reason is WSUS is a pull technology - you approve the update and it will take time to check in with the server and deploy it. If you want to patch RIGHT NOW, WSUS is not your tool. Whether that's a RMM/scripting tool like Kaseya for the MSPs or something like it.shavlik.com, consider having something else so you can PUSH a patch and do it right now.
3. Reboot before you being patching. So many times something bad reported is not the result of a patch at all, but a condition that was going to occur if you'd merely had rebooted. It's amazing how many times drives suddenly decide to stop working when you reboot a system. So reboot BEFORE you begin patching so you can ensure that the system is robust and healthy before you update.
4. Ask yourself - does it make sense that you are seeing this issue. I've had people swear that KB####### stopped all Internet connectivity. Okay. Did you google that KB number? Do you see other dead bodies? Did you ask on the sbs2k@yahoogroups.com list if others have seen this? If you are the ONLY one reporting this, chances are that whatever you are seeing is not a result of a patch. Also know EXACTLY what you installed. When you reboot and should something occur, look at the update(s) you just installed. Does it make sense that a patch for outlook express on a server would stop web sites from running if the OE patch never touches the binaries that a web site uses to do it's job? Or perhaps is it some misconfiguration you did when you were messing around with the web site that came to light when you rebooted it. There's always a cause and effect. I shouldn't get too much off onto my rant box but it annnnooooys me no end when people claim that their update settings changed without their permission or after a patch. I will go to the mat on this statement: Windows update never spontaneously changes settings. It also never gets changed by an update. If your windows update settings change, it's because someone or something changed them. Installing Office 2007 was a HUGE silent flipper of machines to automatic updates as at the very end of the install it would ask "Would you like me to keep you up to date?" and it would change your WU over to Microsoft update and then turn automatic updates on if you unknowingly said "Yes" to it. Bottom line ask yourself if it makes sense that after applying X that Y freaks out.
5. When you say "my Server did this today and yet nothing changed" ... ask yourself if nothing really has changed. Really? Nothing at all? No antivirus updates? No spam def file updates? Nothing at all changed? So many times I'll see "nothing changed" turns into "that stupid antivirus program was blocking it".
6. Don't layer up on third party programs. This is more true for workstations than servers as we tend to keep servers less clogged up with third party software, but I have even seen patching issues caused by the very tools you were trying to use to remotely monitor the system you are trying to maintain. Just because you've used that tool for years, doesn't mean that they too don't push out updates.
7. Reach out to the community and check to see if others are seeing it. In the recent case of the Win7 sp1 failures, it's safe to say that everyone that took the advice to edit the pending.xml would do it again in a heartbeat. They HAD to get their machines operational. They HAD to get them back functional and they ALL feel that calling Microsoft is not an efficient use of their time. Most have the SP now declined or blocked for deployment. Remember there are venues to get help from Microsoft that isn't as highly critical but you'll still get a venue to ensure that Microsoft sees the trends. The SBS partner forum at http://social.microsoft.com/Forums/en-US/partnerwinserversbs/threads is where you can post things and make sure Microsoft sees it. Cost. Free. Time. Time of an email.
8. Plan for patching. Sign up for the Microsoft security alerts and know the amount and type of updates that are coming. Also you can usually plan that one month will be a heavy month, one month will be a light month. So plan/staff/read accordingly.
Hopefully this bit of a Friday night ramble helps to identify those areas I look at when I tackle patching.
![[There's a reason that Yoda is the unofficial mascot of SBS. Size indeed matters not.]](http://www.sbslinks.com/images/headertop.png "There's a reason that Yoda is the unofficial mascot of SBS. Size indeed matters not!")