On Sydney and Security

I’m finally getting back in the swing of things following the week I spent in Sydney with my wife and friends. We headed down to Australia for the SMB Security Summit put on my Trend Micro and SBSFAQ.com, and a bit of sightseeing as well. It was a long trip, and I have a renewed respect for the efforts our Australian counterparts to come to the US as often as they do. I certainly couldn’t imagine making another trip like that for quite a while, despite my issues with flying in general.

But I gladly went to the conference, not only to help out my friend Wayne Small, who offered me an opportunity to speak and share my expertise in the forum, but also to learn. Every chance I have to participate in an event like this is more than just an opportunity to give back to the community, but it’s a great chance for me to listen to other experts and either get reminded of issues that have slipped to the back of my mind, or to acquire new information that I didn’t have before. Being able to mix and mingle with the likes of Dana Epp, Amy Babinchak, Susan Bradley, Wayne Small, Dean Calvert, and many, many others and pick their brains about issues I’m facing with my company or my clients was a fabulous opportunity.

There were several common themes that prevailed during the myriad of discussions both in and out of conference that week. Two of the key ones were the importance of least privilege and improving authentication. Clearly, Dana’s AuthAnvil offering from Scorpion Software was a big point of discussion for bringing affordable and easy-to-manage two-factor authentication into the micro and small business arena. But more than just a sales pitch, Dana makes a clear case for the importance of two-factor authentication and how implementation of such a system can significantly improve security for even the smallest operations.

The interesting take on least use privilege, however, was not from the user perspective, but from an administration perspective. Amy and I discussed in our session on security and remote support the importance of realizing that as more and more IT shops begin to provide remote support to their widening client base, those shops cannot and should not increase the security risk to their clients in order to make it easier for them to support those clients.  There was a lot of good discussion during our session stemming from some very insightful questions, and I think we all came away from the day with a good sense of things to think about within our own firms as we move forward.

One practical point that I’m starting to implement in my operation is the use of AuthAnvil to help protect those servers we support who have port 3389 open to the Internet, even temporarily. With a combination of an additional administrator-equivalent account on the network, installation of the AuthAnvil software, and a requirement that access to the server be protected by two-factor authentication, we can significantly reduce the risk of having port 3389 open to the Internet as well as increase the level of documentation when these sites are accessed. That, and it gives us an “in” to discuss two-factor authentication with our clients and work to really help them reduce their own security vulnerabilities.

Thanks to everyone who participated in the conference and helped make it a real benefit to those who were able to attend. Thanks for the insightful questions that got us all thinking, and thanks for the opportunities to not only help others improve their own operations, but to help me bring my own ship in a little tighter as well.

On Follow-Up

In yesterday's post, I covered the problems being seen in the community regarding the unexpected behavior on SBS 2003 R2 boxes because of a problem with a WSUS definition update. Given the volume of traffic that post generated (more hits in the first 4 hours of that post than any other single post on this blog, period), there were a lot of people impacted by this issue, and apparently not a lot of information out there. Yes, i found a number of threads in other discussion foums, but most hinted at the behavior an didn't document the full code of the errors, etc. So there was a lot of internet traffic and human effort expended over this issue yesterday.

Late yesterday afternoon (well, my time anyway) the Official WSUS Blog finally put up a post about the issue and detailed the causes behind it. A few hours earlier, the folks at the Official SBS Blog put up a post detailing the resolution, specifically noting that the normal course of updates for the WSUS services on the server would fix the problem so that today everyone's SBS boxes should be back to normal.

I checked on the last of my managed servers this morning, the one I left untouched to test this theory for myself, and sure enough, it updated and WSUS and the Performance Reports are back to "normal" on the servers.

So, all's well that ends well, right? Ah, not exactly.

This event has raised some concern in the community about the WSUS product and the SBS R2 implementation of WSUS. For the remainder of this post, I'm not speaking for the community, but from my own personal concerns about the topic.

Hindsight allows us to look back and see that, in the grand scheme of things, this was not a major catastrophe. In fact, the server that I left completely untouched yesterday to test the automatic update fix had no performance issues at all. The customer who uses this server didn't lose a piece of e-mail, didn't lose access to the server, didn't lose any productivity, in fact, they were never aware that there was even an issue that we were looking at. That's good, because that's one less client I have to explain this to, and that makes my life a little easier today.

But at the time we were dealing with this yesterday, we didn't have that insight. What initially looked like a Performance monitor issue quicky became a WSUS issue, and in the midst of it, we had no idea if WSUS was completely broken or what it might take to get it back or what other functionality might be affected. To be honest, when something affects a class of devices across the world, I'm a litlte more apt to spend time to figure out how this could be impacting my own client base, who I am ultimately responsible for. The lack of information was frustrating (one of the reasons I put the post up yesterday, so that hopefully someone who was seeing the issue could get concrete evidence that there was a larger problem and someone was looking into it, even if it wasn't an official Microsoft source) and I really, really hate operating in a vacuum. In total, our operation lost 75% of our business day identifying the problem, diagnosing the problem, communicating with others about the problem, and ultimately implementing the workaround for a few of our clients to get them back on track, given that we still didn't know the breadth of the problem. And I know we were not the only business impacted in this way.

Ultimately, I'm concerned that given the nature of the problem and the "fix," the community has absolutely no way to ensure that this issue won't happen again. By the very nature of the way WSUS operates, and specifically the way SBS R2 implements WSUS, the exact type of mistake made by Microsoft yesterday could happen again and bring down thousands of WSUS processes again. This fact is what is giving me serious pause about WSUS in general and the SBS R2 implementation of WSUS.

In the interest of full disclosure, I am NOT a WSUS guru by any stretch of the imagination. The extent of my understanding of the R2 implementation of WSUS is to make sure that I leave the default settings enabled so that I can see the Green Check of Health and not the Blue Check of Misconfiguration, which should help me better identify when my R2 installations are out of compliance. Reports say that those who manually installed WSUS, specifically configuing it to only identify updates that are needed by that particular installation, were not affected by the problem yesterday. In fact, since the problematic update was for a BETA build of a product that I do not have installed at ANY of my client sites since I am not participating in that particular beta, I should not have had any system pull down the dictionary for that particular product. But somehow, an SBS R2 box with a single NIC card (i.e., could never run ISA to begin with, much less one that was not participating in the ISA Nitro beta) got the definition update for this beta program and lived with a crashed WSUS for a full 24 hours. At least, that's the way I understand it, given my relative inexperience with WSUS.

This simply should not have happened.

For the next few days, I now get to spend time learning about WSUS and see how I can modify the configuration of WSUS on the servers I manage to minimize the risk of this happening again. This means I have to reprioritize my workload so that I can try to make sure my clients have a lower risk of being affected by a problem that, quite frankly, may never appear again. But given Murphy's Law, if I take the road that it won't happen again so I don't need to do anything, as soon as I leave the country (which is happening in less than a week) another mistake will happen that will impact these boxes, and the rest of my operation will be left scrambling to deal with the issue while I'm stuck in a plane. Thanks a lot, Microsoft, for recalibrating my work week for me.

Understand, I don't specifically fault Microsoft for making a mistake. Who among us hasn't made mistakes? Though some have said that this type of mistake shoudl never have occured, well, stuff happens, you know. What I do fault Microsoft for is the design of the system which allowed this particular mistake to have such a widespread impact on systems that should never have seen this specific update, ever. How did a server that's not even capable of running ISA get a definition update for a product that's not even a released product? This is what I have to spend time on now, getting a better understanding of how WSUS works so that I better understand the risks I am putting on my clients by using this tool.

Wait, did I just say that running WSUS increases the risk vector for my clients? I thought the entire purpose of WSUS was to help reduce the risk vector for my clients. Ironic.

On Errors

This morning (November 12, 2007) a rash of reports are floating around the net about problems viewing the Monitoring report on SBS servers. This appears to be an issue with SBS R2 servers with the R2 WSUS installed. Several factors indicate the problem:

1. The daily monitoring e-mail shows the following instead of the normal performance report:

   	The page cannot be displayed
   An error occurred on the page you are trying to view.
   To work around this problem, perform the following steps. After each step, try again to access the page. Ensure that the MSSQL$SBSMONITORING service is started. Ensure that the server is not low on memory or disk space. Restart the server. Verify that the server is functional and that there are no system-wide problems. Run the Set Up Monitoring Reports and Alerts task in the Server Management Monitoring and Reporting taskpad.

2. You will see the same verbage when you open the Monitoring node in the Server Management console on the server.
3. When you look in the event logs, you will see an error similar to:

Event Type: Error
Event Source: ServerStatusReports
Event Category: None
Event ID: 1
Date:  11/12/2007
Time:  6:00:15 AM
User:  N/A
Computer: SERVER
Description:
Server Status Report:
 URL:  http://localhost/monitoring/perf.aspx?reportMode=1&allHours=1
 Error Message: The specified string is invalid.
Parameter name: Title
 Stack Trace:    at Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer..ctor(GenericReadableRow row)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory..ctor(GenericReadableRow row)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.BuildUpdateCategoryCollection(GenericReadableRow[] categoryRows)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.GetAll(DateTime fromSyncDate, DateTime toSyncDate)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories(DateTime fromSyncDate, DateTime toSyncDate)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories()
   at Microsoft.SBS.UpdateServices.DataProvider.GetScheduledUpdates()
   at Microsoft.SBS.UpdateServices.StatusPage.Utility.GetStatusItems()
   at usage.frmPerf.PopulateStatusItems()
   at usage.frmPerf.renderReportWorker()
   at usage.frmPerf.renderReport()

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
4. When you try to open the Update Services node in the Server Management console, you will see a page similar to the following:

   Server Error in '/UpdateServices' Application.
   --------------------------------------------------------------------------------

   The specified string is invalid. Parameter name: Title
   Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

   Exception Details: System.ArgumentException: The specified string is invalid. Parameter name: Title

   Source Error:

   
   Line 194:  </div>
   Line 195:  <%Response.Flush();
   Line 196:    RenderPage();%>
   Line 197:  <div id="divForm" style="display: none">
   Line 198:  <form id="formMain" method="post" runat="server">
    

   Source File: c:\inetpub\UpdateServices\Home.aspx    Line: 196

   Stack Trace:

   
   [ArgumentException: The specified string is invalid.
   Parameter name: Title]
      Microsoft.UpdateServices.Internal.StringValidation.ValidateUpdateContainerTitleString(String paramName, String value) +256
      Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer.set_Title(String value) +19
      Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer..ctor(GenericReadableRow row) +182

   [WsusInvalidDataException: The specified string is invalid.
   Parameter name: Title]
      Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer..ctor(GenericReadableRow row) +397
      Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory..ctor(GenericReadableRow row) +24
      Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.BuildUpdateCategoryCollection(GenericReadableRow[] categoryRows) +418
      Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.GetAll(DateTime fromSyncDate, DateTime toSyncDate) +134
      Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories(DateTime fromSyncDate, DateTime toSyncDate) +23
      Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories() +52
      Microsoft.SBS.UpdateServices.DataProvider.GetScheduledUpdates() +140
      Microsoft.SBS.UpdateServices.StatusPage.Utility.GetStatusItems(Boolean waitingForSyncStart) +3199
      Microsoft.SBS.UpdateServices.StatusPage.formHome.RenderPage() +23
      ASP.Home_aspx.__Render__control1(HtmlTextWriter __output, Control parameterContainer) in c:\inetpub\UpdateServices\Home.aspx:196
      System.Web.UI.Control.RenderChildren(HtmlTextWriter writer) +27
      System.Web.UI.Control.Render(HtmlTextWriter writer) +7
      System.Web.UI.Control.RenderControl(HtmlTextWriter writer) +243
      System.Web.UI.Page.ProcessRequestMain() +1926

   --------------------------------------------------------------------------------
   Version Information: Microsoft .NET Framework Version:1.1.4322.2407; ASP.NET Version:1.1.4322.2407

    

[Edited at 3:00pm CST]
I think there's a workable resolution for this now. I've done this on a couple of my sites, and it's resolved the issue. Here are the step-by-step instructions that got this working:

1. Open a command prompt.
2. Type the following at the prompt and hit Enter:
   osql -E -S %COMPUTERNAME%\WSUS
3. Type the following lines at the prompt and press Enter after each one:
   use SUSDB
   Update tbPrecomputedCategoryLocalizedProperty
   Set Title = Replace(Title, '"', '')
   Where Title like '%"%'
   go
   [note that in the Set Title line, the characters are single-quote, double-quote, single-quote following Title and two single-quotes before the close parenthesis; also, in the Where Title line, the characters are single-quote, percent, double-quote, percent, single-quote]
4. After you enter the "go" line, you should get a response that tells you how many rows were affected.
5. Type the following lines at the prompt and press Enter after each one:
   Update tbPreComputedLocalizedProperty
   Set Title = Replace(Title, '"', '')
   Where Title like '%"%'
   go
   [note that the Set Title and Where Title lines are exactly the same as the first set of commands you entered]
6. After you enter the "go" line, you should get a response that tells you how many rows were affected. It may take a little longer for this one to process.
7. Type "quit" and press Enter to get out of OSQL.

Once you finish this, you should be able to go back into the Update Services node of Server Management and click Refresh to bring up the WSUS status again. Please note that I personally have only done this work for a couple of systems, and it resolved the issue (for the time being) on those boxes. This is NOT a permanent fix and could well break again if Microsoft issues another update that includes doulbe-quotes in the update title.

More information about the behind-the-scenes reasons for these issues were found in a thread at forums.techarena.in and another at SANS. At the time of this edit, there has been no info posted at the WSUS team blog.

[Edited at 3:40pm CST]
I have been informed that the faulty information that was pushed into WSUS yesterday has been updated and *should* automatically get pulled in during the next scheduled WSUS update. WSUS 3.0 can do a manual sync to get the update now, and WSUS 2.0 should get it at 10pm local time tonight. A post is expected from the Official SBS Blog later today. I plan to leave one of my servers in this state to confirm that this operation works as expected tonight, but will manually run the osql steps tomorrow if it does not.