OnQ : Frustrations

On SA Fulfillment

eriq — Thu, 02 Apr 2009 16:35:55 GMT

This past week, I went through the process to get the SBS 2003 Software Assurance fulfillment for a couple of my customers who are finally needing to activate those SA rights. The process was a little more cumbersome than I had hoped, and I could not find a location that documented the process in a single location. So, to hopefully help someone else who may be needing to get this done, and to make you aware of a couple of hiccups in the process, here's the scoop.

First, you have to place an order for the SBS 2008 media from MS Fulfillment. Unlike many other products where you can download the installation media through the eOpen site, SBS has the installation key printed and attached to the media set, so you have to get a media order placed. This is done through the VLSC Fulfillment line at 800-336-0098, You will need the Open License number and Authorization code to confirm the license, then you will confirm other details of the order. So long as you have the media shipped to the license holder, there is no additional shipping charge. Piece of cake, once you know the right number to call.

Second, you need to call a different group if you are activating SA for SBS 2003 Premium, because that comes with additional media (Server 2003, ISA 2006, Outlook, Sharepoint Designer). The number to call for that fulfillment is 866-326-7110. The first order I called in for had no problems. We provided the License Number and verified the customer details, and the order was placed. The second time I called in I reached an agent who had no idea what I was talking about. After going round and round with the agent, he checked with his supervisor and got a laundry list of things they wanted us to provide to be able to process the order. I opted not to continue down that road (since I didn't when we placed the first order and by all accounts I shouldn't have had any issues). I called in again later hoping to reach a different agent who knew what I was talking about, and when he also had no idea what the SBS 2003 SA Fulfillment entailed, I bailed and will wait to try again later to get someone who is able to figure out what's going on.

I've heard other anecdotal evidence of issues with getting the SBS 2003 Premium Make Good materials, so it's clear that this is not a common process that this group is dealing with. However, if you need to get the SA media for either, hopefully you won't run into the same type of trouble that I am.

UPDATE:
Just to make sure I hadn't got my wires crossed, I called back into the 866-326-7110 number and asked to confirm that the order we placed a couple of days ago had actually shipped. This wasn't a bogus call, as we did not get a shipping confirmation e-mail with the tracking number like we were told. The agent who answered the phone was able to pull up the order and provide the shipping confirmation (they hadn't entered the e-mail address after all, which is why no shipping notification was received). So I know I'm talking to the right people after all. After getting the confirmation, I asked if she could place the smae order for a different customer, and she got the order completed without any issues, just like the first order we placed.

I'm not sure yet what the "magic words" are to get the agents at the second line to look in the right place in the script to get the SBS 2003 Premium SA order done, but I don't have many more of these to do, so I'll wing it should I have any trouble in the future. And if I do figure out the "magic words" I'll post back here.

On Migration - Exchange and Quotas

eriq — Mon, 27 Oct 2008 17:45:00 GMT

This post will be the first in what I hope will be a short series related to issues I've encountered in my SBS2008 migration of my internal server. I'd love to say "hey, everything worked as advertised" but we're not quite that lucky. But to start off, I have a very common setup that will probably catch some folks off-guard, so here it is in the blogosphere for someone to find and figure out.

First, I have a large mailbox. Like 8GB. Yeah, it needs pruning, I know. My first hint of trouble was when I ran the move mailbox part of the SBS migration. The move mailbox process stopped because of the default quotas in Exchange 2007. Yes, 2GB is far better than the 200MB default in Exchange 2003, but that's beside the point. So I went right in and removed the quotas on the mail store and fired up the move mailbox tool again. This time it completed without error.

Jump to this morning. I am going through the migration CHM file and get to the point about the users not showing up in the SBS console. I can either do the ADSIEDIT fix for that, or I can just run the Change User Role wizard (or whatever it's called). Boom, all of my users show up in the console. Woo hoo!

A little bit later, Susan (yes, that Susan) pings me that my mail is getting rejected because my mailbox is full. D'oh! That's right, the default Standard User role has the 2GB Exchange quota enabled by default. Dummy that I was, I didn't change the user role before applying it (I actually ran across this during migration testing and made a note about changing the Standard User role to remove the Exchange quota, but forgot about it this morning while I'm trying to do a gajillion other things) and now my 8GB mailbox is dying over 2GB quota. No problem, I change the User Role, then I open my user object to make sure the setting is removed, and all is well.

Would that it were that simple.

An hour later, I still can't get into OWA (well, I can get in, but it's yelling at me that I'm over quota and won't do anything until I fix it - if you haven't seen the OWA UI for being over quota, try it - there's nothing subtle about it). I quit and restart Outlook, it fires off the Mailbox Cleanup wizard (again, yes, I should do a cleanup, but that's not the issue). I look at the my user object in the SBS Console, no quota. I look at the message store in the Exchange console, no quota. I ping David Shackelford and he has me run a couple of PS cmdlets, all show the quota is disabled. Dave suggests I restart the Transport service. No love.

Finally, Dave sends me to http://technet.microsoft.com/en-us/library/bb684892.aspx and suggests I restart the Information Store service. Bingo, OWA and Outlook stop yelling at me, and mail starts flowing again.

I don't know why Exchange couldn't figure out in a two-hour window that I had adjusted the quota settings, but it didn't. I had to forcibly restart the information store before it woudl check the quota again and allow me to get back to my precious e-mail.

So, lessons learned for today:

Modify the quota settings on the mail store before migrating any user mailboxes over if you have users with mailboxes over 2GB.
Modify the Standard User Role before you touch any user objects during your migration and remove the Exchange quota if you have users who are above the limit.
If you should get a user who goes above the quota and you need to restore their access quickly, adjust the quota settings, then restart the Information Store service. No, you shouldn't have to do that, but it fixes the issue.

On 18 months

eriq — Fri, 26 Sep 2008 18:54:17 GMT

Back in March 2007, I put up a couple of posts about the missing audible alarm in the Dell PERC RAID controller. There was a flurry of activity following those posts, and that's been the most read pair of posts from this blog. Well, 18 months has passed, and I haven't posted much else about the situation, mostly because nothing much has changed. One of the last conversations I had with someone within Dell engineering was that they were looking into adding the alarm back in the PERC 6 controller, but no revisions were planned for the PERC 5. I'm OK with the latter part of that, as it would be difficult to go back and re-engineer the existing card, so long as the next card would get it back.

But last week I put in two new servers, the first ones I've rolled out with the PERC 6 controller, and guess what - no audible alarm on the controller. Either version. (One of the servers I rolled has both the PERC 6/i and the 6/E, neither has the alarm.)

I had reason to contact Dell yesterday for another set of issues and, as luck would have it, I spoke with the same engineer who worked with me the first time I called in about the missing alarm. While he helped me resolve the other issues, we talked more about the missing alarm and how I was really disappointed to find out that the latest card still doesn't have it. Once again, I asked about how best to provide feedback to Dell regarding this issue, and once again he pointed me to the Dell IdeaStorm site. While talking, we both went to the site and searched on "audible alarm" and guess whose post was the first one returned in the search - yep, the one I put up in April of 2007. Surprisingly, unlike other posts with lower scores on that page, this item has not been marked as "reviewed" which means that Dell has not been keen enough to even look at this issue.

Well, it's time to get back in the saddle on this issue, because I think that Dell has made a significant mistake in removing the alarm from the controller. Sure, I understand that they received a number of calls from customers who didn't like hearing the alarm when something went wrong with the controller, but the correct response is not to remove the alarm altogether. Ship the controller with the alarm disabled by default, and let those of us who rely on the alarm as a warning sign enable it if we choose.

So, for those of you who work with Dell servers who still see this as a huge shortcoming, please take as many of the following actions as you are comfortable with.

Locate my post on IdeaStorm and comment/promote it.
Contact Dell Support for one of your servers in production and ask them how to enable the audible alarm on the PERC controller. When they tell you it can't be done, escalate the call.
Contact your sales agent at Dell and explain to them how you're considering switching to a different hardware vendor that does offer RAID controllers with audible alarms.

This is one of those cases where I think the community absolutely can make a difference.

On Updates and PERCs

eriq — Mon, 28 Jan 2008 14:45:00 GMT

I've had a few comments show up on the series of PERC 5/i posts I had early in 2007. There have been a few questions about the status of things, so rather than respond in the comments, I thought I'd summarize what I know at this point in a separate post.

Bottom line, the alarm status of the PERC 5/i has not changed, nor will it in all likelihood. In discussions with the engineering folks at Dell, apparently there were a number of people who complained about the alarm with such ferocity that the design team decided it was best to take it out altogether. Akin to driving in a nail with a sledgehammer, I think this was a misguided and completely incorrect overreaction to the problem. If Dell wanted to make the default setting on the controller to have the alarm OFF instead of ON, I would have been fine with it. Make me enable the audible alarm if I want it, but leave it off for those who don't. But to completely remove the functionality is just beyond me.

At this time, I cannot tell if the next series of the PERC controller will have this functionality restored. The engineering folks that I spoke with said that the feedback they're getting is still in favor of having the alarm removed. I said "you're talking with the wrong people, then." I invited them to have their researchers include me in their feedback request, and I'd be more than happy to add the logical reasons to have the ability to have an alarm present but quiet by default. If you're in the same boat as I am, please take a moment to contact Dell Support and voice your thoughts on the matter. Apparently (as I've been told), they listen to loud feedback.

Secondly, on the issue of proactive monitoring, Dell still does not have a tool that will generate an alert if the array goes into a degraded condition. We have been using HoundDog to provide proactive monitoring of the health of the array for my systems with the PERC 5 controllers, and it has worked very well for our operation. You do have to install the Server Administrator tools to generate the SNMP alerts that HoundDog picks up on, but cost of the HoundDog service and running Server Administrator on the box is far less than the cost of not knowing that an array is having trouble. There are, of course, a couple of quirks. One, the SMNP trap will alert on battery conditions on the controller as well, and not tell you it's a battery issue in the alert. If the server gets powered off for any reason, the battery on the controller starts "draining" to maintain the configuration information, and when the server is powered back on, the battery goes into "recharge" mode, which triggers an SNMP even in Server Administrator, and then by HoundDog. I haven't been able to find out how to modify that, but it's probably good to know about battery conditions, so I'm leaving it alone. Two, not all Dell servers run Server Administrator, specifically the SC-series servers. I have one SC server with a mute PERC controller, and I'm tring to figure out how to monitor that. SNMP doesn't work, as Server Administrator isn't present to generate the SNMP configuration that HoundDog is looking for. I downloaded and installed the LSI software, but it has no mechanism for generating alerts, so I'm still digging on that one.

So we're not fully there yet, but getting comfortable. I will not be purchasing an SC-class server from Dell in the future, but outside of that, HoundDog has given me what I need to keep my team alerted to any problems with array controllers at a very minimal cost. I would still prefer to have the option of dealing wiht an audible alarm, and I still mention my frustration with that every time I call Dell support (which really isn't that often, but I did speak with the original tech who took my call last February over the past week about another issue, and he rememberd who I was and brought up the subject himself).

On Patience

eriq — Fri, 11 Jan 2008 16:14:00 GMT

Sometimes life throws you a curve ball. Sometimes it throws you a curve ball, followed by a sinker, a wicked change-up, then a knuckleball. Yeah, that's my story this week.

I was asked to come to Redmond to work on a training and certification project for Microsoft. I've made a number of trips to Seattle, so making arrangements was nothing new. The company bringing us out had us booked into a hotel that wasn't really close to the MS campus, and I didn't feel like dealing with a shuttle back and forth all week, so I decided to rent a car for this trip. Ooh! New and exciting! I'll finally get to drive around the area and begin to really get a feel for the layout of the Seattle area. I made arrangements for the car, had the flight booked, hotel was set, so last Sunday I started my travel adventure.

Please note, all names have been changed to protect the innocent. From me.

An associate of mine, I'll call him Bill since that's not his name, was coming for the same project and arrived a couple of hours before I did. I had let him know of my plans for the car rental, but had told him there was really no point in waiting 2.5 hours after his very long flight just to catch a ride with me. Well, as soon as I touched down, I got a text message from Bill with his cell phone number, then got a call from Bill. Turns out Bill's flight had been delayed in Salt Lake City because of snow, and he was still at the airport. I offered to take Bill to the hotel once I picked up the rental car, and plans were made.

So I picked up my bags from the carousel (after about a 25 minute delay) and headed over to the car rental stand. When I made my reservation with Steve's Auto Rental (again, not their name), I chose them not because they were the cheapest (they weren't) or the most expensive (they weren't) but becuase they were one of the lowest cost rental places with a gate at the airport, even though you had to take a shuttle to their actual site. I mention that specifically because when we got to the gate at the airport, it was closed and we were instructed to take the shuttle directly. Finding the shuttle was fun, but the shuttle driver finally found us (the lot told him the wrong pick up point) and we made our way to the lot to get the car.

Back in 2003, I had lasik laser vision correction performed so I no longer had to lug my glasses around. Second-best health thing I have done for myself, and I highly recommend it to anyone who wears glasses. Why do I mention this? Because I turned 40 the Friday before I left for Seattle. Still not seeing the connection? My driver's license expired when I turned 40. Still guessing? I never got around to removing my glasses restriction from my license when I had lasik done, so I went down to the DMV, on my birthday (why do anything in advance) to take the vision test in person and get the restriction removed. Which means I have a temporary license card until they send me the new one in 2-3 weeks.

I told you that to tell you this - when I went to get my license out to complete the rental of the car, the temporary card wasn't in my wallet. Gone. Must have fallen out at some point between Friday and Monday and just wasn't there. So I rummaged around trying to find it, hoping the counter clerk might take pity, but that's the one thing she absolutely stuck to was not renting to me with an expired license. Fortunately, Bill was there with me and he took the reservation and the car. Whew! Situation resolved.

It still took a long time to get through the process of getting the car, even with the proper documentation, so it was nearly 30 minutes later before Bill was able to go get the car. He pulled the car up to the front door of the lot, where our luggage was, and leaving it running to warm up the heater (did I mention it's cold in Seattle this time of year?) he got out and stepped around back to open the hatch on the back of the SUV import, but couldn't get it open. We both tried to get it to open, to no avail. The "handle" as it were was not terribly obvious, so Bill went back to the driver's seat to see if there was a secret latch needed to open the hatch when he realized that the driver's door was locked. With the car running. Well, at least we knew why the hatch wouldn't open.

Bill went back in to the rental agent and explained the dilemma. And we waited. Because they have no spare keys for the vehicles. And the rental agent wasn't really sure what to do. Shortly, Bill spoke with a manager who attempted to make a replacement key based on the keycode for the car. No dice. A short while later, the manager made another key, thinking the first key wasn't made properly. No dice. The guy who really knew about the keys was driving the shuttle, and we had to wait for him to return. He tried to make a key, same result. Eventually, he tried making a key using the keycode for the other vehicle of the same make/model, just in case, and it worked. We quickly loaded our luggage in the vehicle and headed towards the hotel. And arrived just a scan 3 hours after my flight landed.

Fast forward to my return trip on Thursday. I arrived at the airport shortly before 8am for a 10am flight back to DFW. Having plenty of time, I meandered through the bookstores to get something to read on the flight back, then waited for my regular food spot to open so I could get "lunch" before getting on the plane. I got to the terminal about 40 minutes before flight time, 10 minutes before boarding should have started. And noticed that my flight number was on a different gate. And was showing a delayed flight time. Noon instead of 10. So, i stood in line to see what the story was, and the gate agents start relaying a lot of information. First, we'll all have to go pick up our luggage from the baggage claim area and recheck at the ticket counter. Then those who are terminating their flight in Dallas won't have to pick up their luggage, but everyone else will. Then those who are connecting through to Cancun won't have to pick up their luggage, and they've already been moved to a different flight, but they were processed before the rest of us were. Then the gate agents (now plural instead of singular), started handing out cards with the American Airlines reservation number on them so we can call and make other arrangements, because now the flight has been bumped back to 3:15pm instead of noon. Only I didn't get a card, because I'm next in line, and apparently it will take less time for me to wait to talk with the gate agent than it will to call in.

The 11:30 flight to Dallas was already overbooked because of the adjustments they made to those with connecting flights, but they got me on standby. But they did go ahead and book me a confirmed seat on the 1:30 flight to Dallas. Good thing that Seattle to Dallas is a popular route, because some folks were booked onto later flights, but at least I think everyone got an alternate flight arranged with minimal difficulty.

While waiting for the standby line for the 11:30 flight, I received an automated noticiation that the flight originally scheduled for 9:50am, then 12:00, then 3:15, had just flat been cancelled. Suddenly I was really glad I didn't decide to just wait and stay on the original flight. As expected, I didn't make the 11:30 flight, which didn't actually leave until 12:00, so after they cleared the flight, I was called up to get my boarding pass for the 1:30 flight. Then I went back to my corner to wait.

While waiting for the 1:30 flight, the gate agents started making announcements about that fight being full and those who had not already checked in for standby but were wanting standby weren't going to get standby on that flight and to wait until the next flight at 2:50. 1:00 came and went, and the plane we were supposed to get on to fly out at 1:30 wasn't at the terminal yet, so no one had deplaned and the crew had not started cleaning the plane when we should have been boarding. Huge surprise given the way the day had gone thus far.

At about 1:15, the gate agent started asking for those with flexible travel plans to check in at the counter, as they were going to offer them $300 flight vouchers if they would take a seat on the 2:50 flight, which they pointed out *was* running on time. I was really, really tempted to volunteer my seat, but since that would have put me back in Dallas (if everything were actually on time) at 9pm and my wife isn't crazy about driving in the dark, I passed on the offer for basically a free round-trip back to Seattle and stuck with my booked 1:30 flight.

Which started boarding at 1:45.

Once I finally got on the plane, I realized that I was in a bulkhead seat, the first row immediately behind the first class section. While I initially rejoiced at the notion that I'd have all kinds of leg room, I also realized that I wouldn't be able to stash my carryon on the floor near me. And since that particular seat had no room in its overhead bin (that's where they store the "emergency" supplies), I had to stick my carryon items in the overhead compartment across the aisle. Fortunately, I had just bought a new (and fairly long) book, so I pulled that from my backpack and settled in with lots of legroom for a four hour flight with a good book.

As it turns out, the flight didn't have any issues once we took off (just after 2pm) and with the tailwinds helping push us west to east, the four hour flight only took about 3 hours. All in all, I arrived safely, although later than I would have liked. The flight didn't take off until well after I was originally scheduled to land, but again, it could have been much worse. I could have been with the group trying to get to Cancun who ultimatly missed their connecting flight. Or the group going to Ft. Lauderdale who opted to reroute to Miami and drive to Ft. Lauderdale from there. Or the couple trying to get to Panama City who ended up having to stay overnight in Dallas to catch the first flight the next day. I was able to sleep at home in my own bed and didnt' have to drive another 4 hours in a car following a landing.

Can't wait to see what transpires on my next flight back from Seattle. In February.

On Downgrading

eriq — Tue, 18 Dec 2007 16:04:00 GMT

Well, my experiment with Leopard and Parallels has come to a close. As of this morning, I'm back running Tiger and Parallels build 4128. I was able to get Build 5582 running on Leopard and mostly had everything the way it should be, but performance was HORRIBLE! Now, I had made a number of changes all at the same time, so it could have been any one of them. I was running Leopard on an external firewire drive, so that could have caused performance issues (however, my XP hard disk image has been running on another external firewire drive for months, and that posed no problems). I also updated to OFfice 2007 SP1 because of the lingering performance issues I've seen with Office 2007 in general, and overall that seems to have done nothing. Plus, I did upgrade Parallels from 4128 to 5582, and who know.

So I retracted in several steps. First, I went back to my default Tiger system, keeping Office 2007 SP1 installed and Parallels build 5582. Same performance issues. So it's not Leopard itself or running the main OS from an external firewire disk that caused the problems. Then I went back and installed Parallels build 4128, and bingo, all my performance issues went away., even leaving Office 2007 SP1 on (I won't say that Office 2007 performance has improved over what I was seeing, it just went back to what it had been before the Leopard/Parallels test.

So, to keep my sanity and performance intact, I'm sticking with this for the time being. Next step will be to give that Fusion a real run for it's money and see how insane that makes me before I go back to this configuration. I must say, though, that I am getting more and more disappointed with the steps, or lack thereof, that Parallels is making with their recent builds. And I've been unable to get anyone at Parallels to follow up on the requests I've made for assistance, even after following their guidelines for requesting support. We'll see what Fusion has to say about all of this.

On Follow-Up

eriq — Tue, 13 Nov 2007 12:25:00 GMT

In yesterday's post, I covered the problems being seen in the community regarding the unexpected behavior on SBS 2003 R2 boxes because of a problem with a WSUS definition update. Given the volume of traffic that post generated (more hits in the first 4 hours of that post than any other single post on this blog, period), there were a lot of people impacted by this issue, and apparently not a lot of information out there. Yes, i found a number of threads in other discussion foums, but most hinted at the behavior an didn't document the full code of the errors, etc. So there was a lot of internet traffic and human effort expended over this issue yesterday.

Late yesterday afternoon (well, my time anyway) the Official WSUS Blog finally put up a post about the issue and detailed the causes behind it. A few hours earlier, the folks at the Official SBS Blog put up a post detailing the resolution, specifically noting that the normal course of updates for the WSUS services on the server would fix the problem so that today everyone's SBS boxes should be back to normal.

I checked on the last of my managed servers this morning, the one I left untouched to test this theory for myself, and sure enough, it updated and WSUS and the Performance Reports are back to "normal" on the servers.

So, all's well that ends well, right? Ah, not exactly.

This event has raised some concern in the community about the WSUS product and the SBS R2 implementation of WSUS. For the remainder of this post, I'm not speaking for the community, but from my own personal concerns about the topic.

Hindsight allows us to look back and see that, in the grand scheme of things, this was not a major catastrophe. In fact, the server that I left completely untouched yesterday to test the automatic update fix had no performance issues at all. The customer who uses this server didn't lose a piece of e-mail, didn't lose access to the server, didn't lose any productivity, in fact, they were never aware that there was even an issue that we were looking at. That's good, because that's one less client I have to explain this to, and that makes my life a little easier today.

But at the time we were dealing with this yesterday, we didn't have that insight. What initially looked like a Performance monitor issue quicky became a WSUS issue, and in the midst of it, we had no idea if WSUS was completely broken or what it might take to get it back or what other functionality might be affected. To be honest, when something affects a class of devices across the world, I'm a litlte more apt to spend time to figure out how this could be impacting my own client base, who I am ultimately responsible for. The lack of information was frustrating (one of the reasons I put the post up yesterday, so that hopefully someone who was seeing the issue could get concrete evidence that there was a larger problem and someone was looking into it, even if it wasn't an official Microsoft source) and I really, really hate operating in a vacuum. In total, our operation lost 75% of our business day identifying the problem, diagnosing the problem, communicating with others about the problem, and ultimately implementing the workaround for a few of our clients to get them back on track, given that we still didn't know the breadth of the problem. And I know we were not the only business impacted in this way.

Ultimately, I'm concerned that given the nature of the problem and the "fix," the community has absolutely no way to ensure that this issue won't happen again. By the very nature of the way WSUS operates, and specifically the way SBS R2 implements WSUS, the exact type of mistake made by Microsoft yesterday could happen again and bring down thousands of WSUS processes again. This fact is what is giving me serious pause about WSUS in general and the SBS R2 implementation of WSUS.

In the interest of full disclosure, I am NOT a WSUS guru by any stretch of the imagination. The extent of my understanding of the R2 implementation of WSUS is to make sure that I leave the default settings enabled so that I can see the Green Check of Health and not the Blue Check of Misconfiguration, which should help me better identify when my R2 installations are out of compliance. Reports say that those who manually installed WSUS, specifically configuing it to only identify updates that are needed by that particular installation, were not affected by the problem yesterday. In fact, since the problematic update was for a BETA build of a product that I do not have installed at ANY of my client sites since I am not participating in that particular beta, I should not have had any system pull down the dictionary for that particular product. But somehow, an SBS R2 box with a single NIC card (i.e., could never run ISA to begin with, much less one that was not participating in the ISA Nitro beta) got the definition update for this beta program and lived with a crashed WSUS for a full 24 hours. At least, that's the way I understand it, given my relative inexperience with WSUS.

This simply should not have happened.

For the next few days, I now get to spend time learning about WSUS and see how I can modify the configuration of WSUS on the servers I manage to minimize the risk of this happening again. This means I have to reprioritize my workload so that I can try to make sure my clients have a lower risk of being affected by a problem that, quite frankly, may never appear again. But given Murphy's Law, if I take the road that it won't happen again so I don't need to do anything, as soon as I leave the country (which is happening in less than a week) another mistake will happen that will impact these boxes, and the rest of my operation will be left scrambling to deal with the issue while I'm stuck in a plane. Thanks a lot, Microsoft, for recalibrating my work week for me.

Understand, I don't specifically fault Microsoft for making a mistake. Who among us hasn't made mistakes? Though some have said that this type of mistake shoudl never have occured, well, stuff happens, you know. What I do fault Microsoft for is the design of the system which allowed this particular mistake to have such a widespread impact on systems that should never have seen this specific update, ever. How did a server that's not even capable of running ISA get a definition update for a product that's not even a released product? This is what I have to spend time on now, getting a better understanding of how WSUS works so that I better understand the risks I am putting on my clients by using this tool.

Wait, did I just say that running WSUS increases the risk vector for my clients? I thought the entire purpose of WSUS was to help reduce the risk vector for my clients. Ironic.

On Errors

eriq — Mon, 12 Nov 2007 15:18:00 GMT

This morning (November 12, 2007) a rash of reports are floating around the net about problems viewing the Monitoring report on SBS servers. This appears to be an issue with SBS R2 servers with the R2 WSUS installed. Several factors indicate the problem:

The daily monitoring e-mail shows the following instead of the normal performance report:

	The page cannot be displayed
An error occurred on the page you are trying to view.
To work around this problem, perform the following steps. After each step, try again to access the page. Ensure that the MSSQL$SBSMONITORING service is started. Ensure that the server is not low on memory or disk space. Restart the server. Verify that the server is functional and that there are no system-wide problems. Run the Set Up Monitoring Reports and Alerts task in the Server Management Monitoring and Reporting taskpad.

You will see the same verbage when you open the Monitoring node in the Server Management console on the server.
When you look in the event logs, you will see an error similar to:

Event Type: Error
Event Source: ServerStatusReports
Event Category: None
Event ID: 1
Date:  11/12/2007
Time:  6:00:15 AM
User:  N/A
Computer: SERVER
Description:
Server Status Report:
URL:  http://localhost/monitoring/perf.aspx?reportMode=1&allHours=1
Error Message: The specified string is invalid.
Parameter name: Title
Stack Trace:    at Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer..ctor(GenericReadableRow row)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory..ctor(GenericReadableRow row)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.BuildUpdateCategoryCollection(GenericReadableRow[] categoryRows)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.GetAll(DateTime fromSyncDate, DateTime toSyncDate)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories(DateTime fromSyncDate, DateTime toSyncDate)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories()
   at Microsoft.SBS.UpdateServices.DataProvider.GetScheduledUpdates()
   at Microsoft.SBS.UpdateServices.StatusPage.Utility.GetStatusItems()
   at usage.frmPerf.PopulateStatusItems()
   at usage.frmPerf.renderReportWorker()
   at usage.frmPerf.renderReport()

http://go.microsoft.com/fwlink/events.asp

When you try to open the Update Services node in the Server Management console, you will see a page similar to the following:
Server Error in '/UpdateServices' Application.
--------------------------------------------------------------------------------

The specified string is invalid. Parameter name: Title
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ArgumentException: The specified string is invalid. Parameter name: Title

Source Error:

Line 194:  </div>
Line 195:  <%Response.Flush();
Line 196:    RenderPage();%>
Line 197:  <div id="divForm" style="display: none">
Line 198:  <form id="formMain" method="post" runat="server">

Source File: c:\inetpub\UpdateServices\Home.aspx    Line: 196

Stack Trace:

[ArgumentException: The specified string is invalid.
Parameter name: Title]
   Microsoft.UpdateServices.Internal.StringValidation.ValidateUpdateContainerTitleString(String paramName, String value) +256
   Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer.set_Title(String value) +19
   Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer..ctor(GenericReadableRow row) +182

[WsusInvalidDataException: The specified string is invalid.
Parameter name: Title]
   Microsoft.UpdateServices.Internal.BaseApi.UpdateContainer..ctor(GenericReadableRow row) +397
   Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory..ctor(GenericReadableRow row) +24
   Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.BuildUpdateCategoryCollection(GenericReadableRow[] categoryRows) +418
   Microsoft.UpdateServices.Internal.BaseApi.UpdateCategory.GetAll(DateTime fromSyncDate, DateTime toSyncDate) +134
   Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories(DateTime fromSyncDate, DateTime toSyncDate) +23
   Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetUpdateCategories() +52
   Microsoft.SBS.UpdateServices.DataProvider.GetScheduledUpdates() +140
   Microsoft.SBS.UpdateServices.StatusPage.Utility.GetStatusItems(Boolean waitingForSyncStart) +3199
   Microsoft.SBS.UpdateServices.StatusPage.formHome.RenderPage() +23
   ASP.Home_aspx.__Render__control1(HtmlTextWriter __output, Control parameterContainer) in c:\inetpub\UpdateServices\Home.aspx:196
   System.Web.UI.Control.RenderChildren(HtmlTextWriter writer) +27
   System.Web.UI.Control.Render(HtmlTextWriter writer) +7
   System.Web.UI.Control.RenderControl(HtmlTextWriter writer) +243
   System.Web.UI.Page.ProcessRequestMain() +1926

--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:1.1.4322.2407; ASP.NET Version:1.1.4322.2407

[Edited at 3:00pm CST]
I think there's a workable resolution for this now. I've done this on a couple of my sites, and it's resolved the issue. Here are the step-by-step instructions that got this working:

Open a command prompt.
Type the following at the prompt and hit Enter:
osql -E -S %COMPUTERNAME%\WSUS
Type the following lines at the prompt and press Enter after each one:
use SUSDB
Update tbPrecomputedCategoryLocalizedProperty
Set Title = Replace(Title, '"', '')
Where Title like '%"%'
go
[note that in the Set Title line, the characters are single-quote, double-quote, single-quote following Title and two single-quotes before the close parenthesis; also, in the Where Title line, the characters are single-quote, percent, double-quote, percent, single-quote]
After you enter the "go" line, you should get a response that tells you how many rows were affected.
Type the following lines at the prompt and press Enter after each one:
Update tbPreComputedLocalizedProperty
Set Title = Replace(Title, '"', '')
Where Title like '%"%'
go
[note that the Set Title and Where Title lines are exactly the same as the first set of commands you entered]
After you enter the "go" line, you should get a response that tells you how many rows were affected. It may take a little longer for this one to process.
Type "quit" and press Enter to get out of OSQL.

Once you finish this, you should be able to go back into the Update Services node of Server Management and click Refresh to bring up the WSUS status again. Please note that I personally have only done this work for a couple of systems, and it resolved the issue (for the time being) on those boxes. This is NOT a permanent fix and could well break again if Microsoft issues another update that includes doulbe-quotes in the update title.

More information about the behind-the-scenes reasons for these issues were found in a thread at forums.techarena.in and another at SANS. At the time of this edit, there has been no info posted at the WSUS team blog.

[Edited at 3:40pm CST]
I have been informed that the faulty information that was pushed into WSUS yesterday has been updated and *should* automatically get pulled in during the next scheduled WSUS update. WSUS 3.0 can do a manual sync to get the update now, and WSUS 2.0 should get it at 10pm local time tonight. A post is expected from the Official SBS Blog later today. I plan to leave one of my servers in this state to confirm that this operation works as expected tonight, but will manually run the osql steps tomorrow if it does not.

On Leopard, Part 2

eriq — Sat, 27 Oct 2007 11:05:00 GMT

Strike one for Apple. Late Friday afternoon I knew that my copy of Leopard wasn't going to arrive on the 26th as stated on the sales order, so I decided it was time to try and get as much detail as I could. When I had spoken with a sales rep fromt eh Apple Store online earlier in the day, they didn't know what the status on the order was, just that it hadn't been prepared for processing yet. Since the order page still had not been updated, I decided to call in again. The Agent I spoke with this time was clearly a little more flustered than the Agent I spoke with in the morning, and after talking with her for a few minutes, it became clear why.

I was definitely NOT the only person calling about the status of the order.

Unfortunatly, this second Agent had no better information to give other than "it hasn't shipped and I cannot tell you why, but we're working as fast as we can to rectify the problem and get the product shipped." Not being satisfied with this response, I asked who I could speak with to try to get more information. The Agent transferred me to Corporate Customer Relations and gave me the number. After about 10 minutes on hold, I finally spoke with the CR Agent and took a little different tactic with my concern. Honestly, I wasn't terribly put out that I wouldn't get my copy of Leopard on the 26th as advertised, but I was a little more than frustrated that the party line was "we don't know what's going on." I explained to the CR Agent that if I had been able to get some concrete information about the order the first time I called, I could have A) altered my schedule so I wasn't sitting at the office all day waiting on a package that was not coming, and B) made arrangements to head down to an Apple Store locally to be ready for the Launch Party that started at 6pm and get my copy of Leopard there. But knowing at 4pm that the product wasn't going to even ship by the 26th didn't help at all. All in all, I waited aout 30 minutes on hold with the CR Agent as he dug around trying to get any information that he could. That's when I knew for sure that it wasn't a party line I was being fed. No one had any clue what was going on. Finally, the CR Agent came back and said that he was able to find someone who would be able to get me some additional information, and ironically it was someone within the Apple Busines Agent program. So I hung up with the CR Agent and called my ABA contact who, I was told, would be able to provide additional information.

She did.

She was able to tell me definitively that my order, along with a number of others, was not going to ship in time to arrive on the 26th or even ship on the 26th. But that was it. Nothing like "we didn't have enough product to cover the number of orders that came in through the online store" or "we put all the media in the retail outlets and online orders will get shipped when additional stock arrives." In fact, other than confirming that my order wasn't going to ship on the 26th, she was as in the dark as anyone else. I reiterated my main concern, which was that the lack of information kept me from being able to make necessary business decisions (like passing on a couple of local support jobs because I couldn't leave the office in case the package had actually been shipped), and that if Apple is going to make claims about product being delivered on the day of the launch to users who placed orders on the site, that they really better be able to stand behind those claims. She got the message.

Now I have to say that ay no point in any of these calls did I ever lose my temper or even raise my voice. I've worked on the other end of support lines for too long to "lose it" with any phone agent. My years in support management have taught me that receiving negative feedback in a constructive manner is just as important as receiving positive feedback, and since most of the time people who are unhappy with a service provider will tell everyone they know EXCEPT the service provider, when I have an opportunity to share negative feedback, I not only do so, but I try to get to someone at the management level and deliver the info in the same way I wanted to receive it myself.

When I did finally get home last night (I did work a few remote calls before leaving the office), I checked my order status at the Apple Store again. Still says "Delivers on October 26th." This morning I got up and, out of curiosity, checked the order status again. And it's been updated! Now instead of a status of "Not Shipped" and "Delivers on October 26th" I have a status of "Prepared for shipment" and "Ships by October 29 - October 30." Now that's useful information. That tells me that if I wait for the online order to process, I won't get Leopard until at least mid-week. So now I have the opportunity to try to find a copy at the local Fry's or the Apple Store that's closest to the house. CompUSA is already sold out of their stock, and they were offering a $30 mail-in rebate. Fry's is closer, and opens earlier, so I'll try there first, and head to the Apple Store as the last resort. So there might be updated Mac Connectivity docs this weekend after all. then again, this might be a sign that I need to take the weekend off. We'll see...

On Customer Service

eriq — Mon, 23 Apr 2007 19:52:00 GMT

What the hell has happend to the notion of customer service in today's business environment? In the 20+ years I've been in the IT industry, I've always done my best to ensure that my clients are pleased with the service they've received from myself or my team, becuase I've occasionally received poor treatment from other agencies, and I've never wanted to have anything like that kind of reputation. The more I deal with larger companies, though, the more I think that the focus on customer service has simply been done away with.

Today's example comes from a worldwide shipping organization. I recently had a package shipped in from out of the country, and today I received a notice in the mail that I had an unpaid invoice to that shipper. I called to inquire about the unpaid invoice (since I'd never received an invoice and had no clue what I could have had shipped that I didn't already pay for) and the first agent I deal with faxed the invoice over to me. Looking it over, it still didn't make any sense. So I called back a second time to get clarification.

Boy, was that a mistake. After the phone agent explained what the charge was for, I expressed concern that there was nothing in the packaging that indicated that I'd be accepting any additional shipping charges by accepting delivery of the package, and that I had not received any invoice or communication of any kind about additional charges. Before I could say that I'd be sending a payment off for the charges, the floodgates opened. The phone agent immediately went into yelling mode and accused me of attacking the company. She said that they expect people to know that there are customs charges for packages coming in from out of the country. When I cut her off to explain that I wasn't trying to "attack" anyone but was trying to get clarification on the unexpected charges, she cut me off and started yelling louder that they normally waive these charges for someone that doesn't understand about customs charges and that my account was clear. Then she repeated that she cleared the charges because it wasn't cost effective to have these conversations, gave her name, and hung up.

If you've been reading my recent posts on the fun with Dell, you know that I generally try to give a company an option to improve it's product when I see an issue, because that's what I'd want to do. So you might expect that, since I have the phone agent's name, I'd call the shipper, speak to a manager, and discuss the phone call in more detail, giving the manger an opportunity to discuss handling customers with the phone agent, and why yelling at customers is generall not a good idea.

Not this time. This time, I'm speaking with my dollars. I'm no longer using that shipper for sending packages (not that I do a lot of this, but they've lost all future outbound shipments from me). I'm also going to request any vendor I deal with to use a different shipping option, including the US Postal service where applicable, instead of this particular shipping company, because I don't want them to make any more money on shipments that have anything to do with me or my company. I recognize that I can't completely circumvent this shipping company in all likelihood, but I'm going to do everything I can. Because I don't want to get into a discussion of how much billable time I lost tracking down this particular shipment because they didn't attempt redelivery during the window they indicated, and I had to travel to their holding facility to pick it up myself. I'm not going to give them example after example after example of how they could make simple changes to improve their customer satisfaction on their product.

All because the agent lost her cool and started yelling at me.

You yell at me, you lose my business. I'd expect the same from anyone who got yelled at by me or one of my employees. Fire me if one of us yells at you. Because that's just unprofessional. And if you can't treat me professionally, much less as a human being, you're not getting my business.

And even if I had "attacked" the company (whatever that means), that's still not an excuse to lose control and shout at a customer. That's where I draw the line, and that's why I will never voluntarily use the services of this shipping company again.

On Feedback

eriq — Tue, 10 Apr 2007 16:05:00 GMT

In previous posts, I’ve lamented about the issues I’ve had with the PERC 5/i controller in Dell servers that I have rolled out to clients and put in my own office. Today, I had a conference call with Dell support and someone from the NOS group about the design of the card. The bottom line at the end of the call is that right now, Dell has no solution for being able to provide any automated notification for an array problem or failure, either through software or hardware. They have heard my feedback and are escalating the issue through the engineering group, but they have no answer for me at this time.

I don’t have a problem with this. In the three weeks I’ve been working with Dell on this problem, they’ve been very responsive and proactive about communicating with me when they have updates on the issue. Personally, I think that the argument I’ve presented is solid enough, and while everyone I have spoken with agrees with me, I’m just not sure that it’s a case of one person really making a difference.

So, at the recommendation of the Dell folks, I have posted the issue to their feedback site, Ideastorm. This is a site Dell has set up to accept direct feedback from their customers, and the engineering group, among others, takes the feedback given on the site seriously.

Well, as seriously as they can. Honestly, after looking through a few posts, there are some garbage posts up there, just like you’d expect from any unmoderated public-facing site that asks for thoughts and opinions. Still, they asked, and I complied.

This is where I want to get assistance from the community, and this request is two-fold. I have spoken with several VARs in the small business space who were totally unaware that there was not audible alarm on the PERC 5/i controller, and most were really quite surprised. So, first, if you interact with other VARs who you know sell or support Dell equipment, please let them know that the PERC 5/i has no audible alarm on it. They may not care, but they should at least know. Second, if you want to see this design changed back to having an audible alarm, or at the very least some automated notification tools running on the server directly (no, IT Assistant will not run on an SBS server, nor will it run on a 64-bit OS), check out my post at Ideastorm and comment or vote on it. If engineering sees a large number of VARs are not happy with the state of the hardware, there will likely be more impetus to take action quickly. The squeaky wheel getting the grease and all that. You will have to register on the site to make a comment and vote, but if you are concerned about this situation, please take the time to do so. Alternately, contact your Dell support or account rep and discuss the situation with them.

On SP2 and Avoidance

eriq — Sun, 01 Apr 2007 13:52:00 GMT

For what it's worth, I now have an official stance on Windows Server 2003 SP2:

Don't install it. At least not yet.

I keep running into folks who have issues with their servers following the installation of SP2. the majority of issues seem to be cause by the enabling of Receive Side Scaling in SP2. This is more than just my experience, too. The Official SBS Blog also notes that RSS may be at the core of a number of problems seen in the SBS community following the installation of SP2. MS has posted KB927695 which gives a registry hack to disable RSS on the server, but the description of the problem in the KB doesn't correlate to the behaviors we're seeing in the field, so the average bloke trying to figure out why he suddenly can't get to external FTP sites or the iTunes Music Store isn't going to find this KB directly.

Though the KB tells the reader to disable RSS in the registry, I've been able to get resolution in all cases I've handled thus far by going into the Configuration properties of the NICs on the server and setting Recive Side Scaling to Disabled. This is a far more palatable resolution for most people than going in and modifying the registry.

Another common thread we've seen in blogs and newsgroup posts is that ISA is also a factor in the mix. My recent experiences have not had ISA involved, which is why I'm looking at RSS as the core culprit. It's the common thread that leads to the core resolution - disabling RSS seems to take care of a vast majority of issues.

Plus, I have yet to see a real business case for pushing ahead with the installation of SP2, especially in the small business arena. In fact, of the Top Ten Reasons to install SP2 listed at the TechNet site, I see exactly one reason, ironically listed as #1 in the list, that I might want to install SP2, but that's only for sites that I haven't been keeping up to date with Microsoft Updates - SP2 does offer a single rollup for all the updates that have been released since SP1, and if a bunch haven't been installed, then yeah, it could be easier to install SP2 than to singly install all the updates. But given that reason #3 is causing a lot of grief out in the community and that the other "reason" just really seem more like marketing spin than actual technical/business benefits, I cannot in good conscience install this product on my client's servers, nor can I recommend that anyone take the plunge right now and see what's happening. That stance will change once we get a better grip on the breadth of the problems that are being caused by SP2 and quick access to the resolutions for them, but I'm nowehere near that point yet.

One request from you, the gentle reader. If you have encountered a problem followin the install of SP2 that was resolved by disabling Receive Side Scaling, please take a moment and either drop me a note with a brief description of what the problem was, or just leave a comment here on the blog. I'd really like to get an idea of what's out there that's really getting hit, because I don't think all of the possible scenarios have bubbled up to the top just yet.

On Dell's PERC 5/i

eriq — Fri, 30 Mar 2007 21:37:00 GMT

So in a previous post, I railed about my surprise at finding out that the Dell PERC 5/i controller has no audible alarm and why that's a concern for me. Well, here we are, nearly two weeks later, and after much going round and round with Dell on the issue, I have more information, and it's not necessarily good.

My specific initial issue is that, besides not having an audible alarm, Dell's Server Management software (Open Manage) doesn't have a way to send notifications about problems with the RAID controller, either the controller itself or the failure of an element attached to the controller. After my initial support call with Dell about the issue, they indicated that the IT Assistant software should run on the Windows 64-bit box, and that will send notifications when an issue is detected. I've since found out that no, the latest version of IT Assistant that's posted on the Dell web site will NOT, in fact, run on the 64-bit Windows platform. Of course, everything about IT Assistant tells you that it really, truly, should be run on another box, but for this specific instance, that's not going to be possible.

In digging further into this, however, I've uncovered a couple of other issues that concern me. Given that IT Assistant seems to be the preferred way to actively monitor the RAID, I thought I'd try to install it on the SBS box I have at the office. No dice. IT Assistant will install onto a server platform, but not SBS. It's a hard block. So for all my SBS servers with PERC 5/i cards, I have nothing from Dell that I can run on the server box that will monitor the helath of the RAID controller and notify me when there's a problem. With my other RAID controllers, I can fall back on the audio alert at the very least for notification of a problem, but don't have that option here.

Last Thursday, at my SBS user group meeting, I mentioned my frustrations with the situation in a side conversation, and one of the folks I was talking with was as taken aback as I was when I first figured this out. He just put a number of servers with PERC 5/i cards in them out in production and was also unaware that the controller had no audible alarm mechanism. I've mentioned this to a couple of other folks as well, with pretty much the same reaction.

This week, I started putting together specs for a couple of new servers for a couple of projects and, knowing the challenges of the PERC 5/i, decided to look at other controller options for these boxes. Unfortunately, I've found that, currently, the only RAID controller that Dell provides that supports RAID 5 is, surprise surprise, the PERC 5/i. There are other controllers, but those only support RAID 0 or 1. And I don't yet have confirmation if those controllers have audible alarms on them or not. So, even if I was able to "settle for" a RAID 1 solution (and to be fair, on one box it's not unrealistic), I still think I'd be in the same situation.

I've been working with a couple of folks at Dell on trying to find a reasonable resolution to this problem. Of course, there's always the recommendation that I can run IT Assistant on a separate workstation to monitor the array card in the server and send notifications back to me if/when there's a problem. But that's not necessarily a realistic solution at some sites. Now I have to install a piece of software on a workstation that has to be running all the time and may or may not interfere with what the user of the workstation is trying to do. I simply can't afford to stick a dedicated box at each of my client sites to do this monitoring, nor can I ask them to dedicate a workstation to do this themselves. It looks like I'm going to have to go third party for a solution, and while that's probably less costly than doing a dedicated workstation to monitor the array, it's still an added expense that I really don't think I should have to incur in order to be proactive with my clients.

I honestly believe the folks I've spoken with at Dell understand my plight. While they have not committed to anything, there have been discussions about changes to engineering on future controllers to ensure an audible alarm among other possibilities. Based on a series of messages that floated around this afternoon, I know the issue has been escalated internally, but still have no clear direction on where to go.

At the end of the day, two weeks after I first placed the call regarding the failed array and lack of notification of the failure, I still have a box that I will have to manually monitor for RAID health. I'm hoping for a better solution, and I expect that I'll just have to be patient.

I sure hope that data cable doesn't pop off the drive connector again, tho...

It doesn't what??

eriq — Sat, 17 Mar 2007 22:52:00 GMT

So I got back into the office following the MVP summit and need to do some work on one of my test boxes, and I'm running into a little trouble with a piece of software complaining about the CPU not being 64-bit (this is on a server acquired a month ago specifically for 64-bit software testing) and it suggested that I check some settings in the BIOS. OK, no problem. Reboot the server, and just happen to look at the RAID controller firmware during the BIOS boot and thought it was taking a little longer than normal to go past. Sure enough, when the firmware finishes initializing, it reports that the main VD is degraded. Yikes! I popped into the controller BIOS and it says the drive is missing. *sigh* I hear a call to Dell coming in the near future.

Before calling, I pop the case to see that everything looks OK (this is done with power OFF, by the way, and with a good grounding strap), and I noticed that the cable connecting drive 0 to the controller has come off the connector. Not just worked a little lose, mind you, but completely and totally off. No wonder the array was degraded and the controller reported a drive missing. I reconnected the cable (which doesn't fit exactly the way I want, and I'll mention that in my call to Dell, because I can kinda see why the connector popped off in the first place), power on the server, and get back into the controller BIOS. Drive shows, but it's showing Foreign (again, not a huge surprise), so I make one of the other unallocated drives in the unit (long story, don't ask) a global hotspare, and the array starts rebuilding immediately.

When I got the Dell support rep on the phone, which didn't take very long, his initial thought was that the drive was bad and he was ready to courier it to me to meet the 4-hour on-site warranty that I got with the box, but after I explained the situation, we went through and got the drive back to a working state, and it seems to be in good shape, save for the concerns I have about the cable pressure. But I asked him about my real concern, which was that the controller alarm didn't sound when the array failed. He reviewed the docs on the controller, and dropped this little tidbit of info on me:

The PERC5/i controller DOES NOT SUPPORT AN ALARM.

Excuse me? The PCI controller (which has exactly the same firmware, apparently) supports an audible alarm for when there's a hardware problem on the controller, but the on-board one does not? Could someone try to explain this to me in a way that makes ANY sense at all whatsoever?

My Dell reps are going to get a bit of an earful come Monday. Besides the lack of alarm support on the controller and the cable pressure, it's clear that the firmware for the on-board controller doesn't support a number of functions that the PCI version does, including, but not limited to, flashing the LED on the drives.

I know Dell has their ups and downs, and I've defended them most of the time that people insist on deriding the hardware or support lines, but this is just plain lunacy. And it wasn't anything that was obvious - the support rep had to dig through a number of docs to finally determine that the on-board controller has no alarm support.

So, note to self. Either Dell gets this fixed, or I stick with PCI RAID controllers from now on. That's like buying a car where none of the dashboard alert lights are present and you're supposed to guess when you have an oil pressure problem, or are close to running out of gas, etc., etc., etc.

Good grief...

On Updates

eriq — Wed, 14 Mar 2007 06:34:00 GMT

Well, turns out it's not just me that can be a little off when it comes to updating things. Today, (well, yesterday based on the time this actually gets posted) Microsoft released Service Pack 2 for Windows Server 2003. Only there are a couple of issues with the release. Still waiting on additional information, but here's what can be found out at the moment:

1. W2K3SP2 is being offered through WindowsUpdate/MicrosoftUpdate. I had it in my head somewhere that Service Packs weren't offered through Windows Update, but apparently I got my wires crossed on that one.
2. There are issues if you need to upgrade a server TO Windows SBS 2003 SP1. As documented in KB932600, you have to uninstall SP2 if you need to upgrade a Windows 2003 server to SBS 2003, or upgrade an eval copy of SBS 2003 to production SBS 2003. Sorta makes sense, as when you run the SBS 2003 installer, it's going to put SP1 on the server and will have issues if SP2 is already there, but it's interesting nonetheless.
3. You have to uninstall IE7 from your Windows Server 2003 box before you can install SP2. This is according to a Technet article. I'm not sure what the story is on this, but I have several servers I'll need to uninstall IE7 from before I'm able to install SP2 apparently. Joy.
4. It really is going to get pushed out through Automatic Updates, so if you don't want to have it go on automatically, you can get a tool that will block deployment of SP2.

More to come, I'm sure...

UPDATE 3/14/07 - Seems some folks out there have been able to install SP2 without removing IE7. Still waiting on final word on this, but there is at least anecdotal information that #3 above may not actually be an installation stop point.

UPDATE 3/15/07 - The issue with needing to uninstall IE7 prior to installing SP2 is an error in the release notes. The document should be updated shortly, but there is no need to uninstall IE7 prior to installing SP2. Also, the SP will not be pushed down through Automatic Updates until June 2007. It does show up as a Critical Update when you manually go to the Microsoft Update web site and check for updates, but Automatic Updates is not offering it until later. It does appear in WSUS as an approvable installation. The Official SBS Blog also has a post about installing SP2 on SBS 2003.

On Planning

eriq — Wed, 17 Jan 2007 15:13:00 GMT

What's the exact phrase? "The best laid plans of mice and men oft go awry"? Yeah, that pretty much sums it up. Sometimes Mother Nature just gets in the way. Take this past week, for instance. Tuesday, I started getting the head cold that's been going around on. Wednesday, I end up all day on a server migration that should have taken about 6 hours (sound familiar?) Thursday, i take a very rare day off and visit my doctor, to confirm that I have a sinus infection, but am also told that I've got the beginnings of walking pneumonia. Friday I have an all-day activity that ends up being walking tours of several areas, and I have to bail out early in the afternoon because I just don't have the energy to keep up. Then, with the ice covering the Dallas/Ft. Worth metroplex area al weekend (or at least the threat that finally comes through late Sunday), I opt to stay in bed all weekend to try and recover.

All that to say that even though the regularly-scheduled broadcast of eOnCall ran on time last Thursday, I didn't get the podcast version posted to the web site. And i've yet to get in to record the show for this coming week, so teh broadcast version may well be a repeat. Since we got iced in again today in Texas, perhaps I'll be able to get together with the crew at Tucker Communications and get a show put together later this afternoon.

In any case, the show from last week is finally up, and there will be some show broadcast on Thursday, but we'll have to see if it's new content or a repeat.

On Misteaks

eriq — Sat, 02 Sep 2006 18:50:00 GMT

It's just one of those things, I guess, but when it causes so much pain, it's hard to see it as a little thing. I mean, come one, everyone makes little mistakes occasionally, don't they? I do, I own up to it when I do, also. And I've made some doozies in my day, too. But the mistakes I've made, including the big ones, have only impacted a small number of people.

But occasionally, there's the little slip up with the big impact. Yesterday saw another one of those, and while I was tangentially impacted, it really didn't hit me personally very hard. But those it did hit, it hit hard. You see, because of a mix up in a virus scanning search string, the product from a certain anti-virus vendor started identifying a core Windows system file as infected and quarantined the file. Oops.

Now don't get me wrong, I admire anti-virus vendors and the work that they do. They often times have a thankless job. It's amazing that they are able to create search strings for virus program patterns that will really only find the virus code in a file and not (usually) mis-identify a regular, uninfected program as a problem. Given all the different programs and files that can be present on any given computer at any given time, it's amazing that there aren't more false positives than there are.

Still, this is one that really should have been caught before it got out the door. I would think that if you are testing updates to a program, one of the basic things you would check is to make sure that it didn't break the OS it was running on. I have not done much software development recently, but back in the day, I wrote code for a Linux-based system, and we used a few scripts to configure certain file areas on the system. One of the first things I tested was to make sure the file system changes we made wouldn't render the system unbootable. Seems fairly basic to me. And maybe this company did do some level of testing, but somehow it still strikes me as odd that this wouldn't have been found in a thorough testing environment.

And this is not the only company that has released software that rendered systems unbootable. At least one other anti-virus vendor released a virus engine update that wreaked havoc. An uninterruptible power supply vendor had a program that, when not updated in a timely fashion, would keep a system from booting normally. And a certain operating system supplier has been guilty of releasing updates to its own code that had drastic unexpected results. It happens. Really, it does, and fortunately it doesn't happen more often than we see it happen.

Let me wrap up by saying that I'm really glad that this particular situation didn't happen close to Patch Tuesday. I can only imagine what would have happened if people went to install updates, rebooted, and found that their systems didn't come back up. The Microsoft phone lines would have been ringing off the hook, and a lot of people would have been chasing down the wrong road. This is actually a perfect example of why I always restart a server before installing security updates. If there's a problem with the server coming up before installing the updates, there very likely would have been problems with it coming up after the updates installed. And where would I have focused my ire, I mean, troubleshooting efforts? Right, at Microsoft. It would be easy to blame the patches for causing the no boot situation. If this had happened a couple of weeks ago, MS would have been crucified by thousands of callers blaming the security updates, and lots of time and energy would have been wasted looking in the wrong place for the cause.

Just goes to show that you should take any server reboot seriously. I always make sure that, before I reboot any server (mine or a client's) that I have the time to go onsite if a problem should present itself. It's very easy to become lax towards server reboots because, most of the time, they work exactly as they should. But in the case of some people yesterday who rebooted with their guard down, they had a bit of a rude awakening. Hopefully, the worst is over for now for this particular incident. Who knows where the next one will come from...