Cluebat-man to the rescue : SysAdmin

Could not create NTDS settings on domain controller...

vanDooren — Tue, 14 Apr 2009 16:24:00 GMT

Could not create NTDS settings yadayadayada on domain controller CN=yadayadayada. The RPC server is unavailable.

This was the error message greeting me (with some meaningful text instead of the yadayadayada of course :)) when I tried to add a backup domain controller to the domain of our test and development network. It was the first time I encountered an error at the DC promotion stage.

I checked the usual things (network connections, privileges, etc) but nothing jumped out at me. Now, if an error dialog mentions RPC, then the usual error is either a DNS error, or a DNS server that has not yet refreshed its zone information.

So I opened the DNS config, and discovered that only 1 NIC of our the new DC was registered in DNS. Our networks are multihomed (3 networks in parallel) and yet only 1 address was registered for the NIC. This was a bit odd. There should be 2 (one network has no DNS on purpose).

A quick ping revealed that one of the NICs had no connectivity. It turned out that the primary DC was a bad network connection, due to some wiggling with the cables. Plugging it all the way and then re-registereing the NICs of the new DC with DNS solved the problem.

I still don't understand why we got that problem, because the disconnected NIC was last in the binding order, so the new DC should have used the available connection anyway. My guess is that it retrieved the address via DNS, and got the address of the NIC that was disconnected. Ah well. Live and learn.

The kerberos client received a KRB_AP_ERR_MODIFIED error

vanDooren — Thu, 02 Apr 2009 10:43:00 GMT

This is what I got in the event logs yesterday afternoon:

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 4
Computer: SE-SMURF01
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the server PC-BLABLA09$. The target name used was . This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (FOO.BAR.STRIPE.LOCAL), and the client realm. Please contact your system administrator.

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 4
Computer: SE-SMURF01
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the server PC-BLA09$. The target name used was RPCSS/PC-BLA10. This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (FOO.BAR.STRIPE.LOCAL), and the client realm. Please contact your system administrator.

I had replaced those machines a week ago, and everything seemed to work fine. So I didn't understand why these errors were suddenly popping up. The applications running on those computers where throwing a wobbler as well. Some googling later I found 2 remarks that were useful.

The first one was that someone fixed it by taking the computer out of the domain, renaming it, changing the SID, and changing the IP address. While this is overkill on the scale of killing a mouse with a thermonuclear weapon, it pointed in the direction of a network level problem.

The second remark was by a Microsoft employee who explained that DNS misconfiguration can be the source of problems like this. If kerberos thinks it is communicating with pcA it encrypts the kerb ticket with the password of pcA. but if the ticket then ends up on pcB because of the DNS mismatch, the above events will be logged.

At that moment I realized that I had changed the IP address of an adapter on PC-BLA10 because it conflicted with PC-BLA09. The reason everything worked fine initially was because that port had been left disconnected until 2 days ago when I configured the correct IP address. The conflict was resolved and the DNS information was updated, but that didn't mean that the DNS caches were up to date. So I cleared the DNS cache of the DNS server, and used ipconfig /flushdns to clear the resolver cache on the domain controller and PC-BLA10, and the problem disappeared.

Scheduled task would not run

vanDooren — Wed, 25 Feb 2009 08:13:00 GMT

The scheduled task for running ntbackup on our fileserver would not run anymore. It would stop with a return code of 0x80, and nothing worthwhile in the log file or event log. It ran fine for months, apart from the occasional hickup such as having no more free tapes in the loader, or a tape not being imported correctly.

I tried running the backup batch file manually, and that worked fine. I ran it under the dedicated backup account, and that worked fine too. But it wouldn't run automatically, and starting the task manually when logged in didn't work either. I couldn't find any useful debugging info so I turned to google. I found several people who has similar problems, but most had to do with a bug in the HID service which somehow interfered with ntbackup. I looked around some more and finally I found something that ended up giving me the final clue.

I opened up the task manager and noticed that there were 21 instances of xcopy still running, 23 instances of psexec, and 3 instances of ntbackup. These were probably left over from some tests I had done with the backup script of one of the new servers (a backup domain controller) which uses the 3 aforementioned programs.

After killing these dead processes I tried to run the backup task again, and everything worked fine this time. In my naivety, I had assumed that ending a task would end the processes it had spawned, but that seems to have been a mistake. Ah well, everything is working again, and now that I know the cause, I can prevent this from happening again.

This is why I try to understand a problem, rather than just rebooting the machine. A reboot would have fixed the problem in less time then I needed to figure this out. But then I wouldn't have known what happened (I hate that) and I wouldn't be able to prevent this from happening again. I would have been stuck in a regular but unknown problem -> reboot cycle.

More Terminal Server License Server weirdness

vanDooren — Tue, 06 Jan 2009 06:40:00 GMT

Last week I had a very peculiar problem with the Terminal Server licensing.

I couldn't connect to the licensing server anymore, even though it was running. The Terminal Server Licensing app couldn't detect it anymore. The only thing that still worked was a network ping, but as far as Windows was concerned, we were running unlicensed.

The weird thing was that Terminal Server itself disagreed, and kept working without complaints. One of these servers had been running for a year already, so it was not the 120 day grace period that was hiding an underlying problem.

At the time I didn't really understand why it still worked, but my colleague pointed out the the system time was wrong. The licensing server is a virtual machine, and for some reason, the time of the virtual machine was not linked to the NTP controlled time of the host. And as we all know, if the system times are off by n minutes, the mutual authentication fails and no secure connection can be established.

As it turns out, Terminal Server Licensing is ever more stupid than I already knew. Not only doesn't it count licenses, but the Terminal Servers don't even connect to the license server to ask for licensing. Because if they did, then they would have complained about it since connecting was no longer possible.

Backing up event logs in their normal binary format

vanDooren — Fri, 19 Dec 2008 13:15:00 GMT

Some time ago I was looking for a way to make backups of the eventlogs of our server, preferably without me ever having to do anything anymore J

This was not success. There are a couple of tools to make text exports from eventlogs. These have the disadvantage that they make the logs very large and unsuitable for looking at with the even viewer. As an added bonus, these are slow too (since every item has to be read individually).

EventQuery.vbs is one of these tools and it is so dog slow that it is almost a crime that it has been included with Windows. Ever wanted to bring a domain controller slow to a crawl? Simple: Use EventQuery.vbs to export the security event log. That’ll do it.

Anyway, I checked with my fellow MVPs, and there is no built-in (or free) tool to save event logs as binary event log files, so I gave up.

But then I accidentally discovered that there is a win32 API that exposes the event log functionality. And as luck would have it, I am a programmer and thus perfectly suited to make such a thing myself. It would have been trivial to make it at work, but I decided to do this at home so that I could share the code.

It is of course perfectly possible that such a free tool exists and I simply didn't find it.

The code

The code is simple, though the line count was increased by the desire to implement error handling.

int _tmain(int argc, _TCHAR* argv[])
{
_TCHAR* computerName = NULL;
_TCHAR* logName = NULL;
_TCHAR* file = NULL;

//populate the variables, based on the input arguments
if(argc == 3)
{
    computerName = NULL;
    logName = argv[1];
    file = argv[2];
}
else if(argc == 4)
{
    computerName = argv[1];
    logName = argv[2];
    file = argv[3];
}
else
{
    Usage();
    return -1;
}

//Open the eventlog
HANDLE hEventLog = OpenEventLog(computerName, logName);
if(NULL == hEventLog)
{
    _tprintf_s(TEXT("Could not open eventlog %s on computer %s\n"),
      logName, computerName);
    PrintError();
    return -1;
}

//backup the event log to file
if(FALSE == BackupEventLog(hEventLog, file))
{
    _tprintf_s(TEXT("Could not save eventlog to file %s\n"), file);
    PrintError();
    CloseEventLog(hEventLog);
    return -1;
}

CloseEventLog(hEventLog);
return 0;
}

There is little to tell about the code. It can be invoked with either 2 or 3 command line arguments:

The name of the computer of which the logs have to be backed up. (optional)
The name of the event lot to back up
The path of the file to which the event log needs to be saved.

The application opens the event log, and then makes the backup and closes the eventlog. If something goes wrong, the error code and error message are printed to the command line.

I thought about making the error handing a little more elegant (so that PrintError and CloseEventlog were not used multiple times in the code) but with only 2 function calls, the amount of error handling code would not diminish or be any better.

I left the PrintError function out of this blog because it doesn’t do anything exciting. It retrieves the error code and prints out the corresponding error message.

What you need to know before using this tool

I’ve been using this tool for a week now, and it is working splendidly, but there are a couple of things that aren’t mentioned in the MSDN documentation of the API.

The first is that the supplied path is absolute for the computer of which you are making an even log backup. So if the path is D:\app.evt, it will make the backup to D:\app.evt on the remote computer.

The second thing you need to know is that providing a UNC name for the target file doesn’t really improve things all that much. Because what is happening under the hood is that it is not you who is making the backup.

The event log service is in charge of the event log files, and you are asking it to make a backup. And that is where it all goes pear shaped. The event log service runs with LOCAL_SYSTEM credentials. This means that by default, it has no credentials that are valid on a remote computer. So you cannot make backup files in network locations, because the event log service of the remote computer has no privileges to access the location.

There are 2 exceptions to this rule. 1) the remote location is on the remote computer itself. 2) the remote computer is a backup domain controller.

It is a little known fact (I didn’t know it either) that LOCAL_SYSTEM on a domain controller is a domain administrative account that can pretty much do anything and everything on any domain computer, including the things that even a domain admin cannot.

Conclusion

I needed a convenient tool for making backups of event log files, and now I have one. Both the source and the binary are included as attachments. The source is available under the MIT license, and the exes are available for free to do with them what you want, as long as it is understood that whatever you do with them is your responsibility, not mine.

The eventtool_clr exe is something I made to create and delete event logs. I don’t think I blogged about it yet, but what it does is so simple it’s hardly worth mentioning. Have a look at the sources for more info.

The binary is for those who just want to use it and don’t care. The source is for those who’d rather not run an untrusted binary from some guy on the internet on their internal servers J The sources are simple enough to audit and compile.

On a side note, The API to backup event logs is not exposed on the .NET api for event logs. I don’t know exactly why, but I suspect that it is because of the problems I mentioned. The API is confusing enough that it is really not fit for generic use. After all, you can’t backup to remote systems, and the path is absolute for the remote computer.

I also didn’t want a .NET program, because for this type of thing, nothing beats the convenience and performance of a compiled binary that needs no runtime DLLs or external frameworks.

Passed 70-290 today

vanDooren — Fri, 14 Nov 2008 21:48:00 GMT

I have been silent on my blog for the last couple of weeks. This is because I spent all my time preparing for my first Microsoft exam in a couple of years. But it has been worth it. Today I passed the Microsoft certification exam 70-290: Managing and Maintaining a Windows Server 2003 Environment.

As I am really a software developer, cunningly disguished as a sysadmin, I doubted I would make it at the first attempt. But I took a lot of time to prepare, and tested all the topics in a test environment.

Long story short: I aced the exam with a perfect 100% score :)
The lady in the test centre told me that I was only the second to ever ace a Microsoft exam like that in that test centre, and the first outsider. The other guy
worked there as a trainer.

What makes this particular exam score even sweeter is that
a) I only used the MSPress book (no exam cram crap)
b) I pulverized the record high scores of the guys in the ICT department. :D

Definitely a nice way to start the weekend.

Installing Zune software without internet connection

vanDooren — Fri, 24 Oct 2008 05:42:00 GMT

Last week I bought myself a zune, and since my main computer during the day is my company laptop, I wanted to install it on that one. Unfortunately, going to zune.net didn;t help because the first thing the installer does is to check if Windows Update is enabled. And if it isn't, it aborts. Of course, if you don;t have an internet connection at all, it aborts as well.

My laptop is ruled by company policies, and security is not a laughing matter here. Regardless of what my opinions are, I am not going to tamper with that, since that would be very much a 'Bad Thing (tm)'.

After a lof of googling I discovered that a lot of people have this problem, and the folks at Zune.net seem to think that this situation needs no special attention. Luckily I found the solution on a zune forum. There are lot of alternate approches, but they require some degree of hacking with settings, which I do not want to do.

Download the client package which contains all the files, and not the setup shim you can find on Zune.net
Extract the files
Navigate to the packages folder in x86 folder.
Install wmfdist11-windowsxp-x86-enu.exe
Install windowsxp-kb915865-v11-x86-enu.exe
Install zune-x86.msi
Restart the computer
Connect your Zune and wait for the driver isntallation and device detection

For x64 the setup will be similar, but you won't need the kb patch. It shouldn't be too hard to figure out which files to install if you have an x64 system.

I have to admit I am a bit disappointed by Microsoft and the Zune folks for not explaining this in the FAQ on Zune.net or somewhere else where you can easily find it.

I am not the only one with a corporate laptop and a zune, and many people seem to have had this problem, judging by the number of hits I got.

It seems the Zune decision makers want to force people to be online.

Windows Instant On

vanDooren — Fri, 17 Oct 2008 09:54:00 GMT

Something I picked up on slashdot today: Microsoft is checking if people would want an 'instant on' version of Windows.

As compelling as it sounds, I don't think it is that big a deal, and they shouldn't waste their time with it. These days, evey computer and laptop supports Standby or Hibernation. Between the 2 of those, there is no reason why Microsoft should invent an 'Instant On' option that is limited in what you can do, if it is perfectly possible to resume from hibernation or standby in the same amount of time and have a fully functional system at your fingertips.

But let's for the moment assume that my computer support neither of those options.

My home laptop is an old P3 1GHz with 700 MB RAM. It is not part of a domain , and is fully usable 15 - 20 seconds after I touch the power button. Instant enough for me. My workstations are the same, only it takes 20 seconds or something like that for the system to POST. So even the instant on feature would not save me from having to wait.

Then there are my machines at work. They are part of an enterprise domain. Booting windows takes a relatively short time. It's only when I log in that the wait begins. The delay before I can actually use my laptop is long enough that I can go to the coffee machine, get hot water and brew my own coffee by manually pouring hot water over a drip filter with hand ground beans. By the time my cup of coffee is full, it can happen that I can access the start menu, though that is not a given. Usually it takes another 5 - 10 minutes before the system has finished doing whatever it needs to do.

The reason of course is that -as in a typical enterprise- there are so many group policies which are refreshed. Then there is the virus scanner that is starting its scan, remote management software (sms) that is started, system checks that are performed, services that are started, ...

So Instant On wouldn't help me much there, because the amount of stuff that is going on would be the same. Of course they could prevent this prom happening, but I am pretty sure that no domain admin wants to allow a computer on the network if it hasn't jumped through all the hoops to make sure that it is compliant with all the policies and limitations that are required by the corporate policies.

It would probably best if they ditch Instant On right now, and start focusing on Windows 7, making it robust and responsive.

Installing a service pack: how much space do you need.

vanDooren — Tue, 07 Oct 2008 09:57:00 GMT

Yesterday I had to install VS2008 SP1 on a computer with limited disk space on C:\

Even though VS2008 itself was installed on D:\ the installer (which is 800MB) still required 3.3 GB of space on C:\

I molested C: until I had enough space to perform the upgrade, but I felt something was wrong. After some searching it turns out that due to various issues, the rule of thumb for Microsoft issued service packs is that you should have approximately 4 times the size of the service pack itself as free space.

So for VS2008 SP1 update, you need 4 x 800MB == 3.2GB of free space on C:\
It might be possible to manipulate it a bit by setting %temp% to another drive or so. Whatever.

Configuring an application through a batch file: what domain am I in?

vanDooren — Thu, 25 Sep 2008 06:00:00 GMT

I recently had to make an install script for one of my applications which manipulates the Active Directory. The app itself is executed via a scheduled batch file, and it expects the domain name as a command line parameter. I wanted the batch file to figure out on its own what the domain was of the computer on which it is executed. That turned out to be harder than I thought, but in the end I found an elegant, if rather hairy solution.

FOR /F "tokens=1* delims=REG_SZ " %%A IN ('REG QUERY HKLM\System\CurrentControlSet\Services\Tcpip\Parameters /v Domain') DO (
SET CURR_DOMAIN=%%B
)

This code snippet will query the registry to get the Domain the computer belongs to. Of course, the REG QUERY command returns not just the name, but a tab separated table with the key name, the key type and the key value. That table is then munged by the 'FOR' loop which uses 'REG_SZ ' (the whitespace is a tab character) as a delimiter. The actual domain name itself is then stored in the variable 'CURR_DOMAIN'

It's crude, but it works rather well. This trick saves me the necessity of having someone configure this by hand when the application is installed. Apart from the fact that this is error prone, it would make the installation procedure more verbose.

I haven't tried this on computer that is not part of a domain. Luckily, that is not a situation that will ever occurr on the network that I have to maintain.

Scheduling tasks via a batch file: where am I?

vanDooren — Mon, 09 Jun 2008 13:14:00 GMT

One of the things that comes with configuring software for a controlled environment is that every single thing involved has to be documented. Some people fail to realize what this means.

It means that my installation cannot contain a trivial phrase like 'Add xyz.exe to the scheduled tasks, and have it run every night with these credentials'. Instead, I have to write something fit for 'Deployment for dummies', mentioning every single click, selection and user entry.

This may seem ridiculous, but otoh the purpose of these procedures is that someone without knowledge of this specific deployment can simply follow the procedure, and end up with a system that is identical to the previous / other one. And it also has the advantage that the procedure documents which files / user accounts / other resources are required for the system.

It also means that it can be a daunting task to create and test app deployment. In my case I often choose to build a script file that is part of an XCOPY deployment, and have the administrator (me) run a script.

I usually opt for the script, because it is human readable (with a definition of human = administrator) and you can read them later in case you need to check how something was done. And the major advantage of scripting things is that I don't need human interaction -> I don't need to write an installation manual that is as verbose and comprehensible as the Silmarillion.

Anyway...

One of the issues with scripts is that a script has to know where the other stuff is which it is supposed to install. Sometimes you can get away with assuming it is in the present working directory %CD%, but oftentime you can't. And hard coding path names is not a safe option.

It took some googling, but it turns out that the environment variable %~dp0 does what I want (thank you Wes Haggard) . It expands to the physical location of the script file that is currently being executed. From that point you only need to know the location of the other files relative to the installation script, and Bjorn Stronginthearm's your uncle.

Some of the software I develop runs on systems where it gets scheduled to execute at specific times, and I use schtasks.exe to configure those tasks in my installation script. Interesting tidbit: if you want to use schtasks to schedule a command which contains spaces, you have to enclose the command path inside the /TR argument with \" quotations so that the task scheduler doesn't get confused. It cannot handle spaces in paths.

Ntbackup.exe and the task scheduler

vanDooren — Tue, 13 May 2008 07:40:00 GMT

Last week we discovered that our weekly backup had failed. The only clue we had was the return code 0x1f. There were no log files, or messages in any of the event logs.

It was already the second time this had happened, but the first time the return code was 0.

After some googling I discovered that the return code can sometimes be 0, even if there was an error. So that might explain why I got that the first time. Then I also discovered that 0x1f seems to be a very generic error.

I read a lot of articles, but finally found the reason buried deep in this Microsoft KB article:

If you use the /um option, it is recommended that you not use the /n option to label the media. Instead, permit Ntbackup to use the default date/time as the label name and description. This eliminates the problem of multiple tapes' having the same label name, which can cause RSM to ask for a manual tape mount and prevent Ntbackup from continuing to completion unattended.

The tape names and description were all unique, but the tape label was simlpy 'Weekly DCS Backup'.

The reason we didn't notice this earlier is that I remove the tapes in time. Only last week I was importing some older tapes to retrieve old data, and the tapes were still in the loader, thus causing a label name conflict.

License server doesn't serve license

vanDooren — Wed, 19 Mar 2008 11:41:00 GMT

Did you know that a Windows2003 Licensing server does not manage Terminal Server 'Per User' CALs?

It is really stupid, but from a licensing point of view, a terminal server that is configured to use 'Per User' licenses only checks if it can find a license server. As soon as it does, the license server does the Jedi mind trick (These are not the licenses you are looking for...) and the terminal server becomes a free for all, limited only by the maximum number of connections configured by the administrator.

At first I thought this could not possibly be true (noone would design anything that stupid, right?) but this and this quickly convinced me otherwise.
What makes it double stupid is that the license server allows me to add 'per user' licenses, and then forces me to go through the whole activation obstacle course, only to ignore whatever licenses were added.

The only way to make sure that our licensing is in order, is to run a nightly script to determine how many TS users we have, and then check if we have enough licenses.
This also means that no matter how many per user licenses you install, they will always show 'N licenses installed, N available, 0 used'

Getting rid of the duplicate SPN in Active Directory

vanDooren — Tue, 11 Mar 2008 10:04:00 GMT

Last week I noticed an error in the domain controller event log with Event ID 11:

Event Type:   Error
Event Source: KDC
Event Category:    None
Event ID: 11
Date:         2/29/2008
Time:         1:43:18 PM
User:         N/A
Computer: SE-DOMAINCONTROLLER01
Description:
There are multiple accounts with name MSSQLSvc/ComputerName.DomainName.SysName.Company.Local:1433 of type DS_SERVICE_PRINCIPAL_NAME.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The name of the computer in the SPN was one of the process control databases I had replaced earlier. Our process control servers each have a specific role, and their computer names are tied to that role. This is a side effect from the process control system software that we use, and I cannot change this.

This means that the old server and the new server will have the same name.

The process software also doesn’t allow me to change domain membership while the software is installed, and if anything goes wrong with installing and configuring the new server, I should be able to bring the old system back online as soon as possible.

So I simply disconnected it, made the new server with the same name a domain member, installed the application, synchronized everything and the system worked fine. I checked the event log of the new server, but couldn’t see anything suspect. However, looking back through the DC system log, I discovered that the problem started at that exact time.

I did a little bit of research, and I found out that the servicePrincipalName attribute basically tells anyone who wants to know that a service with a certain principal name (duh) is running with the credentials of the Active Directory account with which it is registered.

Since a service with a specific ID can only run with 1 account, having duplicates on the network is bad.

Using ldifde, I found out that the service principal name MSSQLSvc/ComputerName.DomainName.SysName.Company.Local:1433 was linked to the user account ‘XyzAdmin’ and the computer account ‘SE-XYZ01’

Using adsiedit.msc, I had to delete one of the SPNs from its containing account. I checked the service on the SE-XYZ01 server, and the SQL server was configured to run as local service. This means that the correct SPN link is to the server account, and not the XyzAdmin account.

Unfortunately I couldn’t check anymore because the old server was already ‘recycled’ but I seemed to remember that the SQL service was configured to run with the XyzAdmin account instead. When I deleted the link, I wrote an entry in the server logbook, writing down exactly what I removed where, and I also saved the deleted info in a text file ‘just in case’.

From the moment I did this the error did not occur anymore, so I deleted the right SPN. Even if it would have been the wrong one, I could have put it back easily with either setspn, or adsiedit.msc.

I think I will make it part of the server replacement procedure to make an ldifde dump before and after, so that I can more easily diagnose possible problems. I also added a line in my daily backup scripts to make a backup of this dump every night for solving problems like this.

Another thing I thought of later was to make a disk image of the old server that needs to be replaced. That way I can uninstall the application, take the computer out of the domain correctly, and install the new server and still know for certain that if anything goes wrong, I can restore the old system to the exact same state from which I started without having to waste any time with the backup and recovery procedures, which can take a long time for certain servers.

As an afterthought I asked my fellow MVPs what the point was of having SPNs in AD in the first place. After all, if a service runs with certain credentials, it will be authenticated when it starts, so what is the added value of registering that information persistent in AD?

This is what MVP Joe Kaplan had to say:

Kerberos uses SPNs extensively.

When a Kerberos client uses its TGT to request a service ticket for a specific
service, the service is actually identified by its SPN. The KDC will grant
the client a service ticket that is encrypted in part with a shared secret
that the service account as identified by the AD account that matches the
SPN has (basically the account password).

In the case of a duplicate SPN, what can happen is that the KDC will
generate a service ticket that may be created based on the shared secret of
the wrong account. Then, when the client provides that ticket to the service
during authentication, the service itself cannot decrypt it and the auth
fails. The server will typically log an "AP Modified" error and the client
will see a "wrong principal" error code. I forget the exact error code and
description, but hopefully that's close enough. :)

So, duplicate SPNs are very bad, much in the same way that duplicate UPNs
are bad. Both can cause Kerb auth to break and Windows uses Kerb for auth
everywhere it can.

gpupdate and psexec, and a moment of D'oh

vanDooren — Sat, 23 Feb 2008 13:46:00 GMT

I had to do a security change today which required each XP machine to first update its policies, and then reboot to make sure that the required changes went into effect immediately.

To do the update I used psexec in conjunction with gpupdate /force to tell each machine to update its policies remotely, and I discovered something funny: if no user is logged on to the remote machine, you'll get an error that the user settings could not be updated. Which is fair enough I suppose, except that it took me a while to figure out.

To do the reboot I used psshutdown. Mental note to self: 'If I want to use psshutdown again to reboot all client machines on the network, I should not forget to specify -r' The maintenance people entered the server room just as I thought 'Hm, why is it taking so long for the computers to come back online...?'

Of course when manufacturing noticed that they went down and didn't come back up, they were 'worried'. Luckily all our client machines are located in the server room, with kvm cables running through the plant so booting all of them back on was done in a minute.

It was really cool to have the diagnostics window open and then 'BAM' drop every client node from the network at the same time.

Active Directory disjointed namespace problems

vanDooren — Tue, 22 Jan 2008 07:17:00 GMT

Last week I was redlining SOPs (Standard Operating Procedures) on a test network for some of the Windows Domain specific situations we sometimes encounter on our process control network.

This specific SOP handled Domain Controller promotion and demotion for our process network. It should be noted that in our process control network, Domain Controllers are also DNS servers.

I made a mistake and completely screwed up the Active Directory. Instead of reloading the network, I decided to figure out what was wrong and solve it properly, since there was no rush.

How I screwed up the domain controllers

Just before I promoted the 2^nd DC, I went to the system properties tab, where I made a crucial mistake. My procedure said to clear the DNS suffix, and I cleared the checkbox that said ‘change DNS suffix when domain membership changes’ instead.

With 20/20 hindsight this was a pretty stupid thing to do. What this does is it prevents the new DC from getting the name ‘DC2.networkname.companyname.local’ and instead let it keep its old name ‘DC2’

This problem is known as a disjointed namespace, where the FQDN of a server does not match the domain of which it is a member. Active Directory looks for FQDNs when it needs to replicate or do other things, so without a FQDN you get all sorts of helpful errors like ‘The RPC server is unavailable, this might be a DNS problem’

Again looking with hindsight, the error details and the requested names should have made it obvious that DNS was working just fine, but that the specified name was indeed not available in DNS.

I tried various things to solve this problem, but the most stupid one was probably renaming DC2. I still don’t know why I thought this was a good idea. Perhaps to force a new name registration in AD?

Of course this failed because Active Directory didn’t replicate, so the name change never propagated either. I had only made the problem worse. DC1 still thought that DC2 had its original name, and it wouldn’t even try to find DC2 at its new name.

What I did to make it right again

Making things right when you hose Active Directory is not easy (or sometimes downright impossible perhaps) but there is a way out of the aforementioned mess.

Google turned up this page which explains what a disjointed namespace is, and it links to a Microsoft KB that has a script for making the DNS suffix of a server equal to the domain name.

When that was done, I renamed the DC2 computer object on DC1 (which still had its old name) to the new name of DC2. I changed the TCPIP settings of both DCs so that DC1 became the preferred DNS server for DC2, and vice versa. This was to insure that they could resolve each other.

I fixed the records in the DNS so that DC2s original name was not mentioned anymore, and I verified that the GUID associated with the DC2 alias record was indeed correct.

At this point I could already synchronize from DC1 to DC2, but not the other way around. Running netdiag on DC1 I was informed that LDAP still had a reference to the original name of DC2, and that some DNS records of this DC were not registered correctly on the DNS running on DC2. It told me to wait 30 minutes in order for DNS server replication to succeed.

I used adsiedit.msc on DC1 to throw away a couple of references in AD to the original name of DC2, as well as the FRS settings for DC1 and DC2. There wasn;t much more to do, so I decided to wait for the DNS replication.

One cup of coffee later netdiag gave no errors anymore, and I could successfully replicate between all Domain Controllers.

Moral of the story

There are several conclusions I could make now:

DNS and Active Directory are complex things, and I don’t know enough about them yet. As long as everything keeps working, it is easy enough to administer a Windows Domain, but as soon as there is a serious problem, the administrative wheat is separated from the chaff, so to speak.
If you are running an important network, you have to have SOPs for all the things you do on the network, no matter how simple, and you have to follow them. And as important: test them so that you know they will work.
Have a test environment where you can horse around and experiment to your heart’s content. Knowing what could go wrong if you make mistakes, and knowing how to solve the mess can be invaluable. If you don’t have lots of hardware lying around to simulate your environment, try to run a virtual network using vmware and an old server. I use a DELL 2900 with $g RAM, 4 cores and a RAID5 configuration of 15 KRpm SAS drives which we have on standby as a spare.
But the most important conclusion: in this business you are never done learning. It will take a lot of effort to bring my understanding of Active Directory and DNS to the same level as my understanding of C++ and software development. Lucky for me that I like the world of Systems Engineering.

Terminal Server licensing pitfall

vanDooren — Mon, 17 Dec 2007 12:16:00 GMT

Today I got a phone call from one of the automation engineers, who told me that he couldn't connect to one of the terminal servers anymore.

I set out to investigate, and while verything seemed to be in order, the server management page indicated that the terminal services were not licensed for the offending server. It also said that the evaluation period would end after 120 days, after which it would stop working.

This was odd, because I remember installing the CALs... about 4 months ago when we had to rebuild the network... which is about 120 days approximately. Too much of a coincidence, so I checked the system event log and sure enough there was a message that the grace period had expired. This also told me that the same thing was about to happen for our other terminal servers.

On each of the terminal servers I could open the licensing configuration, and get the error message that no licensing servers were found. I could then manually specify a server name and it would find all licenses, but if I closed it and then opened it again I got the same error. I even got this error on the terminal server that was also acting as a license server, which was ultimate stupidity imo.

I consulted the Windows 2003 Admin companion, which turned up nothing, and the Windows 2003 help files were equally useless.

Then I found this page:

http://www.microsoft.com/technet/community/en-us/terminal/terminal_faq.mspx
and this one:
http://www.msterminalservices.org/articles/Terminal-Server-License-Service-Discovery-Part1.html

Especially the last one is very useful because it has flowcharts detailing how server discovery works for terminal servers. As it turns out, there are quite a few stupid design decisions in there.

First of all, there is no easy way to manually specify licensing servers. Even the licensing configuration snap in doesn't allow this. You have to configure this in the registry of each individual server.

Secondly, terminal servers don't check if they have a license server running locally. Especially in small networks this will be the case. It would be trivial to implement, and makes life easier for end users.

But thirdly and most importantly, the license server setup itself is braindead. You have the option to choose between enterprise mode and domain or workgroup mode. The former should be for large multidomain enterprises, and the latter for small single domains. When the license server was installed, we chose the latter because we are only running a single domain forest.

Unfortunately, in domain mode, the license server record is only broadcast via active directory if the license server itself is a domain controller. If that's not the case, the license server will just sit there without anyone knowing it's there. Even the local terminal service won't know. With the entrprise license server option this is not an issue, because it will always be discovered, even if it is not running on a domain controller.

So once I figured all this out, it took only 5 minutes to manually configure the extra registry keeps to force the terminal service to look at our licensing server. We still get discovery errors in the event log because the discovery process still runs, and still can't find any license servers despite the fact that it uses that server. The 'manage your server' screen also thinks that we are still running without a license.

For this last thing there is a hotfix, but since that is not approved by the software vendor of our process control software, I decided to ignore these messages and simply put an entry in the logbook to explain what happens.

Another day, another example of the lack of usefulnes of the windows documentation. I think it would have been a good idea to document things like discovery process / configuration implications etc in the documentation, but that's just me of course.

Group Policy gotchas

vanDooren — Tue, 04 Dec 2007 11:20:00 GMT

Yesterday I discovered 2 Group Policy gotchas the hard way.Since these things are not written down anywhere, I figured I'd post them on my blog.Note that I am not an enterprise level systems administrator. I am a systems administrator for a single forest single trees single domain network. As such, whatever is a good solution for me might possible not be a good solution at the enterprise level.

Default ‘apply' permission for authenticated users

Whenever you create a new policy object, it's ACL will have the ‘Apply' permission set for all authenticated users.This seems innocuous enough, since you change it anyway when you configure the policy permissions as required. Except that this has a side effect.

Suppose you create a new policy object, configure it (which might take time) and then only change the permissions afterwards. What happens? If someone refreshes its policy settings or applies them before you had the chance to change the security, the policy gets applied without regard for which targets it was really meant.

What is especially insidious in this scenario is that a Domain Controller refreshes its policies every 5 minutes. If the policy contains only user settings this might not be a problem, but in my case the policy contained settings to disable all external mass storage...Goodbye USB, goodbye CDRom...

What's even nastier is that afterwards there is no visible trace that this has happened. The Resultant Set of Policy clearly shows (afterwards) that the policy is not in effect on the DC, so why-oh-why doesn't it see its external mass storage anymore...

The moral of the story: whenever you create a new policy object, either make sure that it is not applied to anyone before you configure the policy settings. Or configure the security settings first so that it only gets applied to the correct targets.

Persistent results of previous policies

Another beginner mistake is failing to realize that sometimes the results of a policy application are permanent.For example, if you create a policy to force some values for certain registry keys, those keys will keep their value even if the policy is removed.

This will leave computers in a state that is not logically consistent with their Resultant Set of Policy, because 2 computers with the same set of applicable policies could end up with different configurations.

One way to achieve this is to have 2 policies: One policy to enable a configuration change, and another to reverse or disable the change when it is not needed anymore. In the group policy list, you can then apply the ‘undo' policy first to all computers, and then apply the ‘do' policy afterwards to the required computers.

Another way would be to have 2 security groups so that all computers or users are members of just one of them. Apply the ‘do' policy to the first group and the ‘undo' policy to the second group. By moving computers from group to group you can then either apply or negate the policies.

The moral of the story: In order to ensure consistency and to make your life easier, you should have some mechanism to revert the results of a policy with persistent results.

Disabling mass storage in a Windows network environment

vanDooren — Tue, 04 Dec 2007 10:58:00 GMT

One of the issues involved in running a pharmaceutical process network is that QA has to give its blessing to the network to allow your company to produce meds. For example, QA has to be convinced that all actions are auditable, that people don't have more rights than they should...One of the other things is that it should not be possible for users to copy data from the network, or put data on local computers. To achieve this, I needed to disable mass storage (floppy, CD, USB) on all computers that aer accessible by non-administrators. Lucky for me, there are only a handful of computers that are outside of a security controlled environment.I wanted to solve this with Group Policy to make it easy to modify

I also found 2 gotchas that you might want to avoid.

The first thing you need to do is to create a new policy using the administrative template described in this KB article, and then configure the correct settings to disable specific mass storage devices.

This alone is not enough, since it only works for devices that were already on the system when the policy was applied. If a user plugs in a new USB disk, the policy will be overridden and overwritten by the system.

To prevent that, you have to add a registry key to the registry security settings, and then set the permission for ‘SYSTEM’ and ‘domain\users’ to ‘Deny full control’

This will prevent the system and the user from accessing the USB mass storage driver when a new device is plugged in.

And because these changes persist even if the Group Policy is lifted, you need to make a second policy to undo these changes.