Here's my "theory". I put it in quotes as there's some parts of the puzzle I'm missing because of not large enough log files..but I'm pretty sure based on what I've seen to come to this conclusion.
So where did I go wrong?
By assuming that my biggest target of the blog/web was where the attacks would come in from. Thus I spent most of my energy ensuring that passwords were proper, that the Microsoft software was patched.
And that's not where they got me.
But where I went wrong was making and taking a risk assumption. And before I detail out that risk I took, it reminds me of this morning on CaliforniaEdition.org where the folks in Lake Tahoe that got their houses destroyed in the Angora fire, many of them ensured they got building permits before December of 2007 so that they wouldn't have to be impacted by the new fire building codes. They wanted to have the flexibility in their budgets to make risk decisions even though they personally know what the risks are. Yet they choose to accept the risk of the older, vulnerable building code rather than the newer, stricter code.
Humans have a natural condition to not thing the worst but think the best. I'm guilty of that as well. The glass is half full, not half empty. As such there are times we don't make the right risk decisions. The human condition that "I'm not big enough"... or "it won't happen again". I took a chance based on the needs of a certain application to leave on the system vulnerable code. I had to for management and adminstrative reasons. But where I failed was not making and taking remedial actions to counter that risk. I made the decision because I didn't think that I the risk I was taking was big enough.
In that sense I was no better than the folks running the Death Star...you see behind this blog server was a weakness. My decisons were made regarding a piece of software running some listserves that are housed on this server.
There was a known weakness in my defense system.
IceWarp Web Mail Multiple Vulnerabilities - Advisories - Secunia:
http://secunia.com/advisories/17046/
I knew about it. But had to be on this older version because when Merak first came out with the new version they broke a key funcationality of the way the listserves that are administered. There's a confirmation means that can be done via email and when they came out with the updated version, it broke this. And they didn't come out with a new version for a long time. Then, everytime I went online to check to see if they had a new version, I failed to see that they already had 9.1 out which fixed it. So I stayed back on an older vulnerable version because at first it wasn't fixed, and then when it was, I didn't realize that the issue I had with the funcationality finally had been fixed.
Where I failed ..was that I should have taken remedial action.
I should have at least gone into the firewall and ensured that the Webadmin port for Merak was limited to those listserve admins (not only myself but others) had access. You can do this with any firewall including the build in one on any server. If I would have merely done that, the hack would not have been able to be accomplished. I also failed because I didn't enable Merak to keep logs for that particular access long enough and I'm not sure I can tell exactly who got in.
So how did I determine this? As part of the analysis done by Microsoft support, there a script that runs that grabs the date and time of every file on the system.
In looking at that report I noticed that the folder that dropped the netsrv.exe service (the funky netbios service that was the entry point) was as follows:
Directory of c:\Program Files
06/06/2008 05:40a <DIR> networ~1 networking
If anyone knows me, they know that there's no way I'm up at 3:40 in the morning unless I'm up to get on a plane. So that wasn't me for obvious reasons, let alone the fact that from the prior visual image of the netsrv.exe service was pointing to that location that appeared to be the entry date/time/point.
So in looking more down this file report I spotted the following:
Directory of c:\Program Files\Merak\html\admin\wizards\data\domain
08/03/2007 07:55p <DIR> ..
08/03/2007 07:55p <DIR> _inc
08/03/2007 07:55p <DIR> .
08/03/2007 07:55p <DIR> _xml
05/27/2008 03:22p 292 wizard~1.php wizard.domain.php
06/06/2008 05:39a 2,462 config.php <<<<<<<
Bingo. That folder is the Webadmin location for the web access to the listserves. And a minute before the config.php file was accessed.
Still going down the file listing I see this....
06/06/2008 05:40a 1,558,528 wmupda~1.exe wmupdatesrv.exe <<<
06/06/2008 05:40a 152,734 auto.exe
06/06/2008 05:40a 4,096 cll.exe
06/06/2008 05:40a 45,171 backdo~1.exe backdoorinstall.exe
06/06/2008 05:40a 6,656 bw.exe
06/06/2008 05:40a 39,936 filedate.exe
06/06/2008 05:40a 25,600 inx.exe
06/06/2008 05:40a 24,064 openp.exe
06/06/2008 05:40a 45,056 psinfo.exe
06/06/2008 05:40a 171,008 pd.exe
06/06/2008 05:40a 591 sec.cmd
06/06/2008 05:40a 30,208 rgv.exe
06/06/2008 05:40a 28,160 tcp.dll
06/06/2008 05:40a 40,448 uptime.exe
06/06/2008 05:40a 581 bw.log
Dumping a backdoor, media sharing and sysinternal tools to get information off the drive. Psinfo.exe is a system info tool.
06/06/2008 05:41a <DIR> drwats~1 dr watson
0 File(s) 0 bytes
Directory of c:\WINDOWS\system32\config\systemprofile\Local Settings\Application Data\Microsoft\Dr Watson
06/06/2008 05:41a <DIR> ..
06/06/2008 05:41a <DIR> .
06/06/2008 05:41a 263,364 drwtsn32.log
06/06/2008 05:41a 54,483 user.dmp
2 File(s) 317,847 bytes
Looks like they bluescreen the box, or forced a user.dmp file.
(By the way for all of the blog authors who got a bit upset that the blog server was down for 4 days, this was the reason why... I wanted to ensure that there was every opportunity available for an investigation into the how this happened. This is why I delayed in getting the server back online. The important thing was to understand HOW, because if we don't understand how we run the risk of setting the server right back up in a flawed manner)
Lessons to be learned...
1. Don't assume that a little crack in your armor won't be the very thing someone targets.
2. They aren't going after Microsoft software as we're getting pretty good at patching that.
3. When you make the decision to run older software, review and take mitigations to protect yourself from the vulnerabilities it now brings.
4. If you don't take the time to learn, and instead rebuild the system as is, you just may set the box back up with the same vulnerabilities.
5. Review your options for getting an image of the vulnerable system.
I can't go into details but lets just say that you have to ensure that the OS license you have allows to grab an image, or get a little creative. I didn't need an image of the server for bare metal restoration, I just needed to grab an image in case we didn't get a log file off the box. Thus I used a free image tool to grab a backup that I could later mount and view http://www.runtime.org/driveimage-xml.htm In addition, I had to keep the box up long enough to grab the needed log files and review tools for Microsoft. To ensure that no additional damage to the database was done, I shut off the web site and closed down access between the two servers. The actual time to image the system after I got and overnighted a hard drive (in lovely bright yellow I might add), to then flatten and rebuild the server only took several hours. If you have an event, you cannot "clean the system back up again". You have to be prepared to flatten and rebuild. But before you do that, keep a copy. If this had been a real physical box in my office I would have yanked the harddrives and saved them. In this case I couldn't, but came up with an alternative.
To paraphrase a comment on this blog...
http://weblogs.asp.net/steveschofield/archive/2008/06/23/sql-injection-information-for-iis-admins-and-developers.aspx
Hi Steven,
In theory, once you plug a box on the internet, it's not secure, no matter what you do. All you can do is do the best you can and follow best security practices. Validating request variables and not trusting user input is probably the one thing that will help the most. It might cause some issues with your application, but if it's done with security in mind, it should be sufficient of a reason.
Good luck,
Steve
In theory, once you plug a box on the Internet, it's not secure, no matter what you do. All you can do is the best you can and follow best security practices. If you make a risk decision to accept the risk of vulnerable software, ensure that you counter that risk with a corresponding action to best limit that risk. Don't assume for one moment that your little flaw in the exhaust port is too small in our case to have some Bad guy crack into it. For every risk you take, stand back and make sure that it's an appropriate decision.
And may the force (the good force) be with you....
