Security: Bypassing Hardware-based Data Execution Prevention
Security is bafflingly hard. I am always impressed with how easy it is to bypass the bast-laid plans of software architects.
Matt Miller and Ken Johnson, two of my colleagues at Positive Networks, have just released a paper, available here, detailing a way to bypass Windows hardware-based data execution prevention (DEP). For those who don't know, hardware-based DEP makes use of features of the latest CPUs to mark memory pages as non-executable, thus preventing certain classes of attacks that make use of code injected into data buffers. It's a great idea in principle, and the implementation is a good one.
The fundamental problem is that, in the real world, any change as drastic as DEP will certainly cause application compatibility problems. As Raymond Chen has covered extensively (here, for instance), Microsoft can't just go around breaking old applications. So, they had to provide a way to disable DEP on a per-program basis. Actually, it's even worse than that: programs often load shared DLLs, and the presence of a known-incompatible DLL requires DEP to be disabled.
Matt and Ken sum it up nicely:
In closing, the authors would like to make a special point to indicate that
Microsoft has done an excellent job in raising the bar with their security
improvements in XP Serivce Pack 2. The technique outlined in this document
should not be seen as a case of Microsoft failing to implement something
securely, as the provisions are certainly there to deploy hardware-enforced DEP
in a secure fashion, but instead might be better viewed as a concession that was
made to ensure that application compatibility was retained for the general case.
There is almost always a trade-off when it comes to providing new security
features in the face of potential compatibility problems, and it can be said
that perhaps no company other than Microsoft is more well known for retaining
backward compatibility.
As is usual with these guys, the paper is well-written and easy to follow. I'd highly encourage reading it - it's an excellent object lesson in security design trade-offs. Matt will be presenting at the BlueHat Conference at Microsoft on October 13-14 on a slightly different topic; check him out if you get a chance.