ab origine ...

Russian translation for all posts is added

Please read the post http://www.shcherbyna.com/?p=14

A case of mysterious BSOD at tcpip!TcpIndicateData+22b

1. The causes

Recently I was observing strange BSODs on my Vista machine quite periodically when dealing with network applications:

Please, read the rest of this post at http://www.shcherbyna.com/?p=11

Third time to become MVP

Today I received a really great news: I am MVP for a 3rd time!

This is my third “first of July” which brings me exciting news. This time I receive award in DDK profile, which is something different from what I had before. My first nomination was SDK area (2005 - 2006), second was also SDK (2006 - 2007), and third is DDK (2007 - 2008) area nomination. Well, … what can I tell … : thanks Microsoft :)

[...]

Please, read the rest of this post at http://www.shcherbyna.com/?p=10 

Mysterious ExUuidCreate function

I was inspired to write this post after being confused by behaviour of ExUuidCreate function in Windows Vista.

This function is used to generate GUIDs at kernel mode. According to documentation: "ExUuidCreate returns STATUS_SUCCESS if successful; otherwise, if the system is not ready to generate a new UUID, it returns STATUS_RETRY." Practically, if you call this function say in driver which is loading by boot loader (if the driver has "Start" registry value equal to 0 due to SERVICE_BOOT_START flag set when creating service using CreateService) you can observe strange behaviour, especially in Vista.

[...]

Please, read the rest of this post at http://www.shcherbyna.com/?p=6 

A change for SoftIce

As SoftIce is officially dead, you might want to find its substitution. There is one product which aims to fully substitute SoftIce – Syser debugger, which proposes pretty close user interface, and “live” debugging of kernel on a host machine:

[...]

Please, read the rest of this post at http://www.shcherbyna.com/?p=5 

Moving to www.shcherbyna.com

I always wanted to have an own domain, hosting mail, blog and probably some important files for me. However, I always felt the leak in time, and this direction was non primary for me. Until today.

As you probably have mentioned, the server msmvps.com (which is hosting http://msmvps.com/blogs/v_scherbina/) was dead for approximately one week. During these 7 days my brain reorganized some tasks, and I decided to make independent blog which (theoretically) will be constantly up : ).

So let’s welcome www.shcherbyna.com web site. I am going to move completely on this blog, but for some period of time I will be duplicating posts to “Ab origine” as well.

Windows Research Kernel

Have you ever think about accessing sources of Microsoft Windows kernel? Now thoughts might become reality. Quote taken from http://www.microsoft.com/resources/sharedsource/licensing/researchkernel.mspx :

“The Windows Research Kernel (WRK) packages core Microsoft Windows XP x64/Server 2003 SP1 kernel source code with an environment for building and testing experimental versions of the Windows kernel for use in teaching and research. The WRK includes source for processes, threads, LPC, virtual memory, scheduler, object manager, I/O manager, synchronization, worker threads, kernel heap manager, and other core NTOS functionality.

The WRK is useful in design projects that allow your students to explore operating system (OS) principles using the Windows kernel sources. It enables advanced teaching and research by facilitating building experiments and projects based on modifying the Windows kernel, as well as by promoting better understanding of the Windows architecture and implementation.”

Although, one will need to sign documents, and apply for a program, this is a good sign in general. A dark monster sheds lights upon its internals.

Microsoft has opened Singularity sources

Everyone is able to download sources at http://www.codeplex.com/singularity

Bug in MSDN: TDI_EVENT_RECEIVE_DATAGRAM & it's handler

If you ever wanted to handle TDI_EVENT_RECEIVE_DATAGRAM (http://msdn2.microsoft.com/en-us/library/ms801156.aspx) event handler in TDI, you would notice that it's declaration is a bit strange(http://msdn2.microsoft.com/en-us/library/ms801622.aspx):

NTSTATUS
  ClientEventReceive(
    IN PVOID  TdiEventContext,
    IN CONNECTION_CONTEXT  ConnectionContext,
    IN ULONG  ReceiveFlags,
    IN ULONG  BytesIndicated,
    IN ULONG  BytesAvailable,
    OUT ULONG  *BytesTaken,
    IN PVOID  Tsdu,
    OUT PIRP  *IoRequestPacket
    );

Guess what? It's declaration is the same as to a handler of TDI_EVENT_RECEIVE which sets event handlers for TCP protocol and not for UDP! If you want to handle UDP incoming data at TDI level, you most likely will need to find a correct declaration of handler function, because using declaration from MSDN will give your driver one more chance to blue screen :)

Thanks to www.osronline.com, I found a correct declaration: http://www.osronline.com/ddkx/w98ddk/vxdtdi_9lt9.htm which should have the following form:

NTSTATUS ClientEventReceiveDatagram(
    IN PVOID TdiEventContext,
    IN LONG SourceAddressLength,
    IN PVOID SourceAddress,
    IN LONG OptionsLength,
    IN PVOID Options,
    IN ULONG ReceiveDatagramFlags,
    IN ULONG BytesIndicated,
    IN ULONG BytesAvailable,
    OUT ULONG *BytesTaken,
    IN PVOID Tsdu,
    OUT PIRP *IoRequestPacket);

The most interesting thing, is that I spent 30 minutes trying to figure out what's wrong in my code, until I noticed the presence of ConnectionContext param in the declaration of handler, so I asked myself: why would I recieve a connection context when handling connectionless protocol data ... ? So I started googling.

Seems like this issue touches both local documentation for WDK and online documentation at http://msdn2.microsoft.com

DRIVER_VERIFIER_IOMANAGER_VIOLATION in Windows Server 2003 SP2 with latest updates ON

Recently, I've received following error when trying to test my TDI filter driver on Server 2003 SP2 with latest updates ON:

DRIVER_VERIFIER_IOMANAGER_VIOLATION (c9)
Arguments: Arg1: 00000208, (Fatal error) This IRP is about to run out of stack locations. Someone may have forwarded this IRP from another stack. (IRP specified.)

This violation message appeared, when I run applications, which make network connections from network drives. Attaching debugger to debugging session gives more information:

***********************************************************************
* THIS VALIDATION BUG IS FATAL AND WILL CAUSE THE VERIFIER TO HALT    *
* WINDOWS (BUGCHECK) WHEN THE MACHINE IS NOT UNDER A KERNEL DEBUGGER! *
***********************************************************************

WDM DRIVER ERROR: [2k03sp2bug.sys @ 0xF77E00C5] This IRP is about to run out
                  of stack locations. Someone may have forwarded this IRP
                  from another stack (Irp = 859E8F48 ).
IRP_MJ_INTERNAL_DEVICE_CONTROL
[ DevObj=849C3E18, FileObject=849A93D0, Parameters=00006C36 00000020 00000000 00000000 ]
http://www.microsoft.com/hwdq/bc/default.asp?os=5.2.3790&major=0xc9&minor=0x208&lang=0x9
Break, Ignore, Zap, Remove, Disable all (bizrd)?

If I choose Ignore, everything goes fine, and there are no fatal errors, however, I don't think this is a normal behavior, so I decided to create a small TDI filter driver, which layers over \Device\TCP & \Device\UDP providers, and works in pass through mode to demonstrate the problem. I hope this small example will help to eliminate the problem.

The driver is really simple, it calls IoAttachDevice, and set's IO function for handling IRP_MJ controls codes. The DeviceIO function is implemented as:

NTSTATUS Test2k03SP2Bug_DispatchIO(IN PDEVICE_OBJECT pDeviceObject, IN PIRP Irp)
{ 
 /// simply dispatch the IRP down to stack and nothing else
 NTSTATUS   ntStatusOfDispatch = STATUS_SUCCESS;
 PIO_STACK_LOCATION pIRPs    = IoGetCurrentIrpStackLocation(Irp);

 if (pDeviceObject == g_TCPOBJ)
 {
  IoSkipCurrentIrpStackLocation(Irp);

  ntStatusOfDispatch = IoCallDriver(g_TCPOLDOBJ, Irp);
 }
 else if (pDeviceObject == g_UDPOBJ)
 {
  IoSkipCurrentIrpStackLocation(Irp);

  ntStatusOfDispatch = IoCallDriver(g_UDPOLDOBJ, Irp);
 }
 else /// unknown DO, do complete IRP
 {
  Irp->IoStatus.Status = ntStatusOfDispatch;
  IoCompleteRequest(Irp, IO_NO_INCREMENT);
 }

 return ntStatusOfDispatch;
}

As you can see, it simply forwards IRP's to stack, no more processing is done in IO function. Complete sources for driver are attached in this post. Compile them, configure verifier to verify driver with all possible verification tests:

verifier.exe /all /driver 2k03sp2bug.sys

Load the driver. Reboot machine (to allow verifier start checks). Find any network share, run application (from share) which makes intensive network traffic (UDP & TCP) and catch BSOD (if you boot with /DEBUG system will halt, so you can attach debugger, otherwise, you will see BSOD with DRIVER_VERIFIER_IOMANAGER_VIOLATION.

Microsoft is hiring advanced professionals

I discovered one more interesting blog for myself: Microsoft Advanced Windows Debugging and Troubleshooting. At the same time, I noticed that Microsoft hires professionals who are good in low level staff - reverse engineering and debugging of complex solutions which requires knowledge of assembler language & advanced knowledge of undocumented internals of Windows.

This, I believe, is a step towards to solving issues that happens only on customer’s side in complex environments only with some unique conditions and test cases. So, if you feel comfortable with asm, Windbg, IDA Pro, OllyDbg, and other tools, hit the road and join Microsoft in this interesting challenge!

explorer.exe is in danger :)

Kaspersky (http://www.kaspersky.com/) recently released a signature update that treats explore.exe as a virus.

This false signature may affect those people who set the option 'delete virus' ON - it let's anvtivirus delete the suspicious binary from file system. I guess, it may cause unpredictable results, even if WFP (Windows File Protection) will try to recover explorer.exe binary. More detailed story is described here: http://www.channelregister.co.uk/2007/12/20/kaspersky_false_alarm/

NG tips document

Newsgroups tips doc has been published recently on my blog. I am pretty sure it will be useful for newbies who are going to use Microsoft newsgroups. For more details, please visit this link.

The case of Task Manager that does not kill

Quite long time ago, my friend Vadym Stetsiak described a bug of Task Manager, which allows to disallow (!) the killing of a process, if it's name is lsass.exe. In order to test this bug, you can rename any executable file into lsass.exe, run it, and than try to kill it from Task Manager. You will fail to kill it under XP:

One year passed. Many updates were released by Microsoft. However, this bug is still present, and it still remains in upcoming SP3 for Windows XP. I checked the latest RC for XP SP3, and the same happens. I asked myself - why? In order to answer this question I did the following:

I opened WinDbg. Attached to running instance of Task Manager. In command window of debugger I typed:

0:003> bp user32!MessageBoxW
0:003> bl

0 e 7e46630a     0001 (0001)  0:**** USER32!MessageBoxW

This action sets breakpoint to MessageBoxW - I just want to catch the moment, when I see msgbox saying that lsass.exe is a critical process, that cannot be deleted. I rename some executable into lsass.exe, and run it. Then I try to kill this application via Task Manager. Breakpoint is hit, I see the following code:

However, I need to know the previous location which called this chunk of code. To do that, I need to analyze the region of code which is upper the call to MessageBoxW. To do that, I would use IDA.

So, I load into IDA the file C:\Windows\System32\taskmgr.exe. Wait untill the file is parsed. Then I jump to 0100c16d: 

Press G and specify 0100c16d:

.text:0100C15C                 lea     eax, [ebp+Caption]
.text:0100C162                 push    eax             ; lpCaption
.text:0100C163                 lea     eax, [ebp+Text]
.text:0100C169                 push    eax             ; lpText
.text:0100C16A                 push    dword ptr [ebx+4] ; hWnd
.text:0100C16D                 call    ds:MessageBoxW

This is exactly what I need. I try to locate the xref to this portion of code. And I see, that this code is called from the following address (the only one xref):

.text:0100C0EE                 push    ds:off_1001F14[edi] ; lpString2
.text:0100C0F4                  push    dword ptr [esi+88h] ; lpString1
.text:0100C0FA                 call    ds:lstrcmpiW                             ; compare name of selected process with some string
.text:0100C100                  test    eax, eax
.text:0100C102                  jz      short loc_100C11D                     ; call MessageBoxW if comparison is OK

As you can see, there is a string comparison which is done using lstrcmpiW function. Let's see, what data is comparing when this code executes:

0:003> bp 0100C0FA
0:003> g

Trying to kill again fake lsass.exe, the breakpoint hit, I see the following data passed as params to lstrcmpiW: сsrss.exe, winlogon.exe, smss.exe, services.exe and finally, lsass.exe. So, as it logically outcomes from research, these names are hardcoded into Task Manager, and are treated as a special processes, however, the check is buggy: I would rather make a check against process path, not just the process name.

Windows XP SP3 RC is released

Standalone Update Package for Windows® XP Service Pack 3 Release Candidate is avialable from this location: http://www.microsoft.com/downloads/details.aspx?FamilyId=75ED934C-8423-4386-AD98-36B124A720AA&displaylang=en  

my first FAQ ...

Well, it's "better later then never". I decided to write a small FAQ document on TDI. The idea to write this document came into my head when I noticed that there is not so much documentation on this field. Yes, I realize that TDI becomes depricated, and after Vista it would not be supported, but ... XP is still popular, and TDI filters, TDI clients are quite often used even nowdays.

Here is initial version of the FAQ.

reallocation to Suisse

I was quite inactive a big amount of time because of my recent reallocation to Switzerland. It took some time for me to settle down, and start a new life in another country. 

I always thought it would be easy to change the place of living. And I was wrong. Different laws, different kind of life makes a big impact on perception, especially at first time. Everything is different. And I am 23. I can imagine what people feel when they change the place of living at the age of 40 ;)

Anyway, everything is getting better. I continue to work in the same profile, and I start to have more time to share myself with NG communities and this web-blog.

double standards

For those who were born in ex Soviet Union countries there is one small (among many big others), but really interesting problem: different Slavonic->Latinics interpretations of names. For example, the name Vladimir can be treated into Ukranian variants: Volodymyr, Volodimir, Volodymir, …

Recently this problem touched me, because my original name (Володимир Щербина) should be translated into Ukranian “Volodymyr Shcherbyna” which differs from Russian “Vladimir Scherbina” alias I used for at least two years while being MVP. I decided to change my name into “official” and “correct” variant just to avoid double standards.

So, from now, I am Volodymyr Shcherbyna ;)

A shame on Kaspersky ...

As one of the stages in my work, I do tests of different antiviruses with components I develop. This allows me to handle incompatibility issues, profiling BSODS and other critical errors that might appear during software lifecycle ;)

These days I was looking at Kaspersky (latest trial version from official site, as it was mentioned in comments the version was not the latest). Shortly speaking, I de-installed it after one day of using. The system was working slowly, I think there were leaks in kernel mode. Since I was making only tests, I tested needed test cases and de-installed the antivirus using _legal_ uninstaller located in program's folder. Now, here is what I see each time when I start my Microsoft Office Outlook:  

Undefeatable files & folders in Windows XP SP2 - a bug in SHFileOperationW

Recently I was surprised with one interesting behavior of my Windows XP box. I was playing with long name files and noticed that major part of my shell extensions do not work with files, whose path is longer then 260 symbols. I also noticed, that Windows Shell does not allow me to create long file names. When you create a folder, you're allowed to specify 260 symbols and that's all! Here is what I see when I try to create text document in "extra long" folder:

       Fig 1. (impossible to create a text document in long named folder)

However, theoretically we're allowed to create "long paths", here is what MSDN says about long file names(CreateFile documentation): In the ANSI version of this function, the name is limited to MAX_PATH characters. To extend this limit to 32,767 wide characters, call the Unicode version of the function and prepend "\\?\" to the path.

OK, I have a chance to test this feature. Let's create a test application that creates several folders with really long names. The application creates a folder on drive C, let's call it "A". Then it creates another folder placed in "A", let's call it "A1", and finally it creates three files in "A1".

I run the application on my computer, and ... folder is created, but windows does not allow me browse the files in it. I see the following msgbox when trying to access the A1:

                    Fig. 2. (impossible to access "A1" folder and see files)                            

     The most interesting thing is that Windows can't delete these files. When you delete the "A" folder Windows says "Cannot delete ... The filename you specified is not valid or too long. Specify a different name."

Things are getting more and more interesting. Setting breakpoint in SoftIce to DeleteFileW && DeleteFileA  functions:

: bpx DeleteFileW

: bpx DeleteFileA

Gives me nothing, functions are not called when I try to delete the folder via explorer context menu.  At the same time, it's worth to check what happens in explorer.exe before files and folders are going to be deleted! Let's set breakpoints in SoftIce to SHFileOperationW, SHFileOperationA functions:

: bd *

: bpx SHFileOperationA

: bpx SHFileOperationW

Then, let's delete folder once again, and ... SHFileOperationW is invoked by explorer.exe. This already gives me a hint, that SHFileOperationW has a bug, and shows strange msgbox without actually removing the folders & files.

Closer look at the function determines the following. Function can be divided into two logical parts. First creates a thread using SHCreateThread in order to display UI (for example, the dialog that tells you : "Do you really want to delete the _item_name") (I think there is not need to cover this direction, everything is pretty simple there). Second part (which is executed in the explorer thread) is actually the logics that enumerates all folders and files and forms the list of items to delete. 

Enumeration is done ... of course, using FindFirstFile and simular Find* functions. Let's take a look at MSDN regarding the internals of WIN32_FIND_DATA structure:

typedef struct _WIN32_FIND_DATA
{ 
[...]
    TCHAR cFileName[MAX_PATH]; 
[...]
}WIN32_FIND_DATA, *PWIN32_FIND_DATA, *LPWIN32_FIND_DATA;

Seems like cFileName member cannot hold more then 260 symbols (MAX_PATH). However, this fact is not the actual answer to my question: where the buggy code fails.

Let's return back to SoftIce. Make sure that breakpoint to SHFileOperationW is set (just write the : bl command and see the list of breakpoints). Delete the folder once again, and step by step continue analysis:

[Inside the SHFileOperationW] Here is the creation of UI thread:[/Inside the SHFileOperationW]

.text:7CA6FDDE 68 2B 5C A9 7C              push    offset sub_7CA95C2B
.text:7CA6FDE3 6A 00                             push    0
.text:7CA6FDE5 56                                  push    esi
.text:7CA6FDE6 68 C3 F1 A6 7C               push    offset loc_7CA6F1C3
.text:7CA6FDEB FF 15 74 1B 9C 7C           call    ds:SHCreateThread
.text:7CA6FDF1 EB 0F                             jmp     short loc_7CA6FE02    (jump to additional checks, and finally it goes to 7CA6F64A)

[Inside the SHFileOperationW, 7CA6F64A] Here is the top level routine (7CA6F64A) that after different checks calls enumeration of files & folders [/Inside the SHFileOperationW]

.text:7CA6F6BC 74 28                            jz      short loc_7CA6F6E6
.text:7CA6F6BE 48                                dec     eax
.text:7CA6F6BF 74 1E                            jz      short loc_7CA6F6DF
.text:7CA6F6C1 48                                dec     eax
.text:7CA6F6C2 74 14                            jz      short loc_7CA6F6D8
.text:7CA6F6C4 48                                dec     eax
.text:7CA6F6C5 8D 45 C4                       lea     eax, [ebp+var_3C]
.text:7CA6F6C8 74 07                            jz      short loc_7CA6F6D1
.text:7CA6F6CA 68 74 3B 9D 7C             push    offset aCopy?   ; "Copy? "
.text:7CA6F6CF EB 1D                           jmp     short loc_7CA6F6EE
.text:7CA6F6D1                   ; ---------------------------------------------------------------------------
.text:7CA6F6D1
.text:7CA6F6D1                   loc_7CA6F6D1:                           ; CODE XREF: DoEnumeration+7Ej
.text:7CA6F6D1 68 64 3B 9D 7C              push    offset aRename  ; "Rename"
.text:7CA6F6D6 EB 16                            jmp     short loc_7CA6F6EE
.text:7CA6F6D8                   ; ---------------------------------------------------------------------------
.text:7CA6F6D8
.text:7CA6F6D8                   loc_7CA6F6D8:                           ; CODE XREF: DoEnumeration+78j
.text:7CA6F6D8 68 A4 4E 9D 7C              push    offset aDelete  ; "Delete"
.text:7CA6F6DD EB 0C                            jmp     short loc_7CA6F6EB
.text:7CA6F6DF                   ; ---------------------------------------------------------------------------
.text:7CA6F6DF
.text:7CA6F6DF                   loc_7CA6F6DF:                           ; CODE XREF: DoEnumeration+75j
.text:7CA6F6DF 68 54 3B 9D 7C               push    offset aCopy    ; "Copy  "
.text:7CA6F6E4 EB 05                             jmp     short loc_7CA6F6EB

[/Inside the SHFileOperationW, 7CA6F64A] Here is the top level routine (7CA6F64A) that after different checks calls enumeration of files & folders [/Inside the SHFileOperationW]

The most interesting things happens when CPU executes this line:

7CA6EC3B   E8 5EFCFFFF      CALL SHELL32.7CA6E89E     

7CA6E89E is the proc that enumerates files & folders, and if something went wrong, it returns non-zero number in EAX, which is mapped to shell error. If this function returnes zero, it means that everything is OK. Let's take a look deeper into it. First looks shows that there is common and well-known sequence of FindFirstFile->FindNextFile->FindClose->recursive calling itself:

[Inside the SHFileOperationW, 7CA6E89E] enumeration of files & folders [/Inside the SHFileOperationW]

.text:7CA6E8AE FF 75 10                          push    [ebp+pszPath]   ; lpFileName
.text:7CA6E8B1 C7 45 F8 01 00 00+           mov     [ebp+var_8], 1
.text:7CA6E8B8 FF 15 EC 15 9C 7C            call    ds:FindFirstFileW

[...]

.text:7CA6E96D 57                                push    edi             ; lpFindFileData
.text:7CA6E96E FF 75 FC                       push    [ebp+hFindFile] ; hFindFile
.text:7CA6E971 FF 15 F0 15 9C 7C          call    ds:FindNextFileW
.text:7CA6E977 85 C0                            test    eax, eax
.text:7CA6E979 75 8E                            jnz     short loc_7CA6E909

[...]

.text:7CA6EA41                   loc_7CA6EA41:                           ; CODE XREF: EnumerateFiles+15Dj
.text:7CA6EA41 FF 75 FC                          push    [ebp+hFindFile] ; hFindFile
.text:7CA6EA44 FF 15 F4 15 9C 7C            call    ds:FindClose
.text:7CA6EA4A 8B C7                             mov     eax, edi
.text:7CA6EA4C E9 A0 00 00 00                jmp     loc_7CA6EAF1

[...]

; two calls to PathAppendW in order to form the path and recursively pass it to 7CA6E89E:

.text:7CA6EA79 50                                push    eax             ; pMore
.text:7CA6EA7A 53                                push    ebx             ; pszPath
.text:7CA6EA7B FF 15 80 1C 9C 7C         call    ds:PathAppendW
.text:7CA6EA81 85 C0                            test    eax, eax
.text:7CA6EA83 74 44                            jz      short loc_7CA6EAC9
.text:7CA6EA85 68 50 87 9C 7C              push    offset a_       ; "*.*"
.text:7CA6EA8A 53                                push    ebx             ; pszPath
.text:7CA6EA8B FF 15 80 1C 9C 7C         call    ds:PathAppendW

[Inside the SHFileOperationW, 7CA6E89E] enumeration of files & folders [/Inside the SHFileOperationW]

And the problem is in the second call to PathAppendW. When the path exceeds the MAX_PATH, the PathAppendW failes. The following code fails in my case:

[inside the buggy shell code]

.text:7CA6EA85 68 50 87 9C 7C              push    offset a_       ; "*.*"
.text:7CA6EA8A 53                                push    ebx             ; pszPath
.text:7CA6EA8B FF 15 80 1C 9C 7C         call    ds:PathAppendW
.text:7CA6EA91 85 C0                            test    eax, eax
.text:7CA6EA93 74 24                            jz      short loc_7CA6EAB9

[/inside the buggy shell code]

And the execution is passed to 7CA6EAB9 which sets the 47Ch error and return from function. I hope this bug will be fixed (btw, it's worth to check it on vista).

P.S. You can use the source file attached to this post to repeat my test case.