the blog => anything goes

Yes, I'm now a Linux developer.

About 6 months back, I got an opportunity to work on the AVR Toolchain - a port of GCC and related software for the AVR and AVR32 microcontrollers. Having always been interested in compilers and programming languages, I grabbed the chance with both hands, and have been happily hacking on it since then. While GCC can be developed on Windows, Linux is the natural platform and is what most developers use, so I had to switch to Linux as well.

The switch to Linux was frustrating for a while, as I suddenly had no access to my favorite tools that I'd gotten used to for years. No Visual Studio, No WinDBG No WinMerge.., the list is long. You start working on a problem, and you realize you need to use some tool, say "Find in Files". Now you have two problems - finding a tool that can do "Find in Files" and knowing enough about it to get the job done , and your original problem. What made things worse was that most of the work was done SSHing into another machine, so practically no GUI applications were available.

So I decided to go cold turkey and stop using Windows for a while, and it paid off handsomely. After a few days of constantly hitting Google and "man <command>" every 5 minutes, I gradually got used to it. One thing that greatly helped ease the transition was that I was already using VsVim, a Visual Studio extension that emulates the Vim editor. If you are a serious software developer, I can't recommend learning Vim enough - it really changes the way you think about editing code (once you get past the infamous learning curve).

Now, after 6 months, I actually favor the command line over a GUI for most applications - it really lets you get a lot done over a few keystrokes (once you know what to key in, of course!).

PS: Only when I logged in did I realize that my last post was made more than a year ago. Time really does fly as you grow older :)

Naggy, the AVR Studio extension I wrote that provided live compiler diagnostics, now supports visually showing conditionally excluded code, again courtesy Clang. You can download the latest version here.

Most of the Clang integration code had to be rewritten for this though, as the "black box" API provided by Clang didn't expose enough details to make this work. It took some time to get familiar with that, and I still don't think I've understood Clang's idea of object ownership completely, but boy, was it fun! I basically register a custom PPCallback with the Preprocessor and then listen to the entry/exclusion of conditional preprocessor directives. What made life difficult was that the callbacks weren't fired by Clang in any meaningful order - you'd sometimes get a callback for #endif before the #if callback, based on whether the #if evaluated to true or false. I eventually ended up just recording the source location and entered/not entered information in the callback. When it's time to show the excluded code, I calculate the actual excluded blocks of code by sorting the source locations and then pairing them.

I had to change Clang code to have the callback determine whether a particular conditional preprocessor block was entered or not. I also discovered and fixed a bug in the codebase, and I'm hoping to push these changes back into the Clang codebase.

Now that the lowlighting code uses the low level API, I couldn't have the diagnostics finding code use the higher level one - that would mean that the source file would get processed twice, once for diag finding and once for lowlighting. So there were some changes there too to unify both of these clients to share the Clang instance and have code processed only once. Hopefully I didn't break anything that worked earlier :)

As usual, the source code is up at https://github.com/saaadhu/naggy/, and you can download the extension itself at https://github.com/downloads/saaadhu/naggy/Naggy%20v0.2.vsix

Choreo is an AVR Studio extension that lets you write and execute macros within the IDE. You can download it from https://github.com/saaadhu/choreo/downloads. The source code is available from https://github.com/saaadhu/choreo/.

If you've used macros in Visual Studio or in any other tool, you probably need no further explanation, but for the others, here's what MSDN says a macro is:

"A macro is a series of commands and instructions that you group together as a single command to accomplish a task automatically. Macros allow you to automate repetitive actions."

Choreo macros are just IronPython functions that have access to DTE, the top level automation object exposed by Visual Studio. Choreo discovers the macro functions at start up and creates commands for them inside AVR Studio, and these commands can then be executed through the Command Window, or through shortcut keys (after binding them to keystrokes, of course).

Here's a macro that writes the current date/time at the current cursor position.

from System import DateTime

def InsertDateTime():
	dte.ActiveDocument.Selection.Text = DateTime.Now.ToString()

To make this macro discoverable by Choreo, save the above code in a .py file (say editor.py), and drop the file inside %localappdata%\Atmel\AvrStudio\5.0\Extensions\Senthil Kumar Selvaraj\Choreo\0.1\Macros. If you're running AVR Studio, you can do Tools -> Refresh Choreo to make Choreo pick up new macros. Choreo creates commands in the format Choreo.<python_file_name>.<python_function_name> for whatever functions it discovers.

You can then bind your macro to a keystroke using the Tools -> Options dialog.

Type the keystroke, say Ctrl followed by ., in the Press shortcut keys box, click Assign and then OK. You should now be able to execute your macro with the configured keystroke when typing in the editor.

Choreo automatically picks up code changes in existing commands, so you don't have to explicitly issue a refresh (Tools -> Refresh Choreo). However, new functions in existing files, as well as new files won't be picked up automatically, so an explicit refresh is needed there.

Record and Replay support is still missing, and commands can't take parameters at the moment, but otherwise, it works pretty well.

Enjoy!

If you’ve programmed in any managed language in Visual Studio, you’d have definitely seen those nagging red squiggles that appear as you type, telling you just how dumb you are, every time you pause typing. Might not be everyone’s cup of tea, but I’ve personally found them very useful; they save quite a few compile-groan-swear-fix-compile cycles.

So I decided to write a similar “squiggly generator” for AVR Studio 5, the product that I’m working on with a bunch of other guys. Naggy is what I call it, and it is a VSIX extension that installs into AVR Studio 5. You can download it from here. Naggy is open source (MIT License), and the source code is available at https://github.com/saaadhu/naggy/.

Mandatory screenshot:

Naggy uses Clang under the hood to do the actual source code analysis, and uses the diagnostic information provided by it to tag appropriate text spans in the editor with the red squiggles. The diagnostic message is shown as a tooltip when you hover the mouse over the squiggle. In a sense, Naggy is little more than a wrapper for Clang – it visually shows what Clang finds. There are a few tricky things to deal with though, like reading toolchain and compiler flags from AVR Studio and passing them on to Clang.

As its version number (0.1) indicates, it is still a very raw product (see issues). In case you run into problems, you can always do Tools –> Extension Manager –> Naggy –> Disable (or Uninstall).

Any feedback is good feedback, so good or bad, do let me know about it. I’m planning to also integrate the static analysis features offered by Clang into AVR Studio, and it all depends on whether people actually find this useful.

There was an interesting problem at work recently, where I basically wanted to generate a whole lot of C language expressions for testing purposes. Rather than generating them by hand, I wondered if I could somehow feed the C language grammar into a program and get most, if not all, possible valid expressions out of it. Of course, I knew that a syntactically valid program might not necessarily by semantically valid, but I figured that I could eliminate or transform those cases manually.

So off I went, using this problem as an opportunity to write some Ruby code. The end result is Ramble, a Ruby program that given a grammar like

start : paragraphs;
paragraphs
: paragraph
| paragraph LINEBREAK LINEBREAK paragraphs
;
paragraph
: sentence
| sentence paragraph
;
sentence
: subject verb object PERIOD
;
subject
: CAT
| DOG
| PONY
;
verb
: EATS
| GULPS
| SWALLOWS
;
object
: HAY
| FOOD
| MILK
;

and some helper code to "stringify" terminal symbols like CAT, DOG and PONY, will generate text like

Cat swallows hay. Pony swallows milk. Dog eats hay. Dog swallows hay. Pony gulps milk. Pony gulps hay.

Pony gulps food. Dog eats food.

Dog gulps milk. Pony eats food. Cat swallows hay.

Cool, don't you think? You can browse the source code online at github here, but the general idea is to parse the grammar file (I'm assuming a simplified yacc format), generate an abstract tree of productions, and then traverse the tree to generate text (with a callback to get text for terminal symbols). Pretty similar to what a compiler does, except that instead of accepting source code as input, it takes a grammar file, and instead of emitting code, it spits out sentences in the language.

I used Treetop to parse the grammar, and wrote some hairy recursive code to filter and operate on the generated tree. Now the grammar could be left recursive i.e., it could have a rule like

A : A | B

and a generator that naively tries to generate every possible sentence recursively would quickly overflow the stack and die. I chose to randomly pick one of the two (or more) options, hoping that running the program repeatedly will give me a variety of sentences.

Now for the bad part - I tried running this on a subset of the C grammar, and it crashed with a stack overflow error. With rules that were transitively recursive, the random choice option wasn't enough to control the recursion before it blew the stack.

And boy, was I wrong about filtering out syntactically correct but semantically valid expressions! The few times the program completed without crashing, the emitted expressions were just total rubbish semantically - there was no way I could have manually massaged those into shape.

It was an interesting exercise though, and it helped scratch two itches at the same time - dabbling with language stuff, and writing serious Ruby code (I'm a Ruby newbie).

The other day, I logged on to my home network and entered gmail.google.com from Firefox. Imagine my surprise when I got a “landing” page with a bunch of URLs, rather than the usual spartan GMail page.

One terrifying thought followed the other. Was my machine infected? Was my WiFi router compromised? Maybe the ASDL modem? Worse, how long had it gone unnoticed?

Ok, time to take a deep breath and start isolating the problem, I figured.

I logged on from an alternate machine and found it worked fine there. I did a nslookup from the command prompt on my machine on gmail.google.com, and got back an IP address in the Google domain.

DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  218.248.241.5

Name:    www3.l.google.com
Address:  209.85.231.100
Aliases:  gmail.google.com

Maybe some browser extension was causing the problem? To isolate that, I tried navigating to the same URL from Chrome and IE. No luck there – I was still getting the wrong page.

Must be some kind of virus then, I figured. I wanted to see the IP address the browser was navigating to, so I fired up Wireshark and watched the traffic. Not surprisingly, I found this

42    11.621942    192.168.1.100    61.1.96.69    DNS    Standard query A gmail.google.com
43    11.659343    61.1.96.69    192.168.1.100    DNS    Standard query response A 64.95.64.197

The IP address in the response (64.95.64.197) ,when run through a reverse lookup, resolved to a domain name that definitely wasn’t Google’s – the name suggested some kind of ad network. But wait, where did the IP address come from? 61.1.96.69. That was the DNS server the request was sent to, and as the Wireshark log shows, it obviously responded with the wrong IP address for the domain name.

Ok, so my browser was sending a request to a hacked DNS server, and that was why I was getting the wrong page. But where did 61.1.96.69 (the DNS server) come from? And why does nslookup use a different DNS server (218.248.241.5)? Time to check the wireless router/ADSL modem then, I thought.

And sure enough, this is what I found in the status page.

DNS 1:     218.248.241.5              
DNS 2:     61.1.96.69

Was my router compromised to use a spiked DNS server? A reverse DNS lookup on 61.1.96.69 showed that it was in fact in the same domain as my ISP (BSNL/Sancharnet). So no, it wasn’t my router – BSNL was giving me the two IP addresses, and the second one, which my browser had used, was poisoned to return wrong IP addresses for requests for the gmail.google.com domain name.

This was the first time I’d encountered DNS cache poisoning, and it is easy to see how dangerous it can be. SSL of course would save the day for secure websites – the fake website won’t be able to produce a valid certificate claiming to be the original website. But what about the millions of websites without a certificate? And how many internet users actually know about https versus http?

Terrifying, ain’t it?

I fixed the problem by forcing my router to use Google's DNS servers rather than BSNL’s. Rather ironic, considering the trigger was incorrect lookup of one of Google’s own subdomains.

Because your application can crash otherwise, that’s why.

While it’s always a good idea to explicitly dispose everything that is disposable, we can usually get away without disposing UI controls because

a. Complex controls aren’t created often – a single instance is often reused.

b. The finalizer kicks in and saves the day.

If you’re creating multiple instances of a System.Windows.Forms.DataGridView, however, watch out, because (b) doesn’t happen at all if you don’t call Dispose or otherwise cause the control to be destroyed.

When debugging a software crash dump recently, I found that the crash occurred because there was an exception when attempting to show the exception handler dialog, and that was because the application had run out of window handles. Being a managed application, that meant that the application was holding to way more UI control instances than is normal.

A quick look at the objects in the heap showed tons of instances of a certain type of control. Looking at the code, it was clear that while the control is created often, there is no code that holds on to the instances indefinitely – each new instance clears all references to the old instance. Sure, the previous instance was not being disposed, but I skipped over that, assuming that the problem must be someone holding on to the instances, or otherwise the finalizer would have cleared things up.

Dumping the gcroots for a random sample of those objects showed a common chain.

   1: DOMAIN(002869F0):HANDLE(Pinned):2013e8:Root:  02ed5250(System.Object[])->

   2:   01efce64(System.Collections.Generic.Dictionary`2[[System.Object, mscorlib],[System.Collections.Generic.List`1[[Microsoft.Win32.SystemEvents+SystemEventInvokeInfo, System]], mscorlib]])->

   3:   01efd3d8(System.Collections.Generic.Dictionary`2+Entry[[System.Object, mscorlib],[System.Collections.Generic.List`1[[Microsoft.Win32.SystemEvents+SystemEventInvokeInfo, System]], mscorlib]][])->

   4:   01efe464(System.Collections.Generic.List`1[[Microsoft.Win32.SystemEvents+SystemEventInvokeInfo, System]])->

   5:   01fed344(System.Object[])->

   6:   020253f4(Microsoft.Win32.SystemEvents+SystemEventInvokeInfo)->

   7:   020253d4(Microsoft.Win32.UserPreferenceChangedEventHandler)->

   8:   020242d0(System.Windows.Forms.DataGridView)

The delegate type in line 7 was a dead giveaway; looking for references to that type using Reflector showed the following code inside DataGridView’s OnHandleCreated method.

   1: protected override void OnHandleCreated(EventArgs e)

   2: {

   3:    // A bunch of other code

   4:    SystemEvents.UserPreferenceChanged += new UserPreferenceChangedEventHandler(this.OnUserPreferenceChanged);

   5: }

That line right there is why the crash happened.

When the DataGridView control’s window handle gets created, it subscribes to an event from a static class (SystemEvents.UserPreferenceChanged). It does unsubscribe from it in the OnHandleDestroyed method, but that gets called only if the control is properly disposed.

Now if you don’t call Dispose, the control remains subscribed to the event; that counts as a strong reference to the control, and it therefore cannot be garbage collected and finalized. Which means that the control and all its associated resources are not going to be released until the application shuts down (or the AppDomain unloads).

In the crashing application, there were other controls subscribed to events from the DataGridView, so it in turn prevented garbage collection and finalization of those controls, and eventually, they used up a really large number of window handles, causing the application to crash when it tried to create a new control.

It’s rather strange if you think about it; the finalizer is supposed to be mechanism to cleanup if Dispose is not called on an object, but for a DataGridView, the finalizer won’t run unless you call Dispose, as otherwise a strong reference to the object will exist. It won’t run after you call Dispose either – the Dispose implementation calls SuppressFinalize(this).

In the previous post, we saw that linking a C++ static library compiled with /EHs to a mixed mode application prevented the destructor from running when an exception is thrown. Here’s the sample project that demonstrates the behavior, in case you aren’t convinced.

This is the code inside the library.

   1: C::C()

   2: { 

   3:     cout << "Constructed" << endl; 

   4: }

   5:  

   6: C::~C()

   7: { 

   8:     cout << "Destructed" << endl; 

   9: }

  10:  

  11: void SomeFunc()

  12: {

  13:     C c;

  14:     throw std::exception("Gone");

  15: }

And this is the code consuming the library.

   1: int main(array<System::String ^> ^args)

   2: {

   3:     try

   4:     {

   5:         SomeFunc();

   6:     }

   7:     catch(Exception ^e)

   8:     {

   9:         Console::WriteLine(e->ToString());

  10:     }

  11:     return 0;

  12: }

OK, so where do we start? First, let’s see if this happens in the normal (no exception) case as well. Commenting out line 14 in the first code snippet and running the code should show us that. Try it out, and you’ll see that the destructor runs now. So the problem must somehow be related to destruction during exception handling. But we have asked the compiler to emit code for C++ exception handling (/EHs), and we are throwing plain C++ exceptions, so why is this happening?

To understand that, we’ll have to dig deeper into how the the VC++ compiler and the CRT implement exceptions. Under the hood, the VC++ compiler uses Win32 SEH (Structured Exception Handling) to implement C++ exceptions. Matt Pietrek’s article explains how SEH works – do give it a quick read. The real quick summary of how it works is this – every function, on entry, creates an exception registration record on the stack that contains the address of the current function’s SEH exception handler and a pointer to the previous exception registration record, and writes the address of the record to the current thread’s TIB (Thread Information Block). When an SEH exception occurs, the OS walks through the registered records  once to determine the handler for the exception, and again to allow cleanup code to run. The handler knows why it was called by looking at the exception record that is passed to it – it has a flag that specifies that information.

We’ll look at how the C++ compiler uses SEH by firing up Windbg and disassembling SomeFunc.

   1: 0:000> x *!SomeFunc

   2: 011015e0 CliConsoleApp!SomeFunc (void)

   3:  

   4: 0:000> u 011015e0 011016ff

   5: 011015e0 6aff            push    0FFFFFFFFh

   6: 011015e2 6815361001      push    offset CliConsoleApp!CorExeMain+0xa5 (01103615)

   7: 011015e7 64a100000000    mov     eax,dword ptr fs:[00000000h]

   8: 011015ed 50              push    eax

   9: 011015ee 83ec10          sub     esp,10h

  10: 011015f1 a118001101      mov     eax,dword ptr [CliConsoleApp!__security_cookie (01110018)]

  11: 011015f6 33c4            xor     eax,esp

  12: 011015f8 50              push    eax

  13: 011015f9 8d442414        lea     eax,[esp+14h]

  14: 011015fd 64a300000000    mov     dword ptr fs:[00000000h],eax

  15: 01101603 a134401001      mov     eax,dword ptr [CliConsoleApp!_imp_?endlstdYAAAV?$basic_ostreamDU?$char_traitsDstd (01104034)]

  16: 01101608 8b0d70401001    mov     ecx,dword ptr [CliConsoleApp!_imp_?coutstd (01104070)]

  17: 0110160e 50              push    eax

  18: 0110160f 68c0451001      push    offset CliConsoleApp!`string' (011045c0)

  19: 01101614 51              push    ecx

  20: 01101615 e8a6010000      call    CliConsoleApp!std::operator<<<std::char_traits<char> > (011017c0)

  21: 0110161a 83c408          add     esp,8

  22: 0110161d 8bc8            mov     ecx,eax

  23: 0110161f ff1540401001    call    dword ptr [CliConsoleApp!_imp_??6?$basic_ostreamDU?$char_traitsDstdstdQAEAAV01P6AAAV01AAV01ZZ (01104040)]

  24: 01101625 8d542404        lea     edx,[esp+4]

  25: 01101629 c744241c00000000 mov     dword ptr [esp+1Ch],0

  26: 01101631 52              push    edx

  27: 01101632 8d4c240c        lea     ecx,[esp+0Ch]

  28: 01101636 c7442408d8451001 mov     dword ptr [esp+8],offset CliConsoleApp!`string' (011045d8)

  29: 0110163e ff15ac401001    call    dword ptr [CliConsoleApp!_imp_??0exceptionstdQAEABQBDZ (011040ac)]

  30: 01101644 6888ea1001      push    offset CliConsoleApp!_TI1?AVexceptionstd (0110ea88)

  31: 01101649 8d44240c        lea     eax,[esp+0Ch]

  32: 0110164d 50              push    eax

  33: 0110164e e8171f0000      call    CliConsoleApp!CxxThrowException (0110356a)

The FS register holds the address of the TIB, so to figure out where our exception handler is, we only need to find out where the FS register is being written into. That’s happening on line 14, and you can see that before that, the compiler emits code to push the the address of a function (on line 6) and the previous exception registration record (on line 7,8).  That’s the setup we were looking for, so the function address pushed must be our SEH exception handler. Let’s go ahead and disassemble that.

   1: 0:000> u 01103615

   2: CliConsoleApp!CorExeMain+0xa5:

   3: 01103615 8b542408        mov     edx,dword ptr [esp+8]

   4: 01103619 8d42f0          lea     eax,[edx-10h]

   5: 0110361c 8b4aec          mov     ecx,dword ptr [edx-14h]

   6: 0110361f 33c8            xor     ecx,eax

   7: 01103621 e81ee5ffff      call    CliConsoleApp!__security_check_cookie (01101b44)

   8: 01103626 b860eb1001      mov     eax,offset CliConsoleApp!_TI1?AVexceptionstd+0xd8 (0110eb60)

   9: 0110362b e934ffffff      jmp     CliConsoleApp!_CxxFrameHandler3 (01103564)

Control jumps to a compiler generated function (_CxxFrameHandler3), and following along the jumps takes us to MSVCR90!_CxxFrameHandler3, the CRT exception handler. That function in turn calls another CRT function, which examines the parameters passed to it. One of the parameters is the exception record, and here’s how it looks.

   1: typedef struct _EXCEPTION_RECORD {

   2:  DWORD ExceptionCode;

   3:  DWORD ExceptionFlags;

   4:  struct _EXCEPTION_RECORD *ExceptionRecord;

   5:  PVOID ExceptionAddress;

   6:  DWORD NumberParameters;

   7:  DWORD ExceptionInformation[EXCEPTION_MAXIMUM_PARAMETERS];

   8:  }  EXCEPTION_RECORD;

   1: 0:000> dd 0012ebd4 

   2: 0012ebd4  e06d7363 00000001 00000000 752f9617

e06d7363 is the exception code for C++ applications, and exception flags is 1. The CRT function first verifies whether it’s a C++ exception by looking for that code. It then looks at the flag to figure out whether it should run the stack unwinding code. Apparently, 1 is the flag value when the OS makes the first pass over the exception registration chain, looking for a handler. We don’t have a catch block in Somefunc, so the function just returns without doing anything significant.

We’ll continue execution and wait for our handler to be called again the second time, hopefully asking it to unwind. Sure enough, control reaches the internal CRT function again, let’s see what the exception record contains this time.

   1: 0:000> dd 0012e5c8

   2: 0012e5c8  c0000027 00000002 00000000 68051870

Oops – it’s not a C++ exception anymore – the error code is c0000027 and not e06d7363. So even though the handler is called to ask it to unwind, the CRT function doesn’t run unwinding logic because it’s not a C++ exception.

That explains why the destructor did not run – running the destructor code is part of the unwinding logic. OK, but who changed the exception code? Let’s look at the call stack.

   1: 0:000> kb

   2: ChildEBP RetAddr  Args to Child              

   3: WARNING: Stack unwind information not available. Following frames may be wrong.

   4: 0012e51c 70aad82d 0012e5c8 0012ef84 0012e674 MSVCR90!_CxxExceptionFilter+0x707

   5: 0012e558 76f665f9 0012e5c8 0012ef84 0012e674 MSVCR90!_CxxFrameHandler3+0x26

   6: 0012e57c 76f665cb 0012e5c8 0012ef84 0012e674 ntdll!RtlRaiseStatus+0xb4

   7: 0012e944 68051870 0012f04c 6806c600 00000000 ntdll!RtlRaiseStatus+0x86

   8: 0012e968 680ccebd 0012f04c 6806c600 00000000 mscorwks+0x1870

   9: 0012ea84 680cd4f0 0012ebd4 0012f04c 0012ebf4 mscorwks!GetMetaDataInternalInterface+0x946a

  10: 0012eac4 680cd675 0012ebd4 0012f04c 0012eba8 mscorwks!GetMetaDataInternalInterface+0x9a9d

  11: 0012eae8 76f665f9 0012ebd4 0012f04c 0012ebf4 mscorwks!GetMetaDataInternalInterface+0x9c22

  12: 0012eb0c 76f665cb 0012ebd4 0012f04c 0012ebf4 ntdll!RtlRaiseStatus+0xb4

  13: 0012ebbc 76f66457 0012ebd4 0012ebf4 0012ebd4 ntdll!RtlRaiseStatus+0x86

  14: 0012ef28 70aadbf9 e06d7363 00000001 00000003 ntdll!KiUserExceptionDispatcher+0xf

  15: 0012ef60 01101653 0012ef78 0110ea88 f8a58fa0 MSVCR90!CxxThrowException+0x48

We see the CRT throwing the exception, but see who caught and triggered the second pass of the SEH handlers – mscorwks.dll, the core CLR engine. It apparently modified the SEH exception’s code when it was called as one of the handlers.

So this is what happened - the C++ code code raised an SEH exception with the error code for C++ exceptions (e06d7363). When the OS walked the exception handler chain, it asked the C++ exception handler whether it would handle the exception, and the exception handler said no, because there is no catch block in our C++ code. The next handler in the chain is the one installed by the CLR. It says yes, it will handle the exception, and in the process, modifies the exception code from e06d7363 to c0000027. When the OS calls the handlers again to ask them to unwind, the C++ handler doesn’t unwind because it doesn’t recognize it as a C++ exception from the error code. And that’s why the destructor did not run.

Why does the CLR exception handler modify the exception code? As this blog post says, it’s because managed exceptions also use SEH under the hood, and the CLR doesn’t want unknown exception codes to be passed to its handlers, for whatever reason. Except for SEH exceptions that it knows about, like access violations, it maps all other unmanaged exceptions to the same error code (c0000027) and treats them as general SEH exceptions.

How do we fix it? Based on what we know, simply adding a catch (…) { throw; } inside SomeFunc should fix the problem – the C++ exception handler will now say yes when asked if it can handle the exception. It of course wouldn’t mangle the exception code, so when the same handler is called for unwinding, it will run the destructor properly. The CLR exception handler will be involved only when the exception is re-thrown, but our destructor would have already run by then.

The right fix though is to compile the library with the /EHa option, which tells the compiler to emit code to handle both C++ exceptions and other SEH exceptions.  That way, the handler will run the stack unwind code when called during the second pass, C++ exception or not. In fact, the compiler doesn’t allow you to compile mixed mode code with /EHs – it would complain that the /clr and /EHs flags are incompatible. Unfortunately, neither the compiler nor the linker complain when linked against code compiled with /EHs – probably because they don’t know about that fact. There is some cost to compiling with /EHa though, your exception handlers would run in cases where they wouldn’t have run before, and I’d guess it also affects compiler optimizations to some extent.

We actually ran into this problem when using mixed mode code linked with Omni ORB, an open source native library for CORBA. It’s compiled with /EHs, and as you’d know by now, that caused some serious resource leak issues when there were exceptions involved. It took some serious debugging to narrow down the problem; missing C++ destructor calls don’t happen everyday, after all.

Consider this piece of C++ code.

   1: using namespace std;

   2:  

   3: class C

   4: {

   5: public:

   6:     C() 

   7:     { 

   8:         cout << "Constructed"; 

   9:     }

  10:     ~C() 

  11:     { 

  12:         cout << "Destructed"; 

  13:     }

  14: };

  15:  

  16: void SomeFunc()

  17: {

  18:     C c;

  19:     throw std::exception("Gone");

  20: }

If you know any C++ at all, you’ll know that when SomeFunc returns, both “Constructed” and “Destructed” will be printed to the console. That is because C++ guarantees that the destructor of an object created on the stack will always run when control leaves the scope, no matter what, and RAII depends on that fact.

You put all this code is in a static library, say PureCPP.lib, and you compile it with the /EHs option, because you want to use C++ exceptions.

You then write a native application to consume this library, statically link to it, and everything works great.

One day, you wake up and realize you’ll have to try out this .NET stuff that everyone is talking about. You discover that there’s this language called C++/CLI that’s great for interfacing with native code. So you fire up VS, create a CLR console application that calls SomeFunc, and link PureCPP.lib against it.

Just when you’re wondering how easy things turned out to be, you notice something strange. There’s something missing in the console output. When you figure out what’s missing, your jaw hits the ground. Mine did too, when I realized that it was the “Destructed” part that was missing. Which means the impossible just happened - the destructor for class C did not run.

What followed was a long and exciting journey into the world of SEH (Structured Exception Handling), exception codes and exception propagation and handling by the CLR versus C++. All that in the next part – stay tuned.

This piece of code results in a warning.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

   6:         string myName = "Senthil";

   7:         Console.WriteLine(obj == myName);

   8:     }

   9: }

The compiler says “warning CS0252: Possible unintended reference comparison; to get a value comparison, cast the left hand side to type 'string'” on line 7. The compiler is right; it has no way of knowing that obj is actually a string. It infers the static type of obj to be System.Object, and if you’ve done any programming in C# at all, you’ll know that the default behavior of the == operator is reference comparison. This code is unintentionally comparing reference values of two local variables, and that’s what the compiler is warning about.

Yet try running the code, and you’ll see that it works correctly; it prints True. Is the compiler wrong?

No. The code appears to work fine because string interning masks the problem. The C# compiler figures that there are two constant strings in the program, and both are equal, so instead of creating two distinct string objects, it creates only one. With strings being immutable, reusing string objects for identical strings is safe. And that’s why this code works, both obj and myName refer to the same string object, which means reference comparison will succeed.

Now that we know why it’s working, it’s easy to prove that the compiler warning is right with a small code change.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

   6:  

   7:         string myName1 = "Sen";

   8:         string myName2 = "thil";

   9:         string myName = myName1 + myName2;

  10:         Console.WriteLine(obj == myName);

  11:     }

  12: }

Fixing the warning is a mere matter of casting obj to the correct type (string) and then doing the equality check.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

   6:  

   7:         string myName1 = "Sen";

   8:         string myName2 = "thil";

   9:         string myName = myName1 + myName2;

  10:         Console.WriteLine(((string)obj) == myName);

  11:     }

  12: }

Or we could use the Equals method, which is virtual and won’t be fooled by the compile time type.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

   6:  

   7:         string myName1 = "Sen";

   8:         string myName2 = "thil";

   9:         string myName = myName1 + myName2;

  10:         Console.WriteLine(obj.Equals(myName));

  11:     }

  12: }

Both of them print True, which is what we want.

Mixing SetEnvironmentVariable and getenv is asking for trouble, as we recently found to our dismay (and exasperation!). It took some serious debugging to figure out the actual problem – let’s see if you can figure it out.

Here’s some C++/CLI code that uses getenv to read the value of an environment variable and print it to the console.

   1: public ref class EnvironmentVariablePrinter

   2: {

   3:     public:

   4:         void PrintEnv(String ^key)

   5:         {

   6:              IntPtr ptr = Marshal::StringToHGlobalAnsi(key);

   7:              const char *ckey = (const char *)ptr.ToPointer();

   8:              const char *cval = getenv(ckey);

   9:              string val = cval == NULL ? "" : string(cval);

  10:  

  11:              cout << "Key is " << ckey << " and value is " << val;

  12:  

  13:              Marshal::FreeHGlobal(ptr);    

  14:         }

  15: };

PrintEnv merely converts the .NET string into an unmanaged one and calls getenv, printing the returned value to the console.

And here’s some C# code that tests the class.

   1: class Test

   2: {

   3:     const string key = "MyKey";

   4:     public static void Main()

   5:     {

   6:         Environment.SetEnvironmentVariable(key, "MyValue");

   7:         PrintValue();

   8:     }

   9:  

  10:     private static void PrintValue()

  11:     {

  12:         EnvironmentVariablePrinter er = new EnvironmentVariablePrinter();

  13:         er.PrintEnv(key);

  14:     }

  15: }

The code uses System.Environment.SetEnvironmentVariable to set the value of the variable and then calls the C++/CLI code to verify that it prints the correct value. And of course, being written in different languages, the two pieces of code must reside in different projects, say CPlusPlusLib.dll and ConsoleApplication.exe, with the latter referencing the former.

No surprises here - this works as expected and prints “Key is MyKey and value is MyValue”.

However, a seemingly harmless change breaks the code big time.

   1: class Test

   2: {

   3:     const string key = "MyKey";

   4:     public static void Main()

   5:     {

   6:         Environment.SetEnvironmentVariable(key, "MyValue");

   7:         EnvironmentVariablePrinter er = new EnvironmentVariablePrinter();

   8:         er.PrintEnv(key);

   9:     }

  10: }

All I’ve done is inlining of PrintValue, yet running this code prints “Key is MyKey and value is” – getenv is now returning NULL instead of “MyValue”.

It gets even more interesting.

   1: class Test

   2: {

   3:     const string key = "MyKey";

   4:     public static void Main()

   5:     {

   6:         Environment.SetEnvironmentVariable(key, "MyValue");

   7:         EnvironmentVariablePrinter er = null;

   8:         PrintValue(); 

   9:     }

  10:  

  11:     private static void PrintValue()

  12:     {

  13:         EnvironmentVariablePrinter er = new EnvironmentVariablePrinter();

  14:         er.PrintEnv(key);

  15:     }

  16: }

This doesn’t work either – the mere declaration of EnvironmentVariablePrinter inside Main makes getenv return NULL for “MyKey”. This can’t be good, can it?

Actually yes, because that is a valuable clue – it means the change in behavior has something to do with JITting. As long as all code that references EnvironmentVariablePrinter is in a separate method, everything works fine (in debug mode, atleast). Things start going south when any such code is in Main itself.

What would the JITter do differently in the two scenarios? Load the assembly containing EnvironmentVariablePrinter (CPlusPlusLib.dll) at different times, of course. When all code referencing EnvironmmentVariablePrinter is inside PrintValue, it will have to load the DLL only when JITting PrintValue, whereas in the other case, it will have to load it when JITting Main. JITting Main obviously occurs much before JITting PrintValue, so DLL load time (relative to other code) is one big difference between the two scenarios that occurs because of JITting.

Why would loading CPlusPlusLib.dll a little early make getenv return NULL?

To understand that, you’ll first have to know how the getenv function works. Windows has APIs to set and get environment variables (SetEnvironmentVariable/GetEnvironmentVariable), and the .NET method P/Invokes into the Windows API to set and get values. getenv, on the other hand, is a CRT function, and does not delegate to the Windows API. Instead, the CRT gets all environment variables and their values when it is starting up (using the GetEnvironmentStrings Windows API), and copies them into its own data structures (MSVCR80!environ). getenv then works on the copied data from then on.

Now do you see the problem? When CPlusPlusLib.dll is loaded early (when JITting Main), the CRT also gets loaded as one of its dependencies, and the startup code that copies environment variables runs right away. At that point, Main hasn’t even been JITted yet, so there’s no way our call to System.Environment.SetEnvironmentVariable could have run by that time. And when it actually runs, it’s too late – the CRT environ block would have been updated much earlier, and calling the Windows API’s SetEnvironmentVariable wouldn’t have any effect on the cached values. When getenv runs, it looks in the cached values and returns NULL.

It’s easy to see why it works in the first case now – CRT loading occurs when JITting PrintValue, and that occurs after our call to SetEnvironmentVariable has executed. Which means that when it calls GetEnvironmentStrings as part of startup, it gets the variable (and its value) that we just set.

Nasty, ain’t it? The actual scenario was a lot more messy – things suddenly stopped working when we linked to a DLL ported to VS2008. We actually figured the problem backwards – we first saw that the CRT load time was different, theorized how getenv works, verified the theory by stepping through the assembly code and looking at the environ block, and once we realized the problem, figured out what was causing early loading of the CRT. Windbg was awesome for debugging this - things would have been very difficult if not for sxe ld:MSVCR90 and x MSVCR80!environ.

The fix was rather simple - in our case, we merely had to move code that set environment variable before CRT load. There’s another twist though; mscorwks.dll, which is the heart of the CLR, loads MSVCR80 when it loads, and you can’t set your environment variables before that, not from managed code anyway. Fortunately, in our case, the getenv call is from a library that links to MSVCR90, so as long as we set the environment variable before that version of the CRT loads, we’re good to go. Until the CLR gets linked to MSVCR90, anyway :).

Attach to what? To the process you want to debug, of course.

How many developers attach to and debug arbitrary processes running on their machines? Very few, I’d imagine. And I’d think that even those few people typically prefer Windbg or an equivalent debugger to the one supplied with Visual Studio.

Which means that aside from ASP .NET developers, almost all of us using Visual Studio attach to the application that we are working on and nothing else. To attach

1. You summon the “Attach to Process” dialog by hitting Ctrl + Alt + P. Or if you are a mouse person, you go to Debug –> Attach to Process.

2. You search for your application in the list of processes shown and select it.

3. You optionally change some settings and then hit OK.

The second step can be particularly annoying, especially if the name of your application starts with a particularly common letter that occurs in the latter half of the English alphabet (it’s a tie between ‘s’ and ‘v’ on my machine). Even otherwise, if you’ve worked on the application for a significant amount of time, the keystrokes to select it becomes part of muscle memory (down arrow, down arrow, Enter, for e.g.,), and you occasionally end up attaching to the wrong application because some other process sneaked in. Surely there must be a better way?

Enter JustAttach – a macro that does just that. It finds out the output file path of the startup project of your solution and automatically attaches to it.

The full macro code is at the end of the blog post. You can also download, unzip, open Macro Explorer (View –> Other Windows –> Macro Explorer) and select Load Macro Project to start using it right away.

Do your fingers a favor by binding the command to a VS shortcut (Tools –> Options –> Keyboard, type JustAttach in the textbox and choose a shortcut like Ctrl + Alt+ Y) ; your fingers will thank you for it :).

   1: Imports System

   2: Imports EnvDTE

   3: Imports EnvDTE80

   4: Imports EnvDTE90

   5: Imports System.Diagnostics

   6:  

   7: Public Module SenthilMacros

   8:     Public Sub JustAttach()

   9:         Dim solutionBuild As SolutionBuild = DTE.Solution.SolutionBuild

  10:         Dim startupProjectName As String = solutionBuild.StartupProjects(0)

  11:  

  12:         If String.IsNullOrEmpty(startupProjectName) Then

  13:             MsgBox("Could not attach because the startup project could not be determined", MsgBoxStyle.Critical, "Failed to Attach")

  14:             Return

  15:         End If

  16:  

  17:         Dim startupProject As Project = FindProject(startupProjectName.Trim())

  18:         Dim outputFilePath As String = FindOutputFileForProject(startupProject)

  19:  

  20:         If String.IsNullOrEmpty(outputFilePath) Then

  21:             MsgBox("Could not attach because output file path for the startup project could not be determined", MsgBoxStyle.Critical, "Failed to Attach")

  22:             Return

  23:         End If

  24:  

  25:         Attach(outputFilePath)

  26:     End Sub

  27:     Sub Attach(ByVal file As String)

  28:         Dim process As EnvDTE.Process

  29:  

  30:         For Each process In DTE.Debugger.LocalProcesses

  31:             If process.Name = file Then

  32:                 process.Attach()

  33:                 Return

  34:             End If

  35:         Next

  36:  

  37:         MsgBox("Could not attach because " + file + " is not found in the list of running processes", MsgBoxStyle.Critical, "Failed to Attach")

  38:     End Sub

  39:  

  40:     Function FindProject(ByVal projectName As String) As Project

  41:         Dim project As Project

  42:         For Each project In DTE.Solution.Projects

  43:             If project.UniqueName = projectName Then

  44:                 Return project

  45:             End If

  46:         Next

  47:     End Function

  48:     Function FindOutputFileForProject(ByVal project As Project) As String

  49:         Dim fileName As String = project.Properties.Item("OutputFileName").Value.ToString()

  50:         Dim projectPath As String = project.Properties.Item("LocalPath").Value.ToString()

  51:  

  52:         Dim config As Configuration = project.ConfigurationManager.ActiveConfiguration

  53:         Dim buildPath = config.Properties.Item("OutputPath").Value.ToString()

  54:  

  55:         If String.IsNullOrEmpty(fileName) Or String.IsNullOrEmpty(projectPath) Then

  56:             Return ""

  57:         End If

  58:  

  59:         Dim folderPath As String = System.IO.Path.Combine(projectPath, buildPath)

  60:         Return System.IO.Path.Combine(folderPath, fileName)

  61:  

  62:     End Function

  63: End Module

  64:  

The previous post discussed having anonymous methods as event handlers and ended with a question – why doesn’t unsubscription work while subscription works out alright?

Vivek got the answer spot on – the way the C# compiler handles and translates anonymous methods is the reason.

Here’s the code involved.

   1: public void Initialize()

   2: {

   3:     control.KeyPressed += IfEnabledThenDo(control_KeyPressed);

   4:     control.MouseMoved += IfEnabledThenDo(control_MouseMoved);

   5: }

   6:  

   7: public void Destroy()

   8: {

   9:     control.KeyPressed -= IfEnabledThenDo(control_KeyPressed);

  10:     control.MouseMoved -= IfEnabledThenDo(control_MouseMoved);

  11: }

  12:  

  13: public EventHandler<Control.ControlEventArgs> IfEnabledThenDo(EventHandler<Control.ControlEventArgs> actualAction)

  14: {

  15:     return (sender, args) => { if (args.Control.Enabled) actualAction(sender, args); };

  16: }

The compiler translates IfEnabledThenDo into this

   1: public EventHandler<Control.ControlEventArgs> IfEnabledThenDo(EventHandler<Control.ControlEventArgs> actualAction)

   2: {

   3:     <>c__DisplayClass1 CS$<>8__locals2 = new <>c__DisplayClass1();

   4:     CS$<>8__locals2.actualAction = actualAction;

   5:     return new EventHandler<Control.ControlEventArgs>(CS$<>8__locals2.<IfEnabledThenDo>b__0);

   6: }

Now the problem should be fairly obvious – every time the function is called, a new object gets created, and the event handler returned actually refers to a method (<IfEnabledThenDo>b__0) on the new instance. And that’s what breaks unsubscription. –= will not remove a delegate of a different instance of the same class from the invocation list – if it did, the consequences would not be pleasant if multiple instances of the same class subscribe to an event.

But why does the compiler translate our lambda expression this way? Raymond Chen has a great blog post explaining why, but the short answer is that it is needed to “hold” actualAction (the method parameter to IfEnabledThenDo) so that it is available when the event handler actually executes.

Now that we know why, the way to get around this issue is to cache the delegate instance returned by IfEnabledThenDo and use the same instance for subscription and unsubscription.

   1: EventHandler<Control.ControlEventArgs> keyPressed;

   2: EventHandler<Control.ControlEventArgs> mouseMoved;

   3:  

   4: public void Initialize()

   5: {

   6:    keyPressed = IfEnabledThenDo(control_KeyPressed);

   7:    mouseMoved = IfEnabledThenDo(control_MouseMoved);

   8:  

   9:    control.KeyPressed += keyPressed;

  10:    control.MouseMoved += mouseMoved;

  11: }        

  12:  

  13: public void Destroy()

  14: {

  15:    control.KeyPressed -= keyPressed;

  16:    control.MouseMoved -= mouseMoved;

  17: }

Knowing how things work under the hood has its advantages, I guess :)

PS : A very small syntactic change to the original example would have made the code work right away. If you’ve followed along this far, you should be able to figure out why.

   1: public void Initialize()

   2: {

   3:    actualAction = control_KeyPressed;

   4:    control.KeyPressed += IfEnabledThenDo();

   5: }        

   6:  

   7: public void Destroy()

   8: {

   9:     control.KeyPressed -= IfEnabledThenDo();

  10: }

  11:  

  12: EventHandler<Control.ControlEventArgs> actualAction;

  13: public EventHandler<Control.ControlEventArgs> IfEnabledThenDo()

  14: {

  15:     return (sender, args) => { if (args.Control.Enabled) actualAction(sender, args); };

  16: }

The syntactic sugar offered by anonymous methods makes them great candidates for writing event handlers; together with smart type inference, they reduce the amount of code written by an order of magnitude.

And that’s without considering the power offered by closures. With event handlers, closures allow you to kind of “stuff” extra parameters into the handler, without changing the actual number of formal parameters. This blog post shows a situation where an anonymous method acting as an event handler makes code simpler, and then goes on to show a gotcha with un-subscription and anonymous methods.

Here’s a simple Control class that fires a bunch of events.

   1: class Control

   2: {

   3:     public class ControlEventArgs : EventArgs

   4:     {

   5:         public Control Control {get;set;}

   6:     }

   7:  

   8:     public bool Enabled { get; set; }

   9:  

  10:     public event EventHandler<ControlEventArgs> KeyPressed;

  11:     public event EventHandler<ControlEventArgs> LeftButtonClicked;

  12:     public event EventHandler<ControlEventArgs> RightButtonClicked;

  13:     public event EventHandler<ControlEventArgs> MouseMoved;

  14: }

Let’s say you’re developing a GUI application with this class, and you want to handle events only if the originating control is visually enabled i.e., Enabled set to true. Pretty reasonable constraint, but tedious to implement, if you go the standard way of adding the check to the start of each of your event handlers.

   1: class GUIApp

   2: {

   3:     public void Initialize()

   4:     {

   5:         Control control = new Control();

   6:         control.KeyPressed += new EventHandler<Control.ControlEventArgs>(control_KeyPressed);

   7:         control.MouseMoved += new EventHandler<Control.ControlEventArgs>(control_MouseMoved);

   8:     }

   9:  

  10:     void control_MouseMoved(object sender, Control.ControlEventArgs e)

  11:     {

  12:         if (e.Control.Enabled)

  13:         {

  14:             /// 

  15:         }

  16:     }

  17:  

  18:     void control_KeyPressed(object sender, Control.ControlEventArgsEventArgs e)

  19:     {

  20:         if (e.Control.Enabled)

  21:         {

  22:             ///

  23:         }

  24:     }

  25: }

With an anonymous method, you could write a far more terse and easy to maintain version

   1: class GUIApp

   2: {

   3:     public void Initialize()

   4:     {

   5:         Control control = new Control();

   6:         control.KeyPressed += IfEnabledThenDo(control_KeyPressed);

   7:         control.MouseMoved += IfEnabledThenDo(control_MouseMoved);

   8:     }

   9:  

  10:     public EventHandler<Control.ControlEventArgs> IfEnabledThenDo(EventHandler<Control.ControlEventArgs> actualAction)

  11:     {

  12:         return (sender, args) => { if (args.Control.Enabled) { actualAction(sender, args); } };

  13:     }

  14:  

  15:     void control_MouseMoved(object sender, Control.ControlEventArgs e)

  16:     {

  17:         ///

  18:     }

  19:  

  20:     void control_KeyPressed(object sender, Control.ControlEventArgs e)

  21:     {

  22:         ///

  23:     }

  24: }

IfEnabledThenDo returns an anonymous function that first checks whether the control is enabled before calling the actual function. The code is much shorter, and the condition is checked only in one place, which makes it easy to modify or add additional logic without having to remember to change every single event handler. Plus, the reads like an English statement – subscribe to the event and if enabled, then do whatever else is necessary.

Great, but unless you are a masochist who revels in littering the code base with hard to reproduce bugs that bomb your app only when demoing to your most important customer, you must, of course, write code to unsubscribe. But there’s no method name to refer to, so you do it the same way as you did when subscribing.

   1: public void Initialize()

   2: {

   3:     control.KeyPressed += IfEnabledThenDo(control_KeyPressed);

   4:     control.MouseMoved += IfEnabledThenDo(control_MouseMoved);

   5: }

   6:  

   7: public void Destroy()

   8: {

   9:     control.KeyPressed -= IfEnabledThenDo(control_KeyPressed);

  10:     control.MouseMoved -= IfEnabledThenDo(control_MouseMoved);

  11: }

This, unfortunately, won’t work – the application will still remain subscribed to those events. Can you figure out why?

Answer and more in the next blog post.

You're writing this really cool and innovative class to calculate the first hundred thousand natural numbers. You think about the API, and you realize that returning an array of the numbers a) might take a long time to complete, and b) is going to cause memory usage to spike up like mad.

So you decide to stream them instead, returning an IEnumerable<T> instance instead of an array. Being a crack C# developer, you use the incredibly powerful yield keyword, rather than rolling your own implementation of the IEnumerable<T> interface.

   1: class MyCoolMathEngine

   2: {

   3:     public IEnumerable<int> GetFirstHundredThousandNaturalNumbers()

   4:     {

   5:         for (int i = 0; i < 100000; ++i)

   6:             yield return i;

   7:     }

   8: }

People start downloading your class and it becomes so popular that they want you to host it in an external service, as a remote component.

Fine, you say, and you pick .NET remoting to provide remote access. You pat yourself on the back for thinking ahead and using an enumerator model – it scales rather nicely. You write the plumbing code to host the class, and with a victorious smile on your lips, you write a test client that instantiates and accesses the method.

Only to find that it crashes with a System.Runtime.Serialization.SerializationException that says a type that you didn’t even write is not marked serializable.

That’s when it hits you, or at least that’s when it hit me, when I was modifying FinalizeTypeFinder to get it load on a different AppDomain.

Because yield return was used, what actually gets back to the caller is an instance of a compiler generated class that implements IEnumerable<int>, and that autogenerated class is not marked serializable. Nor does it derive from MarshalByRefObject, so there’s no way it can be remoted.

The only real solution I can think of is to write a custom implementation of IEnumerable<T>, like we had to do back in the C# 1.1 days. Yet another instance where compiler magic doesn’t quite work out in a particular scenario, I suppose.

My colleague Soundar discovered this rather interesting behavior.

  1: class Test
  2: {
  3:     public static void Main()
  4:     {
  5:         Test test = null;
  6: 
  7:         Console.WriteLine("{0}", test);
  8:         Console.WriteLine("{0}", null);
  9:     }
 10: }

If you run this code, you’ll find that while line 7 prints an empty line, line 8 causes an ArgumentNullException. Note that the test reference is also null, so it should certainly surprise you that the two lines result in different behavior at runtime.

It certainly surprised me enough to make me dig deeper into the reason for the difference. I reasoned that given that the parameter values are identical at runtime, the discrepancy must happen because of a compiler operation – probably method overloading. And sure enough, the two calls resolve to different overloads.

Line 7 resolves to

public static void WriteLine(string format, object arg0);

whereas Line 8 resolves to

public static void WriteLine(string format, params object[] arg);

A peek at the source code using Reflector showed that the second overload throws if arg is null, whereas the first one packs arg0 into an object array and calls the second overload.

Ok, but why did the compiler pick different overloads?

Intuitively, for a method call with a single parameter, you’d expect the overload resolution algorithm to choose a single parameter method over a method with variable number of arguments. And that’s what the compiler did on line 7.

On line 8, the situation is different – null is directly assignable to arg0 and to arg. The overload resolution algorithm had to choose the best function, and it chose the one with the object array.

That appears counter intuitive, until you have code like

  1: class Test
  2: {
  3:     public static void Main()
  4:     {
  5:         SubTest subTest = null;
  6:         M(subTest);
  7:     }
  8: 
  9:     static void M(Test t) { Console.WriteLine("Test"); }
 10:     static void M(SubTest s) { Console.WriteLine("SubTest"); }
 11: }
 12: 
 13: class SubTest : Test { }

You wouldn’t be surprised if the call at line 6 resolved to M(SubTest), would you?

The C# spec’s rules for determining the best match say that

“ Given an implicit conversion C1 that converts from a type S to a type T1, and an implicit conversion C2 that converts from a type S to a type T2, the better conversion of the two conversions is determined as follows:

• If T1 and T2 are the same type, neither conversion is better.
• If S is T1, C1 is the better conversion.
• If S is T2, C2 is the better conversion.
• If an implicit conversion from T1 to T2 exists, and no implicit conversion from T2 to T1 exists, C1 is the better conversion.
• If an implicit conversion from T2 to T1 exists, and no implicit conversion from T1 to T2 exists, C2 is the better conversion.
• … “

In this case, SubTest (T1) is implicitly convertible to Test (T2), and therefore the compiler picks M(SubTest).

Now in our case, the compiler was trying to pick the best conversion between null to object and null to object[]. Applying the same rule as above, object[] is implicitly convertible to object, and therefore the overload resolution algorithm chose  WriteLine(string format, params object[] arg). The params modifier didn’t play a part in overload resolution in this (null) case.

Interesting, ain’t it?

If you’ve done any multithreading programming at all, you must be aware of the volatile modifier. When a field is marked volatile, it tells

1. the JIT compiler that it can’t hoist the field because it may be modified by multiple threads

2. the CLR that the field must be read to and written from with acquire and release semantics.

Given what you’ve read above, the post’s title doesn’t make sense. A local variable, by definition, cannot be accessed from multiple threads. An object referred to by a local variable can be shared among threads, but never the variable itself.

Well, that was true as long as local variables remained just that – local variables. The 2.0 release of C# brought closures to the language, and C# implements capturing of local variables by making them members of a generated class. Now do you see the problem?

  1: public static void Main()
  2: {
  3:     bool stopRunning = false;
  4: 
  5:     Thread t = new Thread(() =>
  6:         {
  7:             while (!stopRunning)
  8:                 Console.WriteLine("Hello");
  9:         });
 10:     t.Start();
 11:     DoSomethingElse();
 12:     stopRunning = true;
 13: }

Nothing out of the ordinary here – I’m creating a thread, passing a lambda to the Thread constructor, and capturing stopRunning inside the lambda.

This code isn’t correct though – for the reasons mentioned in the initial paragraph of this post, stopRunning needs to be declared with the volatile modifier. Unfortunately, you can’t make stopRunning volatile – the compiler complains that local variables cannot be marked volatile.

Oops.

Making stopRunning a member of the class will solve the immediate problem – you can then mark the field volatile, and all is good. However, left at that, it now makes the class non-threadsafe – two threads could call Main, and stopRunning will be shared between them.

I guess this is the price to pay for compiler magic – magic that enables seamless access to local variables from anonymous methods.

ForEachMethodInFile is a Visual Studio macro that lets you do custom actions for each method defined in the current file. I’ve used it in the past to generate logging code to log the start and end of each method, to generate default error handling code etc..  And today someone over at CodeProject wanted to generate a breakpoint at the start of each method. The common thread here is the performing of some custom action for each method in the current file. I figured it would be useful for a lot of people if the “for each method in file” logic is available as a separate library, and that is what is ForEachMethodInFile.

The macro project is available here. To provide a custom action, all you have to do is create a new macro project, and do

  1: Imports System
  2: Imports EnvDTE
  3: Imports EnvDTE80
  4: Imports EnvDTE90
  5: Imports System.Diagnostics
  6: Imports ForEachMethodInFile
  7: 
  8: Public Module BreakOnEachMethodEntry
  9: 
 10:     Public Sub Run()
 11:         ForEachMethod.DoAction(AddressOf AddBreakPoint)
 12:     End Sub
 13: 
 14:     Function AddBreakPoint(ByVal method As CodeFunction)
 15:         DTE.Debugger.Breakpoints.Add(method.Name)
 16:     End Function
 17: 
 18: End Module

You define a callback function that takes the COM object representing a method as a parameter, and then pass on the function to the DoAction method. Your function will be called for each method defined in the file, and the cursor will be positioned at the start of the method before the callback occurs. In this case, I want to add a breakpoint, so I call the appropriate DTE method, passing the current method’s name to tell it where to place the breakpoint.

Here’s the entire code in the ForEachMethodInFile project – it basically recursively traverses code elements in the file, and invokes the callback when it runs into a method.

  1: Imports System
  2: Imports EnvDTE
  3: Imports EnvDTE80
  4: Imports System.Diagnostics
  5: 
  6: Public Module ForEachMethod
  7: 
  8:     Public Sub DoAction(ByRef action As Action(Of CodeFunction))
  9:         ProcessFile(action)
 10:     End Sub
 11: 
 12:     Function ProcessFile(ByRef action As Action(Of CodeFunction))
 13:         Dim selection As EnvDTE.TextSelection
 14:         Dim projectItem As ProjectItem
 15:         Dim fileCodeModel As FileCodeModel
 16:         Dim codeElement As CodeElement
 17:         Dim i As Integer
 18: 
 19:         Dim currentFunction As CodeFunction
 20: 
 21:         projectItem = DTE.ActiveDocument.ProjectItem
 22: 
 23:         fileCodeModel = projectItem.FileCodeModel
 24:         For i = 1 To fileCodeModel.CodeElements.Count
 25:             codeElement = fileCodeModel.CodeElements.Item(i)
 26:             ProcessCodeElement(codeElement, action)
 27:         Next
 28: 
 29:         ' Reformat the modified code
 30:         selection = DTE.ActiveDocument.Selection
 31:         selection.SelectAll()
 32:         selection.SmartFormat()
 33:     End Function
 34: 
 35:     Sub ProcessNamespace(ByVal namespaceElement As CodeNamespace, ByRef action As Action(Of CodeFunction))
 36:         Dim i As Integer
 37:         Dim codeElement As CodeElement
 38: 
 39:         For i = 1 To namespaceElement.Members.Count
 40:             codeElement = namespaceElement.Members.Item(i)
 41:             ProcessCodeElement(codeElement, action)
 42:         Next
 43:     End Sub
 44: 
 45:     Sub ProcessCodeElement(ByVal codeElement As CodeElement, ByRef action As Action(Of CodeFunction))
 46:         If codeElement.Kind = vsCMElement.vsCMElementNamespace Then
 47:             ProcessNamespace(codeElement, action)
 48:         ElseIf codeElement.Kind = vsCMElement.vsCMElementClass Then
 49:             ProcessType(codeElement, action)
 50:         ElseIf codeElement.Kind = vsCMElement.vsCMElementFunction Then
 51:             ProcessMethod(codeElement, action)
 52:         End If
 53:     End Sub
 54: 
 55:     Sub ProcessType(ByVal typeElement As CodeClass, ByRef action As Action(Of CodeFunction))
 56:         Dim i As Integer
 57:         Dim codeElement As CodeElement
 58: 
 59:         For i = 1 To typeElement.Members.Count
 60:             codeElement = typeElement.Members.Item(i)
 61:             If codeElement.Kind = vsCMElement.vsCMElementFunction Then
 62:                 ProcessMethod(codeElement, action)
 63:             ElseIf codeElement.Kind = vsCMElement.vsCMElementClass Then
 64:                 ProcessType(codeElement, action)
 65:             End If
 66:         Next
 67:     End Sub
 68: 
 69:     Sub ProcessMethod(ByVal methodElement As CodeFunction, ByRef action As Action(Of CodeFunction))
 70:         Dim selection As EnvDTE.TextSelection
 71:         Dim editPoint As EnvDTE.EditPoint
 72:         Dim verifyPoint As EnvDTE.TextPoint
 73:         Dim endPointAbsCharOffset As Integer
 74:         Dim column As Integer
 75:         Dim methodRunNotifierSignature As String
 76:         Dim functionStartCode As String
 77:         Dim functionEndCode As String
 78:         Dim parameters As String
 79:         Dim parameter As EnvDTE80.CodeParameter2
 80:         Dim i As Integer
 81: 
 82:         If methodElement.MustImplement Then
 83:             Return
 84:         End If
 85: 
 86:         selection = DTE.ActiveDocument.Selection
 87:         editPoint = selection.ActivePoint.CreateEditPoint()
 88:         verifyPoint = selection.ActivePoint.CreateEditPoint()
 89: 
 90:         ' Move to start of method
 91:         editPoint.MoveToPoint(methodElement.GetStartPoint(vsCMPart.vsCMPartBody))
 92:         selection.MoveToPoint(editPoint)
 93:         verifyPoint.MoveToPoint(methodElement.GetStartPoint(vsCMPart.vsCMPartBody))
 94: 
 95:         action(methodElement)
 96: 
 97:     End Sub
 98: End Module
 99: 
100: 

WinMacro is a tiny little application that can record and replay keyboard and mouse actions that you do on your Windows desktop. It’s similar to the macro facility in Word and Excel, but works across applications.

I wrote the initial version nearly 5 years back, and it proved to be very popular, with approximately 30000 downloads and tons of email from users. Most of the emails were appreciative, and some of them touching, especially those that described how WinMacro was helping people do a better job in fields ranging from cancer research to network testing.

There were quite a few feature requests and bug reports too. WinMacro 2.0 (Beta) (http://winmacro.codeplex.com/) attempts to addresses some of them. The list of new features is available in the download page.

That apart, it was “interesting” to look at code I’d written 5 years ago. I found it positively revolting, to say the least. Lots of copy pasted code, no error handling and plenty of global variables and convoluted code made it scored very heavily in the WTF scale. And this was code I was rather proud of, at that time.

On the flip side, the fact that I found my old code disgusting means I’ve improved my coding skills enough to make that code look terrible. But that is still relative improvement though, it remains to be seen how much I score on the absolute WTF scale, if there is one :)

In the previous blog post, we found that mutating a struct inside a class works if the struct is declared as a field, but doesn’t work if it is declared as a property.

The reason is fairly obvious – if struct fields also returned a copy, then there wouldn’t be any way of mutating the instance at all, even from within the declaring class.

  1: struct S
  2: {
  3:     public int X;
  4: }
  5: 
  6: class C
  7: {
  8:     public S S;
  9: 
 10:     void SetX()
 11:     {
 12:         this.S.X = 1; // Won't work if this.S returned a copy
 13:     }
 14: }

S would essentially act like a readonly field, except that you can’t change it even from within the constructor.

With that out of the way, let’s see how field access works under the covers – here’s the generated IL.

  1: .method public hidebysig static void Main() cil managed
  2: {
  3:     .entrypoint
  4:     .maxstack 2
  5:     .locals init (
  6:         [0] class C c)
  7:     L_0000: nop 
  8:     L_0001: newobj instance void C::.ctor()
  9:     L_0006: stloc.0 
 10:     L_0007: ldloc.0 
 11:     L_0008: ldflda valuetype S C::S
 12:     L_000d: ldc.i4.1 
 13:     L_000e: stfld int32 S::X
 14:     L_0013: ret 
 15: }

The key here is the ldflda instruction – MSDN says it “finds the address of a field in the object whose reference is currently on the evaluation stack”. In contrast, here’s how property access is compiled.

  1: .method public hidebysig static void Main() cil managed
  2: {
  3:     .entrypoint
  4:     .maxstack 1
  5:     .locals init (
  6:         [0] class C c,
  7:         [1] int32 val)
  8:     L_0000: nop 
  9:     L_0001: newobj instance void C::.ctor()
 10:     L_0006: stloc.0 
 11:     L_0007: ldloc.0 
 12:     L_0008: callvirt instance valuetype S C::get_S()
 13:     L_000d: ldfld int32 S::X
 14:     L_0012: stloc.1 
 15:     L_0013: ret 
 16: }

To summarize, for a struct declared in a class, the compiler disallows mutating it if its exposed through an instance property, but allows it if it is a non-readonly field.

How does the compiler detect mutation though? What happens if I call a method on the struct, rather than change a field inside it? More about it in the next post.