the blog => anything goes : C# 3.0

When a C++ destructor did not run – Part 1

Senthil — Fri, 04 Dec 2009 06:01:00 GMT

Consider this piece of C++ code.

   1: using namespace std;

2:

   3: class C

   4: {

   5: public:

   6:     C()

   7:     {

   8:         cout << "Constructed";

   9:     }

  10:     ~C()

  11:     {

  12:         cout << "Destructed";

  13:     }

  14: };

15:

  16: void SomeFunc()

  17: {

  18:     C c;

  19:     throw std::exception("Gone");

  20: }

If you know any C++ at all, you’ll know that when SomeFunc returns, both “Constructed” and “Destructed” will be printed to the console. That is because C++ guarantees that the destructor of an object created on the stack will always run when control leaves the scope, no matter what, and RAII depends on that fact.

You put all this code is in a static library, say PureCPP.lib, and you compile it with the /EHs option, because you want to use C++ exceptions.

You then write a native application to consume this library, statically link to it, and everything works great.

One day, you wake up and realize you’ll have to try out this .NET stuff that everyone is talking about. You discover that there’s this language called C++/CLI that’s great for interfacing with native code. So you fire up VS, create a CLR console application that calls SomeFunc, and link PureCPP.lib against it.

Just when you’re wondering how easy things turned out to be, you notice something strange. There’s something missing in the console output. When you figure out what’s missing, your jaw hits the ground. Mine did too, when I realized that it was the “Destructed” part that was missing. Which means the impossible just happened - the destructor for class C did not run.

What followed was a long and exciting journey into the world of SEH (Structured Exception Handling), exception codes and exception propagation and handling by the CLR versus C++. All that in the next part – stay tuned.

Wrong compiler warning? Or Not?

Senthil — Tue, 01 Dec 2009 05:27:57 GMT

This piece of code results in a warning.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

   6:         string myName = "Senthil";

   7:         Console.WriteLine(obj == myName);

   8:     }

   9: }

The compiler says “warning CS0252: Possible unintended reference comparison; to get a value comparison, cast the left hand side to type 'string'” on line 7. The compiler is right; it has no way of knowing that obj is actually a string. It infers the static type of obj to be System.Object, and if you’ve done any programming in C# at all, you’ll know that the default behavior of the == operator is reference comparison. This code is unintentionally comparing reference values of two local variables, and that’s what the compiler is warning about.

Yet try running the code, and you’ll see that it works correctly; it prints True. Is the compiler wrong?

No. The code appears to work fine because string interning masks the problem. The C# compiler figures that there are two constant strings in the program, and both are equal, so instead of creating two distinct string objects, it creates only one. With strings being immutable, reusing string objects for identical strings is safe. And that’s why this code works, both obj and myName refer to the same string object, which means reference comparison will succeed.

Now that we know why it’s working, it’s easy to prove that the compiler warning is right with a small code change.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

6:

   7:         string myName1 = "Sen";

   8:         string myName2 = "thil";

   9:         string myName = myName1 + myName2;

  10:         Console.WriteLine(obj == myName);

  11:     }

  12: }

Constructing myName dynamically forces the compiler to create another string object; it cannot reuse the interned one. As you’d expect, this piece of code will print False.

Fixing the warning is a mere matter of casting obj to the correct type (string) and then doing the equality check.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

6:

   7:         string myName1 = "Sen";

   8:         string myName2 = "thil";

   9:         string myName = myName1 + myName2;

  10:         Console.WriteLine(((string)obj) == myName);

  11:     }

  12: }

Or we could use the Equals method, which is virtual and won’t be fooled by the compile time type.

   1: class Test

   2: {

   3:     public static void Main()

   4:     {

   5:         object obj = "Senthil";

6:

   7:         string myName1 = "Sen";

   8:         string myName2 = "thil";

   9:         string myName = myName1 + myName2;

  10:         Console.WriteLine(obj.Equals(myName));

  11:     }

  12: }

Both of them print True, which is what we want.

When what you set is not what you get : SetEnvironmentVariable and getenv

Senthil — Tue, 13 Oct 2009 06:14:12 GMT

Mixing SetEnvironmentVariable and getenv is asking for trouble, as we recently found to our dismay (and exasperation!). It took some serious debugging to figure out the actual problem – let’s see if you can figure it out.

Here’s some C++/CLI code that uses getenv to read the value of an environment variable and print it to the console.

   1: public ref class EnvironmentVariablePrinter

   2: {

   3:     public:

   4:         void PrintEnv(String ^key)

   5:         {

   6:              IntPtr ptr = Marshal::StringToHGlobalAnsi(key);

   7:              const char *ckey = (const char *)ptr.ToPointer();

   8:              const char *cval = getenv(ckey);

   9:              string val = cval == NULL ? "" : string(cval);

10:

  11:              cout << "Key is " << ckey << " and value is " << val;

12:

  13:              Marshal::FreeHGlobal(ptr);

  14:         }

  15: };

PrintEnv merely converts the .NET string into an unmanaged one and calls getenv, printing the returned value to the console.

And here’s some C# code that tests the class.

   1: class Test

   2: {

   3:     const string key = "MyKey";

   4:     public static void Main()

   5:     {

   6:         Environment.SetEnvironmentVariable(key, "MyValue");

   7:         PrintValue();

   8:     }

9:

  10:     private static void PrintValue()

  11:     {

  12:         EnvironmentVariablePrinter er = new EnvironmentVariablePrinter();

  13:         er.PrintEnv(key);

  14:     }

  15: }

The code uses System.Environment.SetEnvironmentVariable to set the value of the variable and then calls the C++/CLI code to verify that it prints the correct value. And of course, being written in different languages, the two pieces of code must reside in different projects, say CPlusPlusLib.dll and ConsoleApplication.exe, with the latter referencing the former.

No surprises here - this works as expected and prints “Key is MyKey and value is MyValue”.

However, a seemingly harmless change breaks the code big time.

   1: class Test

   2: {

   3:     const string key = "MyKey";

   4:     public static void Main()

   5:     {

   6:         Environment.SetEnvironmentVariable(key, "MyValue");

   7:         EnvironmentVariablePrinter er = new EnvironmentVariablePrinter();

   8:         er.PrintEnv(key);

   9:     }

  10: }

All I’ve done is inlining of PrintValue, yet running this code prints “Key is MyKey and value is” – getenv is now returning NULL instead of “MyValue”.

It gets even more interesting.

   1: class Test

   2: {

   3:     const string key = "MyKey";

   4:     public static void Main()

   5:     {

   6:         Environment.SetEnvironmentVariable(key, "MyValue");

   7:         EnvironmentVariablePrinter er = null;

   8:         PrintValue();

   9:     }

10:

  11:     private static void PrintValue()

  12:     {

  13:         EnvironmentVariablePrinter er = new EnvironmentVariablePrinter();

  14:         er.PrintEnv(key);

  15:     }

  16: }

This doesn’t work either – the mere declaration of EnvironmentVariablePrinter inside Main makes getenv return NULL for “MyKey”. This can’t be good, can it?

Actually yes, because that is a valuable clue – it means the change in behavior has something to do with JITting. As long as all code that references EnvironmentVariablePrinter is in a separate method, everything works fine (in debug mode, atleast). Things start going south when any such code is in Main itself.

What would the JITter do differently in the two scenarios? Load the assembly containing EnvironmentVariablePrinter (CPlusPlusLib.dll) at different times, of course. When all code referencing EnvironmmentVariablePrinter is inside PrintValue, it will have to load the DLL only when JITting PrintValue, whereas in the other case, it will have to load it when JITting Main. JITting Main obviously occurs much before JITting PrintValue, so DLL load time (relative to other code) is one big difference between the two scenarios that occurs because of JITting.

Why would loading CPlusPlusLib.dll a little early make getenv return NULL?

To understand that, you’ll first have to know how the getenv function works. Windows has APIs to set and get environment variables (SetEnvironmentVariable/GetEnvironmentVariable), and the .NET method P/Invokes into the Windows API to set and get values. getenv, on the other hand, is a CRT function, and does not delegate to the Windows API. Instead, the CRT gets all environment variables and their values when it is starting up (using the GetEnvironmentStrings Windows API), and copies them into its own data structures (MSVCR80!environ). getenv then works on the copied data from then on.

Now do you see the problem? When CPlusPlusLib.dll is loaded early (when JITting Main), the CRT also gets loaded as one of its dependencies, and the startup code that copies environment variables runs right away. At that point, Main hasn’t even been JITted yet, so there’s no way our call to System.Environment.SetEnvironmentVariable could have run by that time. And when it actually runs, it’s too late – the CRT environ block would have been updated much earlier, and calling the Windows API’s SetEnvironmentVariable wouldn’t have any effect on the cached values. When getenv runs, it looks in the cached values and returns NULL.

It’s easy to see why it works in the first case now – CRT loading occurs when JITting PrintValue, and that occurs after our call to SetEnvironmentVariable has executed. Which means that when it calls GetEnvironmentStrings as part of startup, it gets the variable (and its value) that we just set.

Nasty, ain’t it? The actual scenario was a lot more messy – things suddenly stopped working when we linked to a DLL ported to VS2008. We actually figured the problem backwards – we first saw that the CRT load time was different, theorized how getenv works, verified the theory by stepping through the assembly code and looking at the environ block, and once we realized the problem, figured out what was causing early loading of the CRT. Windbg was awesome for debugging this - things would have been very difficult if not for sxe ld:MSVCR90 and x MSVCR80!environ.

The fix was rather simple - in our case, we merely had to move code that set environment variable before CRT load. There’s another twist though; mscorwks.dll, which is the heart of the CLR, loads MSVCR80 when it loads, and you can’t set your environment variables before that, not from managed code anyway. Fortunately, in our case, the getenv call is from a library that links to MSVCR90, so as long as we set the environment variable before that version of the CRT loads, we’re good to go. Until the CLR gets linked to MSVCR90, anyway :).

Anonymous methods as event handlers – Part 2

Senthil — Wed, 09 Sep 2009 05:02:26 GMT

The previous post discussed having anonymous methods as event handlers and ended with a question – why doesn’t unsubscription work while subscription works out alright?

Vivek got the answer spot on – the way the C# compiler handles and translates anonymous methods is the reason.

Here’s the code involved.

   1: public void Initialize()

   2: {

   3:     control.KeyPressed += IfEnabledThenDo(control_KeyPressed);

   4:     control.MouseMoved += IfEnabledThenDo(control_MouseMoved);

   5: }

6:

   7: public void Destroy()

   8: {

   9:     control.KeyPressed -= IfEnabledThenDo(control_KeyPressed);

  10:     control.MouseMoved -= IfEnabledThenDo(control_MouseMoved);

  11: }

12:

  13: public EventHandler<Control.ControlEventArgs> IfEnabledThenDo(EventHandler<Control.ControlEventArgs> actualAction)

  14: {

  15:     return (sender, args) => { if (args.Control.Enabled) actualAction(sender, args); };

  16: }

The compiler translates IfEnabledThenDo into this

   1: public EventHandler<Control.ControlEventArgs> IfEnabledThenDo(EventHandler<Control.ControlEventArgs> actualAction)

   2: {

   3:     <>c__DisplayClass1 CS$<>8__locals2 = new <>c__DisplayClass1();

   4:     CS$<>8__locals2.actualAction = actualAction;

   5:     return new EventHandler<Control.ControlEventArgs>(CS$<>8__locals2.<IfEnabledThenDo>b__0);

   6: }

Now the problem should be fairly obvious – every time the function is called, a new object gets created, and the event handler returned actually refers to a method (<IfEnabledThenDo>b__0) on the new instance. And that’s what breaks unsubscription. –= will not remove a delegate of a different instance of the same class from the invocation list – if it did, the consequences would not be pleasant if multiple instances of the same class subscribe to an event.

But why does the compiler translate our lambda expression this way? Raymond Chen has a great blog post explaining why, but the short answer is that it is needed to “hold” actualAction (the method parameter to IfEnabledThenDo) so that it is available when the event handler actually executes.

Now that we know why, the way to get around this issue is to cache the delegate instance returned by IfEnabledThenDo and use the same instance for subscription and unsubscription.

   1: EventHandler<Control.ControlEventArgs> keyPressed;

   2: EventHandler<Control.ControlEventArgs> mouseMoved;

3:

   4: public void Initialize()

   5: {

   6:    keyPressed = IfEnabledThenDo(control_KeyPressed);

   7:    mouseMoved = IfEnabledThenDo(control_MouseMoved);

8:

   9:    control.KeyPressed += keyPressed;

  10:    control.MouseMoved += mouseMoved;

  11: }

12:

  13: public void Destroy()

  14: {

  15:    control.KeyPressed -= keyPressed;

  16:    control.MouseMoved -= mouseMoved;

  17: }

Knowing how things work under the hood has its advantages, I guess :)

PS : A very small syntactic change to the original example would have made the code work right away. If you’ve followed along this far, you should be able to figure out why.

   1: public void Initialize()

   2: {

   3:    actualAction = control_KeyPressed;

   4:    control.KeyPressed += IfEnabledThenDo();

   5: }

6:

   7: public void Destroy()

   8: {

   9:     control.KeyPressed -= IfEnabledThenDo();

  10: }

11:

  12: EventHandler<Control.ControlEventArgs> actualAction;

  13: public EventHandler<Control.ControlEventArgs> IfEnabledThenDo()

  14: {

  15:     return (sender, args) => { if (args.Control.Enabled) actualAction(sender, args); };

  16: }

Anonymous methods as event handlers – Part 1

Senthil — Tue, 01 Sep 2009 20:22:40 GMT

The syntactic sugar offered by anonymous methods makes them great candidates for writing event handlers; together with smart type inference, they reduce the amount of code written by an order of magnitude.

And that’s without considering the power offered by closures. With event handlers, closures allow you to kind of “stuff” extra parameters into the handler, without changing the actual number of formal parameters. This blog post shows a situation where an anonymous method acting as an event handler makes code simpler, and then goes on to show a gotcha with un-subscription and anonymous methods.

Here’s a simple Control class that fires a bunch of events.

   1: class Control

   2: {

   3:     public class ControlEventArgs : EventArgs

   4:     {

   5:         public Control Control {get;set;}

   6:     }

7:

   8:     public bool Enabled { get; set; }

9:

  10:     public event EventHandler<ControlEventArgs> KeyPressed;

  11:     public event EventHandler<ControlEventArgs> LeftButtonClicked;

  12:     public event EventHandler<ControlEventArgs> RightButtonClicked;

  13:     public event EventHandler<ControlEventArgs> MouseMoved;

  14: }

Let’s say you’re developing a GUI application with this class, and you want to handle events only if the originating control is visually enabled i.e., Enabled set to true. Pretty reasonable constraint, but tedious to implement, if you go the standard way of adding the check to the start of each of your event handlers.

   1: class GUIApp

   2: {

   3:     public void Initialize()

   4:     {

   5:         Control control = new Control();

   6:         control.KeyPressed += new EventHandler<Control.ControlEventArgs>(control_KeyPressed);

   7:         control.MouseMoved += new EventHandler<Control.ControlEventArgs>(control_MouseMoved);

   8:     }

9:

  10:     void control_MouseMoved(object sender, Control.ControlEventArgs e)

  11:     {

  12:         if (e.Control.Enabled)

  13:         {

  14:             ///

  15:         }

  16:     }

17:

  18:     void control_KeyPressed(object sender, Control.ControlEventArgsEventArgs e)

  19:     {

  20:         if (e.Control.Enabled)

  21:         {

  22:             ///

  23:         }

  24:     }

  25: }

With an anonymous method, you could write a far more terse and easy to maintain version

   1: class GUIApp

   2: {

   3:     public void Initialize()

   4:     {

   5:         Control control = new Control();

   6:         control.KeyPressed += IfEnabledThenDo(control_KeyPressed);

   7:         control.MouseMoved += IfEnabledThenDo(control_MouseMoved);

   8:     }

9:

  10:     public EventHandler<Control.ControlEventArgs> IfEnabledThenDo(EventHandler<Control.ControlEventArgs> actualAction)

  11:     {

  12:         return (sender, args) => { if (args.Control.Enabled) { actualAction(sender, args); } };

  13:     }

14:

  15:     void control_MouseMoved(object sender, Control.ControlEventArgs e)

  16:     {

  17:         ///

  18:     }

19:

  20:     void control_KeyPressed(object sender, Control.ControlEventArgs e)

  21:     {

  22:         ///

  23:     }

  24: }

IfEnabledThenDo returns an anonymous function that first checks whether the control is enabled before calling the actual function. The code is much shorter, and the condition is checked only in one place, which makes it easy to modify or add additional logic without having to remember to change every single event handler. Plus, the reads like an English statement – subscribe to the event and if enabled, then do whatever else is necessary.

Great, but unless you are a masochist who revels in littering the code base with hard to reproduce bugs that bomb your app only when demoing to your most important customer, you must, of course, write code to unsubscribe. But there’s no method name to refer to, so you do it the same way as you did when subscribing.

   1: public void Initialize()

   2: {

   3:     control.KeyPressed += IfEnabledThenDo(control_KeyPressed);

   4:     control.MouseMoved += IfEnabledThenDo(control_MouseMoved);

   5: }

6:

   7: public void Destroy()

   8: {

   9:     control.KeyPressed -= IfEnabledThenDo(control_KeyPressed);

  10:     control.MouseMoved -= IfEnabledThenDo(control_MouseMoved);

  11: }

This, unfortunately, won’t work – the application will still remain subscribed to those events. Can you figure out why?

Answer and more in the next blog post.

Remoting IEnumerables

Senthil — Sat, 29 Aug 2009 14:49:45 GMT

You're writing this really cool and innovative class to calculate the first hundred thousand natural numbers. You think about the API, and you realize that returning an array of the numbers a) might take a long time to complete, and b) is going to cause memory usage to spike up like mad.

So you decide to stream them instead, returning an IEnumerable<T> instance instead of an array. Being a crack C# developer, you use the incredibly powerful yield keyword, rather than rolling your own implementation of the IEnumerable<T> interface.

        1: class MyCoolMathEngine

       2: {

       3:     public IEnumerable<int> GetFirstHundredThousandNaturalNumbers()

       4:     {

       5:         for (int i = 0; i < 100000; ++i)

       6:             yield return i;

       7:     }

       8: }

People start downloading your class and it becomes so popular that they want you to host it in an external service, as a remote component.

Fine, you say, and you pick .NET remoting to provide remote access. You pat yourself on the back for thinking ahead and using an enumerator model – it scales rather nicely. You write the plumbing code to host the class, and with a victorious smile on your lips, you write a test client that instantiates and accesses the method.

Only to find that it crashes with a System.Runtime.Serialization.SerializationException that says a type that you didn’t even write is not marked serializable.

That’s when it hits you, or at least that’s when it hit me, when I was modifying FinalizeTypeFinder to get it load on a different AppDomain.

Because yield return was used, what actually gets back to the caller is an instance of a compiler generated class that implements IEnumerable<int>, and that autogenerated class is not marked serializable. Nor does it derive from MarshalByRefObject, so there’s no way it can be remoted.

The only real solution I can think of is to write a custom implementation of IEnumerable<T>, like we had to do back in the C# 1.1 days. Yet another instance where compiler magic doesn’t quite work out in a particular scenario, I suppose.

Overload resolution and null

Senthil — Wed, 08 Jul 2009 06:28:15 GMT

My colleague Soundar discovered this rather interesting behavior.

  1: class Test
  2: {
  3:     public static void Main()
  4:     {
  5:         Test test = null;
  6: 
  7:         Console.WriteLine("{0}", test);
  8:         Console.WriteLine("{0}", null);
  9:     }
 10: }

If you run this code, you’ll find that while line 7 prints an empty line, line 8 causes an ArgumentNullException. Note that the test reference is also null, so it should certainly surprise you that the two lines result in different behavior at runtime.

It certainly surprised me enough to make me dig deeper into the reason for the difference. I reasoned that given that the parameter values are identical at runtime, the discrepancy must happen because of a compiler operation – probably method overloading. And sure enough, the two calls resolve to different overloads.

Line 7 resolves to

public static void WriteLine(string format, object arg0);

whereas Line 8 resolves to

public static void WriteLine(string format, params object[] arg);

A peek at the source code using Reflector showed that the second overload throws if arg is null, whereas the first one packs arg0 into an object array and calls the second overload.

Ok, but why did the compiler pick different overloads?

Intuitively, for a method call with a single parameter, you’d expect the overload resolution algorithm to choose a single parameter method over a method with variable number of arguments. And that’s what the compiler did on line 7.

On line 8, the situation is different – null is directly assignable to arg0 and to arg. The overload resolution algorithm had to choose the best function, and it chose the one with the object array.

That appears counter intuitive, until you have code like

  1: class Test
  2: {
  3:     public static void Main()
  4:     {
  5:         SubTest subTest = null;
  6:         M(subTest);
  7:     }
  8: 
  9:     static void M(Test t) { Console.WriteLine("Test"); }
 10:     static void M(SubTest s) { Console.WriteLine("SubTest"); }
 11: }
 12: 
 13: class SubTest : Test { }

You wouldn’t be surprised if the call at line 6 resolved to M(SubTest), would you?

The C# spec’s rules for determining the best match say that

“ Given an implicit conversion C1 that converts from a type S to a type T1, and an implicit conversion C2 that converts from a type S to a type T2, the better conversion of the two conversions is determined as follows:

If T1 and T2 are the same type, neither conversion is better.
If S is T1, C1 is the better conversion.
If S is T2, C2 is the better conversion.
If an implicit conversion from T1 to T2 exists, and no implicit conversion from T2 to T1 exists, C1 is the better conversion.
If an implicit conversion from T2 to T1 exists, and no implicit conversion from T1 to T2 exists, C2 is the better conversion.
… “

In this case, SubTest (T1) is implicitly convertible to Test (T2), and therefore the compiler picks M(SubTest).

Now in our case, the compiler was trying to pick the best conversion between null to object and null to object[]. Applying the same rule as above, object[] is implicitly convertible to object, and therefore the overload resolution algorithm chose WriteLine(string format, params object[] arg). The params modifier didn’t play a part in overload resolution in this (null) case.

Interesting, ain’t it?

Volatile and local

Senthil — Fri, 03 Jul 2009 05:16:54 GMT

If you’ve done any multithreading programming at all, you must be aware of the volatile modifier. When a field is marked volatile, it tells

1. the JIT compiler that it can’t hoist the field because it may be modified by multiple threads

2. the CLR that the field must be read to and written from with acquire and release semantics.

Given what you’ve read above, the post’s title doesn’t make sense. A local variable, by definition, cannot be accessed from multiple threads. An object referred to by a local variable can be shared among threads, but never the variable itself.

Well, that was true as long as local variables remained just that – local variables. The 2.0 release of C# brought closures to the language, and C# implements capturing of local variables by making them members of a generated class. Now do you see the problem?

  1: public static void Main()
  2: {
  3:     bool stopRunning = false;
  4: 
  5:     Thread t = new Thread(() =>
  6:         {
  7:             while (!stopRunning)
  8:                 Console.WriteLine("Hello");
  9:         });
 10:     t.Start();
 11:     DoSomethingElse();
 12:     stopRunning = true;
 13: }

Nothing out of the ordinary here – I’m creating a thread, passing a lambda to the Thread constructor, and capturing stopRunning inside the lambda.

This code isn’t correct though – for the reasons mentioned in the initial paragraph of this post, stopRunning needs to be declared with the volatile modifier. Unfortunately, you can’t make stopRunning volatile – the compiler complains that local variables cannot be marked volatile.

Oops.

Making stopRunning a member of the class will solve the immediate problem – you can then mark the field volatile, and all is good. However, left at that, it now makes the class non-threadsafe – two threads could call Main, and stopRunning will be shared between them.

I guess this is the price to pay for compiler magic – magic that enables seamless access to local variables from anonymous methods.

Mutating value types – Part 1

Senthil — Thu, 21 May 2009 05:13:48 GMT

Take a look at the following short snippet of code.

  1: using System;
  2: 
  3: struct S
  4: {
  5:     public int X;
  6: }
  7: 
  8: class C
  9: {
 10:     /* More code here */
 11: }
 12: 
 13: class Test
 14: {
 15:     public static void Main()
 16:     {
 17:         C c = new C();
 18:         c.S.X = 1;
 19:     }
 20: }

Without knowing the type definition of C, can you tell whether the code will compile, much less work?

Turns out that you can’t. It depends on whether the S in c.S is defined as a field or a property.

class C
{
    public S S;
}

class C
{
    public S S { get; private set; }
}

The first type definition will compile, the second won’t (error CS1612: Cannot modify the return value of 'C.S' because it is not a variable)

Can you guess why?

Let’s assume the compiler does allow the second type definition to compile. Would you expect the value of X in the instance of S inside C to change? That is, what would be the output of

public static void Main()
{
    C c = new C();
    c.S.X = 1;
    Console.WriteLine(c.S.X);
}

If you’re expecting it to be 1, then you have just broken the value semantics of a struct, S might as well have been defined as a class. The above code is logically identical to

public static void Main()
{
    C c = new C();
    S temp = c.S;
    temp.X = 1;
    Console.WriteLine(c.S.X);
}

Because S is a struct, the property getter always returns a copy (temp), and changing the copy will have no effect on the original instance. With c.S.X = 1, you can’t access the copy either, so the only effect of executing that statement will be to make the poor developer’s eyes go wide as he steps through the code in the debugger, wondering why a simple field assignment refuses to work.

So, the C# compiler is being helpful here by not allowing this kind of code to compile.

Right, so why does it compile if S is defined as a field rather than a property? We’ll see why in the next post. Meanwhile, feel free to post why you think it works.

Yowza! My computer is seeing patterns!

Senthil — Thu, 09 Apr 2009 20:08:24 GMT

Ever since I happened to stumble upon this book on Data Mining, I've been hooked. So much so that I've been brushing up on statistics and probability distributions just to follow along the book.

I'm currently reading the chapter on classification using decision trees, and what appeals to me is the how the technique reduces huge sets of data (records) into simple models that can both describe the data and can predict the class of previously unseen records. If all that goes right above your head, here's a simple example.

P1 P2 Class (Result)

True True True

False False False

True False False

False True False

Given the above tabular values, can the computer figure out that it is the truth table for P1 ^ P2 (with ^ representing logical conjunction, not XOR) ? If it can, it has

1. A model that describes the data, i.e., the data is the conjunction of the two boolean variables.

2. An evaluator that can calculate the result, given P1 and P2, i.e., predict the class. Since all combination of values are already available, there are no unseen records in this case.

A decision tree for the above table looks like this visually :-

with green edges and red edges representing true and false values for the attribute represented by the node, respectively. As you can see, the non-leaf nodes each represent an attribute, with one edge for each possible value of that attribute. The leaf nodes represent the result (class). Since we are dealing with binary attributes, the decision tree turns out to be a binary tree. Note that you can find out the result of { P1 = True, P2 = True } by simply navigating the tree and reading the value of the leaf node ( green edge from P1 to P2, green edge from P2 to T, and then read the value, T = true).

This was so interesting that I actually wrote a generic decision tree builder in C# (download). It uses Hunt's Algorithm to build the decision tree and uses Entropy as the measure to select attributes. At the moment, it can work only on records with nominal attributes (attributes like enums whose values are limited to a finite set). After training, it returns a decision tree, which can then be exported to XML or can be used to predict classes for new records.

Enough of the theory, let's see it in action. Here's how code that uses my tree builder looks.

class TruthTableEntry : Record<bool>
{
    public bool P1 { get; set; }
    public bool P2 { get; set; }

    public override bool Equals(object obj)
    {
        var other = obj as TruthTableEntry;
        if (other == null)
            return false;

        return other.P1 == this.P1 && other.P2 == this.P2;
    }

    public override int GetHashCode()
    {
        return P1.GetHashCode() ^ P2.GetHashCode();
    }
}

static void Main(string[] args)
{
   TruthTableEntry[] table = 
   {
       new TruthTableEntry() { P1 = true, P2 = true, Class = true },
       new TruthTableEntry() { P1 = true, P2 = false, Class = false},
       new TruthTableEntry() { P1 = false, P2 = true, Class = false},
       new TruthTableEntry() { P1 = false, P2 = false, Class = false }
   };

   var tree = new TreeBuilder<TruthTableEntry, bool>().Train(table);
   string predicate = FormPredicate(tree.RootNode, "");
   Console.WriteLine(predicate);
}

The TruthTableEntry class represents a row in the truth table above, and an array of those records are given to the TreeBuilder instance's Train method. The decision tree returned by it is then translated into a predicate by the FormPredicate method. Here's how the tree looks in XML if you call Save on the tree.

<?xml version="1.0" encoding="utf-8"?>
<Node ParentConditionValue="" SplitAttribute="P1">
  <Node ParentConditionValue="True" SplitAttribute="P2">
    <Node ParentConditionValue="True" Class="True" />
    <Node ParentConditionValue="False" Class="False" />
  </Node>
  <Node ParentConditionValue="False" Class="False" />
</Node>

which exactly matches what we expected.

The FormPredicate method attempts to transform the tree into a predicate clause, as expressed mathematically in predicate logic. For the above tree, it returns

(P1 ^ P2) v (~P1 ^ False)

which when reduced through the laws of predicate logic

(P1 ^ P2) v (~P1 ^ False)

= (P1 ^ P2) v False (since False ^ P == False)

= (P1 ^ P2) (since False v P == P)

And there you have it, the computer figured it out correctly. Yowza :)

Confusion++

Senthil — Thu, 26 Feb 2009 13:09:44 GMT

I'll bet a hundred bucks that any entry level C++ interview or exam will somehow drift into questions about the pre and post increment operators. It's almost become a canonical, rite of passage sort of thing.

Now using the operators is one thing, overloading them for your own types is another. In C++, you write something like

class X {
    int val;
public:

    X() : val(0){ }

    X operator++() { val++; return *this; }
    X operator++(int) { X pX = *this; val++; return pX; };

    int Value() { return val; }
};

Fairly straightforward stuff - C++ uses the int overload to distinguish between pre and post, and the post increment overload copies itself before incrementing and returns the copy.

Now let's see how to do this in C#.

class X
{
    public int Value { get; set; }

    public static X operator ++(X p)
    {
        int x = p.Value;
        return new X() { Value = x + 1 };
    }
}

No, there's no separate overload for the other one - the same method works for both pre and post increment operations. The compiler does the work of generating code that exhibits correct pre and post increment behavior. For code like

X x = new X();
X y = x++;

it generates

X x = new X();
X y = x;
x = X.op_Increment(x);

and for code like

X x = new X();
X y = ++x;

it generates

X x = new X();
X y = x = X.op_Increment(x);

Nifty, eh?

But wait, did you notice the difference between the C++ and C# overload implementation? The C# overload does not modify the original at all and always returns a copy, whereas the C++ code always modifies the current instance and returns a copy only for the post increment overload. Looking at the generated code, it should be easy to understand why - the compiler can't do its magic if the overload tinkers with the original instance.

Did you notice that X is a reference type?

X x = newX();
Xy = x;

x++;

Given that x and y are referring to the same object after executing the second line, shouldn't the increment be visible from y as well? It doesn't happen though, because the overload returns a new instance that then gets assigned to x, leaving y referring to the "old" unmodified instance.

All this confusion will not arise if X is a value type. The assignment of x to y will create a copy, so there's no possibility of changes in x reflecting in y. And modifying the passed instance from within the operator overload will have no effect on the original instance either.

Moral of the story : watch out when overloading operators on reference types.

Not all IEnumerable are equal

Senthil — Thu, 04 Sep 2008 19:25:00 GMT

Implementing IEnumerable<T> can turn out to be tricky in certain cases. Consider the following code snippet

namespace Test
{
    class Program
    {
        static void Main(string[] args)
        {
            Consume(new List<string>() { "a", "b", "c" });
        }

        static void Consume<T>(IEnumerable<T> stream)
        {
            T t1 = stream.First();
            T t2 = stream.First();

            Console.WriteLine(t1.Equals(t2));
        }
    }
}

As you'd expect, it prints true. Each First() call results in a call to stream.GetEnumerator(), and each such enumerator returns elements from the beginning of the list, so calling First() twice returns the same (first) element. All good so far.

Here's a tiny class.

        class StringGenerator
        {
            int index;

            public string GetNext()
            {
                return (index++).ToString();
            }
        }

As you can see, it generates whole numbers as strings. Not very convenient to use though, wouldn't it be great if we can wrap it and make it enumerable?

        static IEnumerable<string> ConvertToEnumerable(StringGenerator g)
        {
            string item = null;

            while ((item = g.GetNext()) != null)
                yield return item;
        }

ConvertToEnumerable simply loops over the list of items generated and makes use of yield return to make it enumerable.

Great, now what does

        Consume(ConvertToEnumerable(new StringGenerator()));

print?

It prints false.

False? FALSE? Can you figure out the reason?

If you've read Raymond's posts on implementation of iterators, you should have figured it out by now. The crux of the problem is that all enumerators returned share the same instance of StringGenerator. Calling First() twice results in two calls to GetNext() on the same StringGenerator instance, and the values returned will obviously be different. To verify that, try creating the StringGenerator instance inside the ConvertToEnumerable function - it will print true now.

This bit me when I wrote code that parsed stuff out of an IEnumerable<string> instance. The actual program read text from a file, so I had a ConvertToEnumerable routine just like the one above, except that it took TextReader as the parameter. The Consume method passed the constructed IEnumerable<T> instance to various methods (say Method1 and Method2), with the assumption that whatever Method1 read off the stream won't be read again by Method2.

As we saw just now, this works if Consume is passed an IEnumerable<T> constructed like the StringGenerator case. It fails badly if a List<string> is passed instead. Because I wanted the "read elements off the stream" behavior, I called GetEnumerator() once for the passed IEnumerable<T> and then changed the methods called by Consume to take that IEnumerator<T> instead of IEnumerable<T>. That made the code work correctly for both cases.

Moral : Make sure you understand the implications when yield returning items off a shared item source.

Itertools for C# - Tee

Senthil — Mon, 05 May 2008 17:03:00 GMT

Tee is a cool function that "clones" enumerators whatever their current state is - it basically allows you to branch off an enumerator into as many enumerators as you want, all independent of each other.

You can do something like

List<int> list = new List<int> { 1, 2, 3, 4, 5 };

IEnumerator<int> iterable1 = list.GetEnumerator();

IEnumerator<int> []teedIterables = Itertools.Tee(iterable1, 2);

foreach(int val in teedIterables[0])
{
   Console.WriteLine(val);
}

foreach(int val in teedIterables[1])
{
   Console.WriteLine(val);
}

// prints 1 2 3 4 5 1 2 3 4 5

If there was a iterable1.MoveNext() before the call to Tee, then the output would have been 2 3 4 5 2 3 4 5. The implementation proved to be pretty interesting - and complicated, partly because the C# compiler does not allow the yield statement inside anonymous methods. Here's the implementation.

public static IEnumerator<T>[] Tee<T>(IEnumerator<T> source, int count)
{
    List<T> tValues = new List<T>();
    IEnumerator<T>[] results = new IEnumerator<T>[count];

    for (int i = 0; i < count; ++i)
    {
       int currentLocation = 0;
       results [ i ] = CreateEnumerator(source, new EnumeratorState<T>() { CurrentLocation = currentLocation, TValues = tValues });
    }

    return results;
}

private static IEnumerator<T> CreateEnumerator<T>(IEnumerator<T> source, EnumeratorState<T> state)
{
   while (true)
   {
      if (state.CurrentLocation < state.TValues.Count)
      {
          yield return state.TValues[state.CurrentLocation++];
      }
      else
      {
         if (source.MoveNext())
         {
            state.TValues.Add(source.Current);
            state.CurrentLocation = state.TValues.Count;
            yield return source.Current;
         }
         else
         {
             break;
         }
      }
   }
}

class EnumeratorState<T>
{
   internal int CurrentLocation { get; set; }
   internal List<T> TValues { get; set; }
}

There's a fair bit of code involved, as you can see. The basic idea behind the implementation is to maintain a list as a backing store. Each teed off enumerator attempts to read from the list - if it hits the end of the list, it advances the original enumerator and gets the current value, updates the list and returns that value. The rest of the code is plumbing to work around the inability to use yield inside an anonymous method. The code basically does what the compiler would have done - there's the EnumeratorState class to hold captured variables and a new instance of the class is created and passed to CreateEnumerator, which is in the place of the anonymous method.

Tee is that kind of function that is incredibly useful once you know you can do such a thing. Let's say you have an application that needs to process lines in a file, but the processing can happen in different contexts, independent of each other. Take for example, a log parser, that goes through each line in a file, looking for the start of an operation. The log parser notifies another component once it finds the start of the operation and proceeds through the file looking for the next operation. Normally, you'll solve the problem by adding the component to a list that gets notified after the parser reads each line. With Tee, you can simply tee off an enumerator and pass it to the other component - both the log parser and the component now have independent enumerators and can actually run in parallel. And you can tee off as many enumerators as there are components.

With that, we come to the end of this series of blog posts on implementing itertools in C#. The source code for all the functions implemented in the series is available at http://code.msdn.microsoft.com/itertools.

C# 3.0 compiler bug - Using object initializers, generics and value types

Senthil — Wed, 26 Dec 2007 17:44:00 GMT

There is a description of the bug here. The following piece of code demonstrates the bug.

    class Program
    {
        interface I
        {
            int X { get; set; }
        }

        struct S : I
        {
            public int X { get; set; }
        }

        static void Main(string[] args)
        {
            Func<S>();
        }

        static void Func<T>() where T : I, new()
        {
            var c = new T() { X = 1 };
            Console.WriteLine(c.X);
        }
    }

The code prints 0 instead of 1. The bug goes away if object initializers are not used - changing var c = new T() { X = 1 }; to var c = new T(); c.X = 1; will print 1.

I had assumed all along that object initializers were transformed into property setters at the source code level (or somewhere above IL). This bug doesn't help much to confirm that :)

Anyway, let's see what's causing the problem. Peeking into the generated IL,

   .locals init (
        [0] !!T c,
        [1] !!T <>g__initLocal0,
        [2] !!T CS$0$0000)
        ... // Store zeroed out instance of S in locals[1]
    L_0022: ldloc.1 
    L_0023: box !!T
    L_0028: ldc.i4.1 
    L_0029: callvirt instance void CSharp3Test.Program/I::set_X(int32)
    L_002e: nop 
    L_002f: ldloc.1 
    L_0030: stloc.0 
    L_0031: ldloca.s c
    L_0033: constrained !!T
    L_0039: callvirt instance int32 CSharp3Test.Program/I::get_X()
    L_003e: call void [mscorlib]System.Console::WriteLine(int32)

Can you spot the problem now?

You can see that at L_0023, S is boxed and the boxed instance is used to make the virtual call to set X. You can also see that the boxed instance is not stored in any of the local variables, the callvirt instruction pops it off the stack and then it's gone. To load the value of X, the callvirt instruction is given a reference to c (locals[0]), which contains the unboxed instance of the struct. Which is why we are getting a 0 (the default value), instead of 1. Essentially, the compiler is generating the equivalent IL for something like

   S s = new S();
   I i = s;
   i.X = 10;
   Console.WriteLine(s.X);

This again prints 0 instead of 10, because i is assigned a different (boxed) instance of S and therefore changing i.X doesn't change s.X.

But then how does doing the initialization in two steps (var c = new T(); c.X = 1) work. The answer - Two step initialization does not use boxing when setting the value of X. Here's how the generated IL looks.

   .locals init (
        [0] !!T c,
        [1] !!T CS$0$0000)
    L_0021: stloc.0 
    L_0022: ldloca.s c
    L_0024: ldc.i4.1 
    L_0025: constrained !!T
    L_002b: callvirt instance void CSharp3Test.Program/I::set_X(int32)
    L_0030: nop 
    L_0031: ldloca.s c
    L_0033: constrained !!T
    L_0039: callvirt instance int32 CSharp3Test.Program/I::get_X()
    L_003e: call void [mscorlib]System.Console::WriteLine(int32)

The box instruction is gone, instead, the compiler has generated a constrained prefix for the callvirt instruction. MSDN says that

"When a callvirt method instruction has been prefixed by constrained thisType, the instruction is executed as follows:

If thisType is a reference type (as opposed to a value type) then ptr is dereferenced and passed as the 'this' pointer to the callvirt of method.
If thisType is a value type and thisType implements method then ptr is passed unmodified as the 'this' pointer to a call method instruction, for the implementation of method by thisType.
...

In other words, the address of the instance is passed as the this parameter to the call method - no boxing at all. If you'd looked carefully enough, the getter for X uses the constrained prefix, even in our buggy IL (the first listing).

That explains the bug then, but opens up another question. If virtual dispatch can be done for a value type without boxing, then why not do it that way for

   S s = new S();
   I i = s;
   i.X = 10;

as well? Mind you, virtual dispatch doesn't mean anything for value types, as they cannot derive from or derived by anything else, so any callvirt instruction on value types can be treated as a straight call instruction. The constrained prefix does exactly this, after making sure that this is a value type. So why not use it for all virtual calls? Performance (not many value types implementing interfaces)?

What do you think?

Type inference != Dynamic typing

Senthil — Mon, 19 Nov 2007 16:27:00 GMT

Local variable type inferencing is a new feature in C# 3.5 - a very "handy" one, in that it saves a lot of typing. Basically, it lets you do

var dict = new Dictionary<string, List<int, Dictionary<int, string>>>();

instead of

Dictionary<string, List<int, Dictionary<int, string>>> dict = Dictionary<string, List<int, Dictionary<int, string>>>();

That's pretty cool, but remember, C# is still a statically typed language. dict is statically bound to the type Dictionary<string, List<int, Dictionary<int, string>>> and you cannot change it. You cannot do

dict = Dictionary<int, string>();

in the same method. The compiler will complain that the type on the RHS is not convertible to the statically bound type of dict.

Contrast this to a dynamically typed language like Javascript, which also uses the var keyword to declare variables.

var x = 1;x = new Array();
x = "23";

Or to Python

x = 42x = def fn(z) : return z + 1

In such languages, variables (x, in our example) simply act as placeholders and get bound to the type of the value they are holding at the moment. In the first example, x is holding an integer after executing the first line. After executing the second line, it's holding an array. If the same two lines of code were to be written in a statically typed language, the second line would be correct only if an array is somehow convertible to an int.

The point of the post - don't be misled by the var keyword. The variable it declares is still a good old statically typed variable.

P1	P2	Class (Result)
True	True	True
False	False	False
True	False	False
False	True	False