Jon Skeet: Coding Blog : C#, Wacky Ideas

Eduasync part 13: first look at coroutines with async

skeet — Wed, 22 Jun 2011 20:37:30 GMT

(This part covers project 18 in the source code.)

As I mentioned in earlier parts, the "awaiting" part of async methods is in no way limited to tasks. So long as we have a suitable GetAwaiter() method which returns a value of a type which in turn has suitable methods on it, the compiler doesn't really care what's going on. It's time to exploit that to implement some form of coroutines in C#.

Introduction to coroutines

The fundamental idea of coroutines is to have multiple methods executing cooperatively, each of them maintaining their position within the coroutine when they yield to another. You can almost think of them as executing in multiple threads, with only one thread actually running at a time, and signalling between the different threads to control flow. However, we don't really need multiple threads once we've got continuations - we can have a single thread with a complex flow of continuations, and still only a very short "real" stack. (The control flow is stored in normal collections instead of being implicit on the thread's stack.)

Coroutines were already feasible in C# through the use of iterator blocks, but the async feature of C# allows a slightly more natural way of expressing them, in my view. (The linked Wikipedia page gives a sketch of how coroutines can be built on top of generators, which in the general concept that iterator blocks implement in C#.)

I have implemented various flavours of coroutines in Eduasync. It's possible that some (all?) of them shouldn't strictly be called coroutines, but they're close enough to the real thing in feeling. This is far from an exhaustive set of approaches. Once you've got the basic idea of what I'm doing, you may well want to experiment with your own implementations.

I'm not going to claim that the use of coroutines in any of my examples really makes any sense in terms of making real tasks easier. This is purely for the sake of interest and twisting the async feature for fun.

Round-robin independent coroutines

Our first implementation of coroutines is relatively simple. A coordinator effectively "schedules" the coroutines it's set up with in a round-robin fashion: when one of the coroutines yields control to the coordinator, the coordinator remembers where the coroutine had got to, and then starts the next one. When each coroutine has executed its first piece of code and yielded control, the coordinator will go back to the first coroutine to continue execution, and so on until all coroutines have completed.

The coroutines don't know about each other, and no data is being passed between them.

Hopefully it's reasonably obvious that the coordinator contains all the smarts here - the coroutines themselves can be relatively dumb. Let's look at what the client code looks like (along with the results) before we get to the coordinator code.

Client code

The sample code contains three coroutines, all of which take a Coordinator parameter and have a void return type. These are passed to a new coordinator using a collection initializer and method group conversions; the coordinator is then started. Here's the entry point code for this:

private static void Main(string[] args)
{
    var coordinator = new Coordinator {
        FirstCoroutine,
        SecondCoroutine,
        ThirdCoroutine
    };
    coordinator.Start();
}

When each coroutine is initially started, the coordinator passes a reference to itself as the argument to the coroutine. That's how we solve the chicken-and-egg problem of the coroutine and coordinator having to know about each other. The way a coroutine yields control is simply by awaiting the coordinator. The result type of this await expression is void - it's just a way of "pausing" the coroutine.

We're not doing anything interesting in the actual coroutines - just tracing the execution flow. Of course we could do anything we wanted, within reason. We could even await a genuinely asynchronous task such as fetching a web page asynchronously. In that case the whole coroutine collection would be "paused" until the fetch returned.

Here's the code for the first coroutine - the second and third ones are similar, but use different indentation for clarity. The third coroutine is also shorter, just for fun - it only awaits the coordinator once.

private static async void FirstCoroutine(Coordinator coordinator)
{
    Console.WriteLine("Starting FirstCoroutine");
    Console.WriteLine("Yielding from FirstCoroutine...");

    await coordinator;

    Console.WriteLine("Returned to FirstCoroutine");
    Console.WriteLine("Yielding from FirstCoroutine again...");

    await coordinator;

    Console.WriteLine("Returned to FirstCoroutine again");
    Console.WriteLine("Finished FirstCoroutine");
}

And here's the output...

Starting FirstCoroutine
Yielding from FirstCoroutine...
    Starting SecondCoroutine
    Yielding from SecondCoroutine...
        Starting ThirdCoroutine
        Yielding from ThirdCoroutine...
Returned to FirstCoroutine
Yielding from FirstCoroutine again...
    Returned to SecondCoroutine
    Yielding from SecondCoroutine again...
        Returned to ThirdCoroutine
        Finished ThirdCoroutine...
Returned to FirstCoroutine again
Finished FirstCoroutine
    Returned to SecondCoroutine again
    Finished SecondCoroutine

Hopefully that's the output you expected, given the earlier description. Again it may help if you think of the coroutines as running in separate pseudo-threads: the execution within each pseudo-thread is just linear, and the timing is controlled by our explicit "await" expressions. All of this would actually be pretty easy to implement using multiple threads which really did just block on each await expression - but the fun part is keeping it all in one real thread. Let's have a look at the coordinator.

The Coordinator class

Some of the later coroutine examples end up being slightly brainbusting, at least for me. This one is relatively straightforward though, once you've got the basic idea. All we need is a queue of actions to execute. After initialization, we want our queue to contain the coroutine starting points.

When a coroutine yields control, we just need to add the remainder of it to the end of the queue, and move on to the next item. Obviously the async infrastructure will provide "the remainder of the coroutine" as a continuation via the OnContinue method.

When a coroutine just returns, we continue with the next item in the queue as before - it's just that we won't add a continuation to the end of the queue. Eventually (well, hopefully) we'll end up with an empty queue, at which point we can stop.

Initialization and a choice of data structures

We'll represent our queue using Queue<T> where the T is a delegate type. We have two choices here though, because we have two kinds of delegate - one which takes the Coordinator as a parameter (for the initial coroutine setup) and one which has no parameters (for the continuations). Fortunately we can convert between the two in either direction very simply, bearing in mind that all of this is within the context of a coordinator. For example:

// If we're given a coroutine and want a plain Action
Action<Coordinator> coroutine = ...;
Action action = () => coroutine(this);

// If we're given a plain Action and want an Action<Continuation>:
Action continuation = ...;
Action<Coordinator> coroutine = ignored => continuation();

I've arbitrarily chosen to use the first option, so there's a Queue<Action> internally.

Now we need to get the collection initializer working. The C# compiler requires an appropriate Add method (which is easy) and also checks that the type implements IEnumerable. We don't really need to be able to iterate over the queue of actions, so I've use explicit interface implementation to reduce the likelihood of GetEnumerator() being called inappropriately, and made the method throw an exception for good measure. That gives us the skeleton of the class required for setting up:

public sealed class Coordinator : IEnumerable
{
    private readonly Queue<Action> actions = new Queue<Action>();

    // Used by collection initializer to specify the coroutines to run
    public void Add(Action<Coordinator> coroutine)
    {
        actions.Enqueue(() => coroutine(this));
    }

    // Required for collection initializers, but we don't really want
    // to expose anything.
    IEnumerator IEnumerable.GetEnumerator()
    {
        throw new NotSupportedException("IEnumerable only supported to enable collection initializers");
    }
}

(Note that I haven't used XML documentation anywhere here - it's great for real code, but adds clutter in blog posts.)

For production code I'd probably prevent Add from being called after the coordinator had been started, but there's no need to do it in our well-behaved sample code. We're only going to add extra actions to the queue via continuations, which will be added due to await expressions.

The main execution loop and async infrastructure

So far we've got code to register coroutines in the queue - so now we need to execute them. Bearing in mind that the actions themselves will be responsible for adding continuations, the main loop of the coordinator is embarrassingly simple:

// Execute actions in the queue until it's empty. Actions add *more*
// actions (continuations) to the queue by awaiting this coordinator.
public void Start()
{
    while (actions.Count > 0)
    {
        actions.Dequeue().Invoke();
    }
}

Of course, the interesting bit is the code which supports the async methods and await expressions. We know we need to provide a GetAwaiter() method, but what should that return? Well, we're just going to use the awaiter to add a continuation to the coordinator's queue. It's got no other state than that - so we might as well return the coordinator itself, and put the other infrastructure methods directly in the coordinator.

Again, this is slightly ugly, as the extra methods don't really make sense on the coordinator - we wouldn't want to call them directly from client code, for example. However, they're fairly irrelevant - we could always create a nested type which just had a reference to its "parent" coordinator if we wanted to. For simplicity, I haven't bothered with this - I've just implemented GetAwaiter() trivially:

// Used by await expressions to get an awaiter
public Coordinator GetAwaiter()
{
return this;
}

So, that leaves just three members still to implement: IsCompleted, OnCompleted and GetResult. We always want the IsCompleted property to return false, as otherwise the coroutine will just continue executing immediately without returning to cede control; the await expression would be pointless. OnCompleted just needs to add the continuation to the end of the queue - we don't need to attach it to a task, or anything like that. Finally, GetResult is a no-op - we have no results, no exceptions, and basically nothing to do. You might want to add a bit of logging here, if you were so inclined, but there's no real need.

So, here are the final three members of Coordinator:

// Force await to yield control
public bool IsCompleted { get { return false; } }

public void OnCompleted(Action continuation)
{
    // Put the continuation at the end of the queue, ready to
    // execute when the other coroutines have had a go.
    actions.Enqueue(continuation);
}

public void GetResult()
{
    // Our await expressions are void, and we never need to throw
    // an exception, so this is a no-op.
}

And that's it! Fewer than 50 lines of code required, and nothing complicated at all. The interesting behaviour is all due to the way the C# compiler uses the coordinator when awaiting it.

We need AsyncVoidMethodBuilder as before, as we have some async void methods - but that doesn't need to do anything significant. That's basically all the code required to implement these basic round-robin coroutines.

Conclusion

Our first foray into the weird and wonderful world of coroutines was relatively tame. The basic idea of a coordinator keeping track of the state of all the different coroutines in one sense or another will keep coming back to us, but with different ways of controlling the execution flow.

Next time we'll see some coroutines which can pass data to each other.

Of memory and strings

skeet — Tue, 05 Apr 2011 20:50:47 GMT

This post was provoked by a recent Stack Overflow question which asked whether there was an efficient representation of ASCII strings in .NET.

In particular, the questioner wanted to story hundreds of thousands - possibly millions - of strings in memory, and knowing (or assuming) that they all consisted of ASCII characters, he wanted to avoid the waste of space that comes from storing each character in a .NET string as a UTF-16 code unit.

My answer to the question mostly consisted of saying that I didn't think it would be worth the effort, and giving some reasons. But the more reasons I gave, the more I thought about the subtleties involved, and that it's actually quite an interesting case study into memory use in .NET.

How are we going to measure object size?

If we're going to work out any sort of benefit from a more compact string representation, we'll need to be able to calculate how much memory our objects are taking to start with. Rather than work this out in a purely theoretical way, I've been running tests using code like this:

using System;

class Program
{
    static void Main(string[] args)
    {
        int size = 10000000;
        object[] array = new object[size];

        long before = GC.GetTotalMemory(true);
        for (int i = 0; i < size; i++)
        {
            array[i] = new object();
        }
        long after = GC.GetTotalMemory(true);

        double diff = after - before;

        Console.WriteLine("Per object: " + diff / size);

        // Stop the GC from messing up our measurements
        GC.KeepAlive(array);
    }
}

Obviously that doesn't take into account factors such as memory used by JITting, or anything that could be going on in other threads, but by using a suitably large number of objects, and by performing the division in floating point arithmetic (to avoid a slight variation making an 11.99999 come out as an 11, when it's really just a "12 with something else going on", we can work out the size of objects pretty clearly. The sample above measures the size of a vanilla object, but the code can be adapted very easily.

The first important thing to point out is that C# doesn't guarantee the results of this - it isn't responsible for determining how all of an object is laid out in memory; that's the runtime's job. While there are attributes to affect the layout and padding of the data members of a type in memory, there are other aspects that are out of your control. In this post I won't use any of the layout attributes - we'll just use the defaults.

Not all runtimes are created equal, either. On my laptop I've got Mono 2.8, .NET 3.5 and .NET 4, with the two versions of .NET each having the 32-bit (x86) and 64-bit (x64) CLRs. For the sake of simplicity, I'm going to stick with .NET 4 for this post, but I'll give results for both the x64 and x86 CLRs. To test each of them, I'm compiling with "/platform:x64" or "/platform:x86".

Some simple starter measurements

Before I start creating my own types, let's try a few built-in types, including strings:

Type	x86 size	x64 size
object	12	24
object[]	16 + length * 4	32 + length * 8
int[]	12 + length * 4	28 + length * 4
byte[]	12 + length	24 + length
string	14 + length * 2	26 + length * 2

Note that all the x86 sizes are rounded up to the nearest 4 bytes, and all x64 sizes are rounded up to the nearest 8 bytes.

The string numbers are interesting, because strings are the only non-array types in .NET which vary in size. A long string consists of a single large object in memory. Compare this with Java, where a String is a "normal" type in terms of memory consumption, containing an offset and length into a char array - so a long string consists of a small object referring to a large char array. This distinction will be very important when we come to build an AsciiString type. Another point about measuring string sizes is that it's relatively tricky to measure the size of an empty string - because even if you use the "new string('x', 0)" constructor, the result is still cached - the same reference is returned each time.

You might be forgiven for looking at the numbers above and thinking that the "overhead" of an object is 12 bytes in x86 and 24 in x64... but that's not quite right. Let's build our own straightforward classes and measure them...

Custom classes containing primitives

Here are six classes, all of which are measured with the same simple test code:

class Empty {}
class OneInt32 { int x; }
class TwoInt32 { int x, y; }
class ThreeInt32 { int x, y, z; }

class Mixed1
{
    int x;
    byte b1, b2, b3, b4;
    int y, z;
}

class Mixed2
{
    int x;
    byte b1;
    int y, z;
    byte b2, b3, b4;
}

The last case is mostly to check how the CLR handles an "awkward" class declaration, where the int variables won't naturally be aligned on 4-byte boundaries. The results look odd at first, but we'll make sense of them in a minute:

Type	x86 size	x64 size
Empty	12	24
OneInt32	12	24
TwoInt32s	16	24
ThreeInt32s	20	32
Mixed1	24	32
Mixed2	24	32

A few interesting things to note here:

There's a "base" overhead of 8 bytes per object in x86 and 16 per object in x64... given that we can store an Int32 of "real" data in x86 and still have an object size of 12, and likewise we can store two Int32s of real data in x64 and still have an object of x64.
There's a "minimum" size of 12 bytes and 24 bytes respectively. In other words, you can't have a type which is just the overhead. Note how the "Empty" class takes up the same size as creating instances of Object... there's effectively some spare room, because the CLR doesn't like operating on an object with no data. (Note that a struct with no fields takes up space too, even for local variables.)
The x86 objects are padded to 4 byte boundaries; on x64 it's 8 bytes (just as before)
By default, the CLR is happy to pack fields pretty densely - Mixed2 only took as much space as ThreeInt32. My guess is that it reorganized the in-memory representation so that the bytes all came after the ints... and that's what a quick bit of playing around with unsafe pointers suggests too... but I'm not sufficiently comfortable with this sort of thing to say for sure. Frankly, I don't care... so long as it all works, what we're interested in is the overall size, not the precise layout.

So what does an ASCII string look like?

In this blog post I'm not actually going to implement an ASCII string at all (well, not much). I'm merely pointing out what the data structures would look like. However, it's worth working out what desirable qualities it should have. As far as possible, it should feel like System.String. In particular:

It should be immutable.
It should have fast access to individual characters, and the length.
It should mostly "feel" like an immutable reference type, in that passing a value of type AsciiString around should be cheap, like copying a reference.
It should use as little memory as possible... less than the equivalent string, or it's pointless.
- One caveat to this: in theory that could mean storing 8 characters in every 7 bytes, as ASCII really only uses 7 bits per character. I'm not going to those extremes, but you can think about them if you want.

We're going to store the characters as a byte array. We have three options as to exactly how we handle that byte array:

We could go the Java way, where several strings share references to the same array. Each string then has an offset and a length to say which bit of the array they're interested in.
- Pros: Substring becomes really cheap
- Cons: You can end up having just a tiny substring responsible for keeping a huge character array alive
We could go the .NET way, where each string has its own character data, but the buffer may be longer than necessary... so it stores the length too. (A bit like a List<T>.)
- Pros: Can potentially make building strings cheap, if you just keep whatever potentially oversized buffer you've already got.
- Cons: Wasted space for the unused part of the array, and a field for the length.
We could just have a byte array of exactly the right size - and it already knows its size.

I'm going to assume the third option here. So all the data our type needs is a byte array. That's going to be pretty cheap... we hope. Let's look at what we can build.

Option 1: A simple class

To give a flavour of the implementation, I've decided to implement four members for each option:

A way of creating an AsciiString from a regular string
The Substring overload with both a start and length
The Length property
The indexer returning a char

Hopefully that will give enough of an idea of what's going on to be useful. Note that these aren't production-quality implementations at all... none of the code has ever been run at all. I have made sure it compiles, so just be grateful for that :)

using System;
using System.Text;

public sealed class AsciiString
{
    private readonly byte[] data;

    public AsciiString(string text)
    {
        // TODO: Rather more data validation etc!
        data = Encoding.ASCII.GetBytes(text);
    }

    private AsciiString(byte[] data)
    {
        // This constructor is private: we can trust that it's been called
        // by code which isn't going to modify the contents of the array
        // afterwards.
        this.data = data;
    }

    public AsciiString Substring(int startIndex, int length)
    {
        if (startIndex < 0 || startIndex > data.Length)
        {
            throw new ArgumentOutOfRangeException("startIndex");
        }
        if (startIndex + length > data.Length)
        {
            throw new ArgumentOutOfRangeException("length");
        }
        byte[] newData = new byte[length];
        Buffer.BlockCopy(data, startIndex, newData, 0, length);
        return new AsciiString(newData);
    }

    public int Length { get { return data.Length; } }

    public char this[int position] { get { return (char) data[position]; } }
    // etc...
}

Hopefully this is pretty straightforward - it's meant to be the most "obvious" solution. Note that we've not got the nice locality of reference which the real String class has - it's possible that the an AsciiString could end up with its backing array a long way away in memory, so a indexer operation for a single character could end up with three cache misses - one for the AsciiString object, one for part of the data array storing the length (for argument validation) and one for the part of the data array containing the character we're looking for. That may be unlikely, and it's not the kind of thing I normally think about - but it's probably the kind of thing the BCL team pay a lot of attention to.

We get the same "immutable reference type" behaviour present in the normal string type, however - you can have a null AsciiString reference just as normal, any assignments will just be reference assignments, etc.

What about the size though? There are two objects to consider:

The array, of size 12 + length or 24 + length (x86 and x64 respectively; rounded up to 4 or 8 bytes as well)
The object itself, of size 12 or 24

So we've got a total size of 24 + length or 48 + length, depending on architecture. To show how the break-even point works, here's a little table showing the sizes of string and AsciiString for various sizes on both architectures:

Length	string-x86	string-x64	AsciiString-x86	AsciiString-x64
0	16	32	24	48
1	16	32	28	56
2	20	32	28	56
3	20	32	28	56
4	24	40	28	56
5	24	40	32	56
6	28	40	32	56
7	28	40	32	56
8	32	48	32	56
9	32	48	36	64
10	36	48	36	64
...	..	...	...	...
16	48	64	40	64
24	64	80	48	72
32	80	96	56	80

As you can see, the break-even point in x86 is at length 10; in x64 it's at length 16. After that, we start winning - as we'd expect. The penalty for very small strings is quite hefty though - you'd really better hope you didn't have lots of single-character strings, taking 56 bytes each in x64.

Let's see if we can do better...

Option 2: A simple struct

A lot of the overhead here has come from the fact that we've got an object which only has a single field. The field is all we're interested in... why are we bothering with all the overhead of the object? Let's make it a struct instead, effectively inlining that field wherever we use the type. Assignment, passing arguments to methods etc will still only be copying a reference - it's just the reference will be the byte array rather than a wrapper object.

It all sounds good, but there are two snags:

The value can never be null; that at least diverges from the familiar string behaviour
We won't be able to prevent code from creating an instance of our struct with new AsciiString() - and that won't be good.

We can actually pit these two downsides against each other by making the "default" value a pseudo-null value... we can even throw NullReferenceException just as if it were a reference type. We don't even need to do any work in order to get that NullReferenceException - every member is going to use the data array anyway, and dereferencing that will automatically throw an exception. We might want to change things around a bit to make that the very first thing that can throw an exception, but that's all.

It's nasty, but it appeals very slightly. In an evil kind of way. It makes things slightly more familiar, but at the cost of being generally weird in other ways.

We still need to be able to check whether an AsciiString value is the logical null value. I'll add an IsNull property for that purpose. (An alternative would be HasValue, but that would be confusing with Nullable<T>.)

Most of the code remains untouched - it looks like this:

public struct AsciiString
{
    private readonly byte[] data;

    public bool IsNull { get { return data == null; } }

    // Remainder of code as before
}

Now let's look at the sizes, which should be a lot more favourable than before. Note that I had to change the size-checking code to create an array of type AsciiStruct[] instead of object[] to avoid boxing. Should we take the size of the array itself into consideration when computing the size of the AsciiString? We haven't when working out the size of string... in each case the size of any individual element will be the size of a reference. For the table below, I haven't included it... but bear in mind that this form of measurement would count the size of most value types (int etc) as 0. It just goes to show that when you talk about the size of a data type, you really need to be very precise in what you mean.

Length	string-x86	string-x64	AsciiString-x86	AsciiString-x64
0	16	32	12	24
1	16	32	16	32
2	20	32	16	32
3	20	32	16	32
4	24	40	16	32
5	24	40	20	32
6	28	40	20	32
7	28	40	20	32
8	32	48	20	32
9	32	48	24	40
10	36	48	24	40
...	..	...	...	...
16	48	64	28	40
24	64	80	36	48
32	80	96	44	56

This time, unsurprisingly, AsciiString is always more space-efficient than the normal string. It just takes a certain amount of holding our noses. Speaking of which...

Option 3: Extension methods on byte[]

Suppose we really, really want to have "proper" null references. We don't really need the struct. We could treat any byte array as an array of ASCII characters, with extension methods like this:

public static class ByteArrayExtensions
{
    public static byte[] Substring(this byte[] data, int startIndex, int length)
    {
        if (startIndex < 0 || startIndex > data.Length)
        {
            throw new ArgumentOutOfRangeException("startIndex");
        }
        if (startIndex + length > data.Length)
        {
            throw new ArgumentOutOfRangeException("length");
        }
        byte[] newData = new byte[length];
        Buffer.BlockCopy(data, startIndex, newData, 0, length);
        return newData;
    }
}

The size is the same as with option 2 - in both cases there's just the byte array, basically. This option is truly horrible in many ways though:

You can no longer tell in your code (just through typing) what's meant to be an AsciiString and what isn't
Kiss immutability goodbye
We can't guarantee that all the characters will be valid ASCII any more
We can't add extension properties or extension indexers
We can't make it implement the interfaces we want it to

Obviously, we'd never take this route. I just thought I'd include it for a regular dose of evil-ness.

Conclusion

Implementing an ASCII-only string representation sounds like it should be an obvious win in terms of memory, at the cost of doing a lot of work that's already been done for us in String. However, the most obvious implementation takes a while to break even in memory usage, compared with the normal string type, due to the "special" nature of string within the CLR. We can't mimic the "stretchiness" of string ourselves. The BCL/CLR teams could, of course, if they really wanted to. I'm not holding my breath though.

If we're dead keen on saving space at the cost of some design wonkiness, we can use a value type instead of a reference type. Other than nullity, it works pretty well... but you have all the disadvantages which go with value types, such as the unavoidable parameterless constructor and the need to watch out for boxing.

Aside from anything else, I hope this was useful as a delve into how much space objects actually take up in .NET - and as a way of highlighting the extra memory used when running in the x64 CLR.

The curious case of the publicity-seeking interface and the shy abstract class

skeet — Sun, 03 Oct 2010 19:05:58 GMT

Noda Time has a guilty secret, and I'm not just talking about the fact that there's been very little progress on it recently. (It's not dead as a project - I have high hopes, when I can put some quality time into it.) This secret is called LocalInstant, and it's a pain in the neck.

One of the nice things about giving talks about an API you're currently writing is that you can see which concepts make sense to people, and which don't - as well as seeing which concepts you're able to explain and which you can't. LocalInstant has been an awkward type to explain right from day 1, and I don't think it's improved much since then. For the purpose of this blog post, you don't actually need to know what it means, but if you're really interested, imagine that it's like a time-zone-less date and time (such as "10:58 on July 2nd 2015" but also missing a calendar system, so you can't really tell what the month is etc. The important point is that it's not just time-zone-less, but it's actually local - so it doesn't represent a single instant in time. Unlike every other concept in Noda Time, I haven't thought of any good analogy between LocalInstant and the real world.

Now, I don't like having types I can't describe easily, and I'd love to just get rid of it completely... but it's actually an incredibly powerful concept to have in the library. Not for users of course, but for the implementation. It's spattered all over the place. Okay, the next best step to removing it is to hide it away from consumers: let's make it internal. Unfortunately, that doesn't work either, because it's referred to in interfaces all the time too. For example, almost every member of ICalendarSystem has LocalInstant as one of its parameters.

The rules around interfaces

Just to recap, every member of an interface - even an internal interface - is implicitly public. That causes some interesting restrictions. Firstly, every type referred to in a public interface must be public. So this would be invalid:

internal struct LocalInstant {}

// Doesn't compile: Inconsistent accessibility
public interface ICalendarSystem
{
LocalInstant GetLocalInstant(int year, int month, int day);
}

So far, so good. It's entirely reasonable that a public member's declaration shouldn't refer to an internal type. Calling code wouldn't understand what LocalInstant was, so how could it possibly use ICalendarSystem sensibly? But suppose we only wanted to declare the interface internally. That should be okay, right? Indeed, the compiler allows the following code:

internal struct LocalInstant {}

// Compiles with no problems
internal interface ICalendarSystem
{
LocalInstant GetLocalInstant(int year, int month, int day);
}

But hang on... isn't GetLocalInstant public? That's what I said earlier, right? So we're declaring a public member using an internal type... which we thought wasn't allowed. Is this a compiler bug?

Well, no. My earlier claim that "a public member's declaration shouldn't refer to an internal type" isn't nearly precise enough. The important aspect isn't just whether the member is declared public - but its accessibility domain. In this case, the accessibility domain of ICalendarSystem.GetLocalInstant is only the assembly, which is why it's a valid declaration.

However, life becomes fun when we try to implement ICalendarSystem in a public class. It's perfectly valid for a public class to implement an internal interface, but we have some problems declaring the method implementing GetLocalInstant. We can't make it a public method, because at that point its accessibility domain would be anything referring to the assembly, but the accessibility domain of LocalInstant itself would still only be the assembly. We can't make it internal, because it's implementing an interface member, which is public.

There is an alternative though: explicit interface implementation. That comes with all kinds of other interesting points, but it does at least compile:

internal struct LocalInstant {}

internal interface ICalendarSystem
{
    LocalInstant GetLocalInstant(int year, int month, int day);
}

public class GregorianCalendarSystem : ICalendarSystem
{
    // Has to be implemented explicitly
    LocalInstant ICalendarSystem.GetLocalInstant(int year, int month, int day);
    {
        // Implementation
    }
}

So, we've got somewhere at this point. We've managed to make a type used within an interface internal, but at the cost of making the interface itself internal, and requiring explicit interface implementation within any public classes implementing the interface.

That could potentially be useful in Noda Time, but it doesn't solve our real LocalInstant / ICalendarSystem problem. We need ICalendarSystem to be public, because consumers need to be able to specify a calendar when they create an instance of ZonedDateTime or something similar. Interfaces are just too demanding in terms of publicity.

Fortunately, we have another option up our sleeves...

Abstract classes to the rescue!

I should come clean at this point and say that generally speaking, I'm an interface weenie. Whenever I need a reusable and testable abstraction, I reach for interfaces by default. I have a general bias against concrete inheritance, including abstract classes. I'm probably a little too harsh on them though... particularly as in this case they do everything I need them to.

In Noda Time, I definitely don't need the ability to implement ICalendarSystem and derive from another concrete class... so making it a purely abstract class will be okay in those terms. Let's see what happens when we try:

internal struct LocalInstant {}

public abstract class CalendarSystem
{
    internal abstract LocalInstant GetLocalInstant(int year, int month, int day);
}

internal class GregorianCalendarSystem : CalendarSystem
{
    internal override LocalInstant GetLocalInstant(int year, int month, int day)
    {
        // Implementation
    }
}

Hoorah! Now we've hidden away LocalInstant but left CalendarSystem public, just as we wanted to. We could make GregorianCalendarSystem public or not, as we felt like it. If we want to make any of CalendarSystem's abstract methods public, then we can do so provided they don't require any internal types. There's on interesting point though: types outside the assembly can't derive from CalendarSystem. It's a little bit as if the class only provided an internal constructor, but with a little bit more of an air of mystery... you can override every method you can actually see, and still get a compile-time error message like this:

OutsideCalendar.cs(1,14): error CS0534: 'OutsideCalendar' does not implement inherited abstract member
'CalendarSystem.GetLocalInstant(int, int, int)'

I can just imagine the author of the other assembly thinking, "But I can't even see that method! What is it? Where is it coming from?" Certainly a case where the documentation needs to be clear. Whereas it's impossible to create an interface which is visible to the outside world but can't be implemented externally, that's precisely the situation we've reached here.

The abstract class is a little bit like an authentication token given by a single-sign-on system. From the outside, it's an opaque item: you don't know what's in it or how it does its job... all you know is that you need to obtain it, and then you can use it to do other things. On the inside, it's much richer - full of useful data and members.

Conclusion

Until recently, I hadn't thought of using abstract classes like this. It would possibly be nice if we could use interfaces in the same way, effectively limiting the implementation to be in the declaring assembly, but letting the interface itself (and some members) be visible externally.

A bigger question is whether this is a good idea in terms of design anyway. If I do make LocalInstant internal, there will be a lot of interfaces which go the same way... or become completely internal. For example, the whole "fields" API of Noda Time could become an implementation detail, with suitable helper methods to fetch things like "how many days are there in the given month." The fields API is an elegant overall design, but it's quite complicated considering the very limited situations in which most callers will use it.

I suspect I will try to go for this "reduced API" for v1, knowing that we can always make things more public later on... that way we give ourselves a bit more flexibility in terms of not having to get everything right first time within those APIs, too.

Part of me still feels uncomfortable with the level of hiding involved - I know other developers I respect deeply who hide as little as possible, for maximum flexibility - but I do like the idea of an API which is really simple to browse.

Aside from the concrete use case of Noda Time, this has proved an interesting exercise in terms of revisiting accessibility and the rules on what C# allows.

Non-iterable collection initializers

skeet — Sat, 18 Sep 2010 07:48:19 GMT

Yesterday on Stack Overflow, I mentioned that sometimes I make a type implement IEnumerable just so that I can use collection initializers with it. In such a situation, I use explicit interface implementation (despite not really needing to - I'm not implementing IEnumerable<T>) and leave it throwing a NotImplementedException. (EDIT: As noted in the comments, throwing NotSupportedException would probably be more appropriate. In many cases it would actually be pretty easy to implement this in some sort of meaningful fashion... although I quite like throwing an exception to indicate that it's not really intended to be treated as a sequence.)

Why would I do such a crazy thing? Because sometimes it's helpful to be able to construct a "collection" of items easily, even if you only want the class itself to really treat it as a collection. As an example, in a benchmarking system you might want to be able to add a load of tests individually, but you never want to ask the "test collection" what tests are in it... you just want to run the tests. The only iteration is done internally.

Now, there's an alternative to collection initializers here: parameter arrays. You can add a "params string[]" or whatever as the final constructor parameter, and simply use the constructor. That works fine in many cases, but it falls down in others:

If you want to be able to add different types of values, without just using "params object[]". For example, suppose we wanted to restrict our values to int, string, DateTime and Guid... you can't do that in a compile-time-safe way using a parameter array.
If you want to be able to constructor composite values from two or more parts, without having to explicitly construct that composite value each time. Think about the difference in readability between using a collection initializer for a dictionary and explicitly constructing a KeyValuePair<TKey, TValue> for each entry.
If you want to be able to use generics to force aspects of type safety. The Add method can be generic, so you could, for example, force two parameters for a single entry to both be of T, but different entries could have different types. This is pretty unusual, but I needed it just the other day :)

Now, it's a bit of a hack to have to "not quite implement" IEnumerable. I've come up with two alternative options. These have the additional benefit of not requiring the method to always be called Add any more. I suspect it still would be in most cases, but flexibility is a bonus.

Option 1: Class level attribute

Instead of just relying on IEnumerable, the compiler could detect an attribute applied to the class, specifying the single method name for all collection initializer methods:

[CollectionInitializerMethod("AddValue")]
public class RestrictedValues
{
    public void AddValue(int x) { ... }

    public void AddValue(string y) { ... }
}

var values = new RestrictedValues
{
    3, "value", 10
};

Option 2: Method level attributes

In this case, each method eligible for use in a collection initializer would be decorated with the attribute:

public class RestrictedValues
{
    [CollectionInitializerMethod]
    public void AddInt32(int x) { ... }

    [CollectionInitializerMethod]
    public void AddString(string y) { ... }
}

var values = new RestrictedValues
{
    3, "value", 10
};

This has the disadvantage that the compiler would need to look at every method in the target class when it found a collection initializer.

Obviously both of these can be made backwardly compatible very easily: the presence of an implementation of IEnumerable with no attributes present would just fall back to using Add.

Option 3: Compiler and language simplicity

(I've added this in response to Peter's comment.)

Okay, let's stick with the Add method. All we need is another way of indicating that you should be able to use collection initializers with a type:

[AllowCollectionInitializers]
public class RestrictedValues
{
public void Add(int x) { ... }

public void Add(string y) { ... }
}

At this point, the changes required to the compiler (and language spec) are really minimal. In the bit of code which detects whether or not you can use a collection initializer, you just need to change from "does this type implement IEnumerable" to "does this type implement IEnumerable or have the relevant attribute defined". I can't think of many possible language changes which would be more localized than that.

And another thing...

One final point. I'd still like the ability for collection initializers to return a value, and for that value to be used for subsequent elements of the initialization - with the final return value being the last-returned value. Any methods with a void return value would be treated as if they returned "this". This would allow you to build immutable collections easily.

Likewise you could decorate methods with a [PropertyInitializerMethod] to allow initialization of immutable types with "WithXyz" methods. Admittedly there's less need for this now that we have optional parameters and named arguments in C# 4 - a lot of the benefits of object initializers are available just with constructors.

Anyway, just a few slightly odd thoughts around initialization for you to ponder over the weekend...

Iterate, damn you!

skeet — Tue, 27 Jul 2010 18:52:23 GMT

Do you know the hardest thing about presenting code with surprising results? It's hard to do so without effectively inviting readers to look for the trick. Not that that's always enough - I failed the last of Neal and Eric's C# puzzlers at NDC, for example. (If you haven't already watched the video, please do so now. It's far more entertaining than this blog post.) Anyway, this one may be obvious to some of you, but there are some interesting aspects even when you've got the twist, as it were.

What does the following code print?

using System;
using System.Collections.Generic;

public class WeirdIterators
{
    static void ShowNext(IEnumerator<int> iterator)
    {
        if (iterator.MoveNext())
        {
            Console.WriteLine(iterator.Current);
        }
        else
        {
            Console.WriteLine("Done");
        }
    }

    static void Main()
    {
        List<int> values = new List<int> { 1, 2 };
        using (var iterator = values.GetEnumerator())
        {
            ShowNext(iterator);
            ShowNext(iterator);
            ShowNext(iterator);
        }
    }
}

If you guessed "1, 2, Done" despite the title of the post and the hints that it was surprising, then you're at least brave and firm in your beliefs. I suspect most readers will correctly guess that it prints "1, 1, 1" - but I also suspect some of you won't have worked out why.

Let's look at the signature of List<T>.GetEnumerator(). We'd expect it to be

public IEnumerator<T> GetEnumerator()

right? That's what IEnumerable<T> says it'll look like. Well, no. List<T> uses explicit interface implementation for IEnumerable<T>. The signature actually looks like this:

public List<T>.Enumerator GetEnumerator()

Hmm... that's an unfamiliar type. Let's have another look in MSDN...

[SerializableAttribute]
public struct Enumerator : IEnumerator<T>,
IDisposable, IEnumerator

(It's nested in List<T> of course.) Now that's odd... it's a struct. You don't see many custom structs around, beyond the familiar ones in the System namespace. And hang on, don't iterators fundamentally have to be mutable.

Ah. "Mutable value type" - a phrase to strike terror into the hearts of right-headed .NET developers everywhere.

So what's going on? If we're losing all the changes to the value, why is it printing "1, 1, 1" instead of throwing an exception due to printing out Current without first moving?

Well, we're fetching the iterator into a variable of type List<int>.Enumerator, and then calling ShowNext() three times. On each call, the value is boxed (creating a copy), and the reference to the box is passed to ShowNext().

Within ShowNext(), the value within the box changes when we call MoveNext() - which is how it's able to get the real first element with Current. So that mutation isn't lost... until we return from the method. The box is now eligible for garbage collection, and no change has been made to the iterator variable's value. On the next call to ShowNext(), a new box is created and we see the first item again...

How can we fix it?

There are various things we can do to fix the code - or at least, to make it display "1, 2, Done". We can then find other ways of breaking it again :)

Change the type of the `values` variable

How does the compiler work out the type of the iterator variable? Why, it looks at the return type of values.GetEnumerator(). And how does it find that? It looks at the type of the values variable, and then finds the GetEnumerator() method. In this case it finds List<int>.GetEnumerator(), so it makes the iterator variable type List<int>.Enumerator.

If suppose just change values to be of type IList<int> (or IEnumerable<int>, or ICollection<int>):

IList<int> values = new List<int> { 1, 2 };

The compiler uses the interface implementation of GetEnumerator() on List<T>. Now that could return a different type entirely - but it actually returns a boxed List<T>.Enumerator. We can see that by just printing out iterator.GetType().

So if it's just returning the same value as before, why does it work?

Well, this time we're boxing once - the iterator gets boxed on its way out of the GetEnumerator() method, and the same box is used for all three calls to ShowNext(). No extra copies are created, and the changes within the box don't get lost.

Change the type of the `iterator` variable

This is exactly the same as the previous fix - except we don't need to change the type of values. We can just explicitly state the type of iterator:

using (IEnumerator<int> iterator = values.GetEnumerator())

The reason this works is the same as before - we box once, and the changes within the box don't get lost.

Pass the `iterator` variable by reference

The initial problem was due to the mutations involved in ShowNext() getting lost due to repeated boxing. We've seen how to solve it by reducing the number of boxing operations down to one, but can we remove them entirely?

Well yes, we can. If we want changes to the value of the parameter in ShowNext() to be propagated back to the caller, we just need to pass the variable by reference. When passing by reference the parameter and argument types have to match exactly of course, so we can't leave the iterator variable being type List<T>.Enumerator without changing the parameter type. Now we could explicitly change the type of the parameter to List<T>.Enumerator - but that would tie our implementation down rather a lot, wouldn't it? Let's use generics instead:

static void ShowNext<T>(ref T iterator)
where T : IEnumerator<int>

Now we can pass iterator by reference and the compiler will infer the type. The interface members (MoveNext() and Current) will be called using constrained calls, so there's no boxing involved...

... except that when you try to just change the method calls to use ref, it doesn't work - because apparently you can't pass a "using variable" by reference. I'd never come across that rule before. Interesting. Fortunately, we can (roughly) expand out the using statement ourselves, like this:

var iterator = values.GetEnumerator();
try
{
    ShowNext(ref iterator);
    ShowNext(ref iterator);
    ShowNext(ref iterator);
}
finally
{
    iterator.Dispose();
}

Again, this fixes the problem - and this time there's no boxing involved.

Let's quickly look at one more example of it not working, before I finish...

Dynamic typing to the anti-rescue

What happens if we change the type of iterator to dynamic (and set everything else back the way it was)? I'll readily admit, I really didn't know what was going to happen here. There are two competing forces:

The dynamic type is often really just object behind the scenes... so it will be boxed once, right? That means the changes within the box won't get lost. (This would give "1, 2, Done")
The dynamic type is in many ways meant to act as if you'd declared a variable of the type which it actually turns out to be at execution time - so in this case it should work as if the variable was of type List<int>.Enumerator, just like our original code. (This would give "1, 1, 1")

What actually happens? I believe it actually boxes the value returned from GetEnumerator() - and then the C# binder ~~DLR~~ makes sure that the value type behaviour is preserved by copying the box before passing it to ShowNext(). In other words, both bits of intuition are right, but the second effectively overrules the first. Wow. (See the comment below from Chris Burrows for more information about this. I'm sure he's right that it's the only design that makes sense. This is a pretty pathological example in various ways.)

Conclusion

Just say "no" to mutable value types. They do weird things.

(Fortunately the vast majority of the time this particular one won't be a problem - it's rare to use iterators explicitly in the first place, and when you do you very rarely pass them to another method.)

First encounters with Reactive Extensions

skeet — Sat, 16 Jan 2010 20:13:42 GMT

I've been researching Reactive Extensions for the last few days, with an eye to writing a short section in chapter 12 of the second edition of C# in Depth. (This is the most radically changed chapter from the first edition; it will be covering LINQ to SQL, IQueryable, LINQ to XML, Parallel LINQ, Reactive Extensions, and writing your own LINQ to Objects operators.) I've watched various videos from Channel 9, but today was the first time I actually played with it. I'm half excited, and half disappointed.

My excited half sees that there's an awful lot to experiment with, and loads to learn about join patterns etc. I'm also looking forward to trying genuine events (mouse movements etc) – so far my tests have been to do with collections.

My disappointed half thinks it's missing something. You see, Reactive Extensions shares some concepts with my own Push LINQ library… except it's had smarter people (no offense meant to Marc Gravell) working harder on it for longer. I'd expect it to be easier to use, and make it a breeze to do anything you could do in Push LINQ. Unfortunately, that's not quite the case.

Subscription model

First, the way that subscription is handled for collections seems slightly odd. I've been imagining two kinds of observable sources:

Genuine "event streams" which occur somewhat naturally – for instance, mouse movement events. Subscribing to such an observable wouldn't do anything to it other than adding subscribers.
Collections (and the like) where the usual use case is "set up the data pipeline, then tell it to go". In that case calling Subscribe should just add the relevant observers, but not actually "start" the sequence – after all, you may want to add more observers (we'll see an example of this in a minute).

In the latter case, I could imagine an extension method to IEnumerable<T> called ToObservable which would return a StartableObservable<T> or something like that – you'd subscribe what you want, and then call Start on the StartableObservable<T>. That's not what appears to happen though – if you call ToObservable(), you get an implementation which iterates over the source sequence as soon as anything subscribes to it – which just doesn't feel right to me. Admittedly it makes life easy in the case where that's really all you want to do, but it's a pain otherwise.

There's a way of working round this in Reactive Extensions: there's Subject<T> which is both an observer and an observable. You can create a Subject<T>, Subscribe all the observers you want (so as to set up the data pipeline) and then subscribe the subject to the real data source. It's not exactly hard, but it took me a while to work out, and it feels a little unwieldy. The next issue was somewhat more problematic.

Blocking aggregation

When I first started thinking about Push LINQ, it was motivated by a scenario from the C# newsgroup: someone wanted to group a collection in a particular way, and then count how many items were in each group. This is effectively the "favourite colour voting" scenario outlined in the link at the top of this post. The problem to understand is that the normal Count() call is blocking: it fetches items from a collection until there aren't any more; it's in control of the execution flow, effectively. That means if you call it in a grouping construct, the whole group has to be available before you call Count(). So, you can't stream an enormous data set, which is unfortunate.

In Push LINQ, I addressed this by making Count() return Future<int> instead of int. The whole query is evaluated, and then you can ask each future for its actual result. Unfortunately, that isn't the approach that the Reactive Framework has taken – it still returns int from Count(). I don't know the reason for this, but fortunately it's somewhat fixable. We can't change Observable of course, but we can add our own future-based extensions:

public static class ObservableEx
{
    public static Task<TResult> FutureAggregate<TSource, TResult>
        (this IObservable<TSource> source,
        TResult seed, Func<TResult, TSource, TResult> aggregation)
    {
        TaskCompletionSource<TResult> result = new TaskCompletionSource<TResult>();
        TResult current = seed;
        source.Subscribe(value => current = aggregation(current, value),
            error => result.SetException(error),
            () => result.SetResult(current));
        return result.Task;
    }

    public static Task<int> FutureMax(this IObservable<int> source)
    {
        // TODO: Make this generic and throw exception on
        // empty sequence. Left as an exercise for the reader.
        return source.FutureAggregate(int.MinValue, Math.Max);
    }

    public static Task<int> FutureMin(this IObservable<int> source)
    {
        // TODO: Make this generic and throw exception on
        // empty sequence. Left as an exercise for the reader.
        return source.FutureAggregate(int.MaxValue, Math.Min);
    }

    public static Task<int> FutureCount<T>(this IObservable<T> source)
    {
        return source.FutureAggregate(0, (count, _) => count + 1);
    }
}

This uses Task<T> from Parallel Extensions, which gives us an interesting ability, as we'll see in a moment. It's all fairly straightforward - TaskCompletionSource<T> makes it very easy to specify a value when we've finished, or indicate that an error occurred. As mentioned in the comments, the maximum/minimum implementations leave something to be desired, but it's good enough for a blog post :)

Using the non-blocking aggregation operators

Now that we've got our extension methods, how can we use them? First I decided to do a demo which would count the number of lines in a file, and find the maximum and minimum line lengths:

public static List<T> ToList<T>(this IObservable<T> source)
{
    List<T> ret = new List<T>();
    source.Subscribe(x => ret.Add(x));
    return ret;
}
private static IEnumerable<string> ReadLines(string filename)
{
    using (TextReader reader = File.OpenText(filename))
    {
        string line;
        while ((line = reader.ReadLine()) != null)
        {
            yield return line;
        }
    }
}
...
var subject = new Subject<string>();
var lengths = subject.Select(line => line.Length);
var min = lengths.FutureMin();
var max = lengths.FutureMax();
var count = lengths.FutureCount();

var source = ReadLines("../../Program.cs");
source.ToObservable(Scheduler.Now).Subscribe(subject);
Console.WriteLine("Count: {0}, Min: {1}, Max: {2}",
                  count.Result, min.Result, max.Result);

As you can see, we use the Result property of a task to find its eventual result - this call will block until the result is ready, however, so you do need to be careful about how you use it. Each line is only read from the file once, and pushed to all three observers, who carry their state around until the sequence is complete, whereupon they publish the result to the task.

I got this working fairly quickly - then went back to the "grouping lines by line length" problem I'd originally set myself. I want to group the lines of a file by their length (all lines of length 0, all lines of length 1 etc) and count each group. The result is effectively a histogram of line lengths. Constructing the query itself wasn't a problem - but iterating through the results was. Fundamentally, I don't understand the details of ToEnumerable yet, particularly the timing. I need to look into it more deeply, but I've got two alternative solutions for the moment.

The first is to implement my own ToList extension method. This simply creates a list and subscribes an observer which adds items to the list as it goes. There's no attempt at "safety" here - if you access the list before the source sequence has completed, you'll see whatever has been added so far. I am still just experimenting :) Here's the implementation:

public static List<T> ToList<T>(this IObservable<T> source)
{
    List<T> ret = new List<T>();
    source.Subscribe(x => ret.Add(x));
    return ret;
}

Now we can construct a query expression, project each group using our future count, make sure we've finished pushing the source before we read the results, and everything is fine:

Note how the call to ToList is required before calling source.ToObservable(...).Subscribe - otherwise everything would have been pushed before we started collecting it.

All well and good... but there's another way of doing it too. We've only got a single task being produced for each group - instead of waiting until everything's finished before we dump the results to the console, we can use Task.ContinueWith to write it (the individual group result) out as soon as that group has been told that it's finished. We force this extra action to occur on the same thread as the observer just to make things easier in a console app... but it all works very neatly:

Conclusion

That's the lot, so far. It feels like I'm sort of in the spirit of Reactive Extensions, but that maybe I'm pushing it (no pun intended) in a direction which Erik and Wes either didn't anticipate, or at least don't view as particularly valuable/elegant. I very much doubt that they didn't consider deferred aggregates - it's much more likely that either I've missed some easy way of doing this, or there are good reasons why it's a bad idea. I hope to find out which at some point... but in the meantime, I really ought to work out a more idiomatic example for C# in Depth.

"Magic" null argument testing

skeet — Wed, 09 Dec 2009 18:04:47 GMT

Warning: here be dragons. I don't think this is the right way to check for null arguments, but it was an intriguing idea.

Today on Stack Overflow, I answered a question about checking null arguments. The questioner was already using an extension similar to my own one in MiscUtil, allowing code like this:

public void DoSomething(string name)
{
name.ThrowIfNull("name");

// Normal code here
}

That's all very well, but it's annoying to have to repeat the name part. Now in an ideal world, I'd say it would be nice to add an attribute to the parameter and have the check performed automatically (and when PostSharp works with .NET 4.0, I'm going to give that a go, mixing Code Contracts and AOP…) – but for the moment, how far can we go with extension methods?

I stand by my answer from that question – the code above is the simplest way to achieve the goal for the moment… but another answer raised the interesting prospect of combining anonymous types, extension methods, generics, reflection and manually-created expression trees. Now that's a recipe for hideous code… but it actually works.

The idea is to allow code like this:

public void DoSomething(string name, string canBeNull, int foo, Stream input)
{
new { name, input }.CheckNotNull();

// Normal code here
}

That should check name and input, in that order, and throw an appropriate ArgumentNullException - including parameter name - if one of them is null. It uses the fact that projection initializers in anonymous types use the primary expression's name as the property name in the generated type, and the value of that expression ends up in the instance. Therefore, given an instance of the anonymous type initializer like the above, we have both the name and value despite having only typed it in once.

Now obviously this could be done with normal reflection – but that we be slow as heck. No, we want to effectively find the properties once, and generate strongly typed delegates to perform the property access. That sounds like a job for Delegate.CreateDelegate, but it's not quite that simple… to create the delegate, we'd need to know (at compile time) what the property type is. We could do that with another generic type, but we can do better than that. All we really need to know about the value is whether or not it's null. So given a "container" type T, we'd like a bunch of delegates, one for each property, returning whether that property is null for a specified instance – i.e. a Func<T, bool>. And how do we build delegates at execution time with custom logic? We use expression trees…

I've now implemented this, along with a brief set of unit tests. The irony is that the tests took longer than the implementation (which isn't very unusual) – and so did writing it up in this blog post. I'm not saying that it couldn't be improved (and indeed in .NET 4.0 I could probably make the delegate throw the relevant exception itself) but it works! I haven't benchmarked it, but I'd expect it to be nearly as fast as manual tests – insignificant in methods that do real work. (The same wouldn't be true using reflection every time, of course.)

The full project including test cases is now available, but here's the (almost completely uncommented) "production" code.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Linq.Expressions;

public static class Extensions
{
    public static void CheckNotNull<T>(this T container) where T : class
    {
        if (container == null)
        {
            throw new ArgumentNullException("container");
        }
        NullChecker<T>.Check(container);
    }

    private static class NullChecker<T> where T : class
    {
        private static readonly List<Func<T, bool>> checkers;
        private static readonly List<string> names;

        static NullChecker()
        {
            checkers = new List<Func<T, bool>>();
            names = new List<string>();
            // We can't rely on the order of the properties, but we
            // can rely on the order of the constructor parameters
            // in an anonymous type - and that there'll only be
            // one constructor.
            foreach (string name in typeof(T).GetConstructors()[0]
                                             .GetParameters()
                                             .Select(p => p.Name))
            {
                names.Add(name);
                PropertyInfo property = typeof(T).GetProperty(name);
                // I've omitted a lot of error checking, but here's
                // at least one bit...
                if (property.PropertyType.IsValueType)
                {
                    throw new ArgumentException
                        ("Property " + property + " is a value type");
                }
                ParameterExpression param = Expression.Parameter(typeof(T), "container");
                Expression propertyAccess = Expression.Property(param, property);
                Expression nullValue = Expression.Constant(null, property.PropertyType);
                Expression equality = Expression.Equal(propertyAccess, nullValue);
                var lambda = Expression.Lambda<Func<T, bool>>(equality, param);
                checkers.Add(lambda.Compile());
            }
        }

        internal static void Check(T item)
        {
            for (int i = 0; i < checkers.Count; i++)
            {
                if (checkers[i](item))
                {
                    throw new ArgumentNullException(names[i]);
                }
            }
        }
    }
}

Oh, and just as a miracle – the expression tree worked first time. I'm no Marc Gravell, but I'm clearly improving :)

Update: Marc Gravell pointed out that the order of the results of Type.GetProperties isn't guaranteed - something I should have remembered myself. However, the order of the constructor parameters will be the same as in the anonymous type initialization expression, so I've updated the code above to reflect that. Marc also showed how it could almost all be put into a single expression tree which returns either null (for no error) or the name of the "failing" parameter. Very clever :)

Contract classes and nested types within interfaces

skeet — Sat, 31 Oct 2009 22:04:54 GMT

I've just been going through some feedback for the draft copy of the second edition of C# in Depth. In the contracts section, I have an example like this:

[ContractClass(typeof(ICaseConverterContracts))]
public interface ICaseConverter
{
    string Convert(string text);
}

[ContractClassFor(typeof(ICaseConverter))]
internal class ICaseConverterContracts : ICaseConverter
{
    string ICaseConverter.Convert(string text)
    {
        Contract.Requires(text != null);
        Contract.Ensures(Contract.Result<string>() != null);
        return default(string);
    }

    private ICaseConverterContracts() {}
}

public class InvariantUpperCaseFormatter : ICaseConverter
{
    public string Convert(string text)
    {
        return text.ToUpperInvariant();
    }
}

The point is to demonstrate how contracts can be specified for interfaces, and then applied automatically to implementations. In this case, ICaseConverter is the interface, ICaseConverterContracts is the contract class which specifies the contract for the interface, and InvariantUpperCaseFormatter is the real implementation. The binary rewriter effectively copies the contract into each implementation, so you don't need to duplicate the contract in the source code.

The reader feedback asked where the contract class code should live - should it go in the same file as the interface itself, or in a separate file as normal? Now normally, I'm firmly of the "one top-level type per file" persuasion, but in this case I think it makes sense to keep the contract class with the interface. It has no meaning without reference to the interface, after all - it's not a real implementation to be used in the normal way. It's essentially metadata. This does, however, leave me feeling a little bit dirty. What I'd really like to be able to do is nest the contract class inside the interface, just like I do with other classes which are tightly coupled to an "owner" type. Then the code would look like this:

[ContractClass(typeof(ICaseConverterContracts))]
public interface ICaseConverter
{
    string Convert(string text);

    [ContractClassFor(typeof(ICaseConverter))]
    internal class ICaseConverterContracts : ICaseConverter
    {
        string ICaseConverter.Convert(string text)
        {
            Contract.Requires(text != null);
            Contract.Ensures(Contract.Result<string>() != null);
            return default(string);
        }

        private ICaseConverterContracts() {}
    }
}

public class InvariantUpperCaseFormatter : ICaseConverter
{
    public string Convert(string text)
    {
        return text.ToUpperInvariant();
    }
}

That would make me feel happier - all the information to do with the interface would be specified within the interface type's code. It's possible that with that as a convention, the Code Contracts tooling could cope without the attributes - if interface IFoo contains a nested class IFooContracts which implements IFoo, assume it's a contract class and handle it appropriately. That would be sweet.

You know the really galling thing? I'm pretty sure VB does allow nested types in interfaces...

Iterating atomically

skeet — Fri, 23 Oct 2009 21:20:00 GMT

The IEnumerable<T> and IEnumerator<T> interfaces in .NET are interesting. They crop up an awful lot, but hardly anyone ever calls them directly - you almost always use a foreach loop to iterate over the collection. That hides all the calls to GetEnumerator(), MoveNext() and Current. Likewise iterator blocks hide the details when you want to implement the interfaces. However, sometimes details matter - such as for this recent Stack Overflow question. The question asks how to create a thread-safe iterator - one that can be called from multiple threads. This is not about iterating over a collection n times independently on n different threads - this is about iterating over a collection once without skipping or duplicating. Imagine it's some set of jobs that we have to complete. We assume that the iterator itself is thread-safe to the extent that calls from different threads at different times, with intervening locks will be handled reasonably. This is reasonable - basically, so long as it isn't going out of its way to be thread-hostile, we should be okay. We also assume that no-one is trying to write to the collection at the same time.

Sounds easy, right? Well, no... because the IEnumerator<T> interface has two members which we effectively want to call atomically. In particular, we don't want the collection { "a", "b" } to be iterated like this:

Thread 1	Thread 2
MoveNext()
	MoveNext()
Current
	Current

That way we'll end up not processing the first item at all, and the second item twice.

There are two ways of approaching this problem. In both cases I've started with IEnumerable<T> for consistency, but in fact it's IEnumerator<T> which is the interesting bit. In particular, we're not going to be able to iterate over our result anyway, as each thread needs to have the same IEnumerator<T> - which it won't do if each of them uses foreach (which calls GetEnumerator() to start with).

Fix the interface

First we'll try to fix the interface to look how it should have looked to start with, at least from the point of view of atomicity. Here are the new interfaces:

public interface IAtomicEnumerable<T>
{
IAtomicEnumerator<T> GetEnumerator();
}

public interface IAtomicEnumerator<T>
{
bool TryMoveNext(out T nextValue);
}

One thing you may notice is that we're not implementing IDisposable. That's basically because it's a pain to do so when you think about a multi-threaded environment. Indeed, it's possibly one of the biggest arguments against something of this nature. At what point do you dispose? Just because one thread finished doesn't mean that the rest of them have... don't forget that "finish" might mean "an exception was thrown while processing the job, I'm bailing out". You'd need some sort of co-ordinator to make sure that everyone is finished before you actually do any clean-up. Anyway, the nice thing about this being a blog post is we can ignore that little thorny issue :)

The important point is that we now have a single method in IAtomicEnumerator<T> - TryMoveNext, which works the way you'd expect it to. It atomically attempts to move to the next item, returns whether or not it succeeded, and sets an out parameter with the next value if it did succeed. Now there's no chance of two threads using the method and stomping on each other's values (unless they're silly and use the same variable for the out parameter).

It's reasonably easy to wrap the standard interfaces in order to implement this interface:

/// <summary>
/// Wraps a normal IEnumerable[T] up to implement IAtomicEnumerable[T].
/// </summary>
public sealed class AtomicEnumerable<T> : IAtomicEnumerable<T>
{
    private readonly IEnumerable<T> original;

    public AtomicEnumerable(IEnumerable<T> original)
    {
        this.original = original;
    }

    public IAtomicEnumerator<T> GetEnumerator()
    {
        return new AtomicEnumerator(original.GetEnumerator());
    }

    /// <summary>
    /// Implementation of IAtomicEnumerator[T] to wrap IEnumerator[T].
    /// </summary>
    private sealed class AtomicEnumerator : IAtomicEnumerator<T>
    {
        private readonly IEnumerator<T> original;
        private readonly object padlock = new object();

        internal AtomicEnumerator(IEnumerator<T> original)
        {
            this.original = original;
        }

        public bool TryMoveNext(out T value)
        {
            lock (padlock)
            {
                bool hadNext = original.MoveNext();
                value = hadNext ? original.Current : default(T);
                return hadNext;
            }
        }
    }
}

Just ignore the fact that I never dispose of the original IEnumerator<T> :)

We use a simple lock to make sure that MoveNext() and Current always happen together - that nothing else is going to call MoveNext() between our TryMoveNext() calling it, and it fetching the current value.

Obviously you'd need to write your own code to actually use this sort of iterator, but it would be quite simple:

T value;
while (iterator.TryMoveNext(out value))
{
// Use value
}

However, you may already have code which wants to use an IEnumerator<T>. Let's see what else we can do.

Using thread local variables to fake it

.NET 4.0 has a very useful type called ThreadLocal<T>. It does basically what you'd expect it to, with nice features such as being able to supply a delegate to be executed on each thread to provide the initial value. We can use a thread local to make sure that so long as we call both MoveNext() and Current atomically when we're asked to move to the next element, we can get back the right value for Current later on. It has to be thread local because we're sharing a single IEnumerator<T> across multiple threads - each needs its own separate storage.

This is also the approach we'd use if we wanted to wrap an IAtomicEnumerator<T> in an IEnumerator<T>, by the way. Here's the code to do it:

public class ThreadSafeEnumerable<T> : IEnumerable<T>
{
    private readonly IEnumerable<T> original;

    public ThreadSafeEnumerable(IEnumerable<T> original)
    {
        this.original = original;
    }

    public IEnumerator<T> GetEnumerator()
    {
        return new ThreadSafeEnumerator(original.GetEnumerator());
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    private sealed class ThreadSafeEnumerator : IEnumerator<T>
    {
        private readonly IEnumerator<T> original;
        private readonly object padlock = new object();
        private readonly ThreadLocal<T> current = new ThreadLocal<T>();

        internal ThreadSafeEnumerator(IEnumerator<T> original)
        {
            this.original = original;
        }

        public bool MoveNext()
        {
            lock (padlock)
            {
                bool ret = original.MoveNext();
                if (ret)
                {
                    current.Value = original.Current;
                }
                return ret;
            }
        }

        public T Current
        {
            get { return current.Value; }
        }

        public void Dispose()
        {
            original.Dispose();
            current.Dispose();
        }

        object IEnumerator.Current
        {
            get { return Current; }
        }

        public void Reset()
        {
            throw new NotSupportedException();
        }
    }
}

I'm going to say it one last time - we're broken when it comes to disposal. There's no way of safely disposing of the original iterator at "just the right time" when everyone's finished with it. Oh well.

Other than that, it's quite simple. This code has the serendipitous property of actually implementing IEnumerator<T> slightly better than C#-compiler-generated implementations from iterator blocks - if you call the Current property without having called MoveNext(), this will throw an InvalidOperationException, just as the documentation says it should. (It doesn't do the same at the end, admittedly, but that's fixable if we really wanted to be pedantic.

Conclusion

I found this an intriguing little problem. I think there are better ways of solving the bigger picture - a co-ordinator which takes care of disposing exactly once, and which possibly mediates the original iterator etc is probably the way forward... but I enjoyed thinking about the nitty gritty.

Generally speaking, I prefer the first of these approaches. Thread local variables always feel like a bit of a grotty hack to me - they can be useful, but it's better to avoid them if you can. It's interesting to see how an interface can be inherently thread-friendly or not.

One last word of warning - this code is completely untested. It builds, and I can't immediately see why it wouldn't work, but I'm making no guarantees...

An object lesson in blogging and accuracy; was: Efficient "vote counting" with LINQ to Objects - and the value of nothing

skeet — Sun, 20 Sep 2009 20:48:00 GMT

Well, this is embarrassing.

Yesterday evening, I excitedly wrote a blog post about an interesting little idea for making a particular type of LINQ query (basically vote counting) efficient. It was an idea that had occurred to me a few months back, but I hadn't got round to blogging about it.

The basic idea was to take a completely empty struct, and use that as the element type in the results of a grouping query - as the struct was empty, it would take no space, therefore "huge" arrays could be created for no cost beyond the fixed array overhead, etc. I carefully checked that the type used for grouping did in fact implement ICollection<T> so that the Count method would be efficient; I wrote sample code which made sure my queries were valid... but I failed to check that the empty struct really took up no memory.

Fortunately, I have smart readers, a number of whom pointed out my mistake in very kind terms.

Ben Voigt gave the reason for the size being 1 in a comment:

The object identity rules require a unique address for each instance... identity can be shared with super- or sub- class objects (Empty Base Optimization) but the total size of the instance has to be at least 1.

This makes perfect sense - it's just a shame I didn't realise it before.

Live and learn, I guess - but apologies for the poorly researched post. I'll attempt to be more careful next time.

Generic constraints for enums and delegates

skeet — Thu, 10 Sep 2009 20:09:00 GMT

As most readers probably know, C# prohibits generic type constraints from referring to System.Object, System.Enum, System.Array, System.Delegate and System.ValueType. In other words, this method declaration is illegal:

public static T[] GetValues<T>() where T : struct, System.Enum
{
return (T[]) Enum.GetValues(typeof(T));
}

This is a pity, as such a method could be useful. (In fact there are better things we can do... such as returning a read-only collection. That way we don't have to create a new array each time the method is called.) As far as I can tell, there is no reason why this should be prohibited. Eric Lippert has stated that he believes the CLR doesn't support this - but I think he's wrong. I can't remember the last time I had cause to believe Eric to be wrong about something, and I'm somewhat nervous of even mentioning it, but section 10.1.7 of the CLI spec (ECMA-335) partition II (p40) specifically gives examples of type parameter constraints involving System.Delegate and System.Enum. It introduces the table with "The following table shows the valid combinations of type and special constraints for a representative set of types." It was only due to reading this table that I realized that the value type constraint on the above is required (or a constructor constraint would do equally well) - otherwise System.Enum itself satisfies the constraint, which would be a Bad Thing.

It's possible (but unlikely) that the CLI doesn't fully implement this part of the CLR spec. I'm hoping that Eric's just wrong on this occasion, and that actually there's nothing to stop the C# language from allowing such constraints in the future. (It would be nice to get keyword support, such that a constraint of "T : enum" would be equivalent to the above, but hey...)

The good news is that ilasm/ildasm have no problem with this. The better news is that if you add a reference to a library which uses those constraints, the C# compiler applies them sensibly, as far as I can tell...

Introducing UnconstrainedMelody

(Okay, the name will almost surely have to change. But I like the idea of it removing the constraints of C# around which constraints are valid... and yet still being in the key of C#. Better suggestions welcome.)

I have a plan - I want to write a utility library which does useful things for enums and delegates (and arrays if I can think of anything sensible to do with them). It will be written in C#, with methods like this:

public static T[] GetValues<T>() where T : struct, IEnumConstraint
{
return (T[]) Enum.GetValues(typeof(T));
}

(IEnumConstraint has to be an interface of course, as otherwise the constraint would be invalid.)

As a post-build step, I will:

Run ildasm on the resulting binary
Replace every constraint using EnumConstraint with System.Enum
Run ilasm to build the binary again

If anyone has a simple binary rewriter (I've looked at PostSharp and CCI; both look way more complicated than the above) which would do this, that would be great. Otherwise ildasm/ilasm will be fine. It's not like consumers will need to perform this step.

As soon as the name is finalized I'll add a project on Google Code. Once the infrastructure is in place, adding utility methods should be very straightforward. Suggestions for utility methods would be useful, or just join the project when it's up and running.

Am I being silly? Have I overlooked something?

A couple of hours later...

Okay, I decided not to wait for a better name. The first cut - which does basically nothing but validate the idea, and the fact that I can still unit test it - is in. The UnconstrainedMelody Google Code project is live!

The "dream book" for C# and .NET

skeet — Thu, 20 Aug 2009 16:57:40 GMT

This morning I showed my hand a little on Twitter. I've had a dream for a long time about the ultimate C# book. It's a dream based on Effective Java, which is my favourite Java book, along with my experiences of writing C# in Depth.

Effective Java is written by Josh Bloch, who is an absolute giant in the Java world... and that's both the problem and the opportunity. There's no-one of quite the equivalent stature in the .NET world. Instead, there are many very smart people, a lot of whom blog and some of whom have their own books.

There are "best practices" books, of course: Microsoft's own Framework Design Guidelines, and Bill Wagner's Effective C# and More Effective C# being the most obvious examples. I'm in no way trying to knock these books, but I feel we could do even better. The Framework Design Guidelines (also available free to browse on MSDN) are really about how to create a good API - which is important, but not the be-all-and-end-all for many application developers who aren't trying to ship a reusable class library and may well have different concerns. They want to know how to use the language most effectively, as well as the core types within the framework.

Bill's books - and many others which cover the core framework, such as CLR via C#, Accelerated C# 2008 and C# 3.0 in a Nutshell - give plenty of advice, but often I've felt it's a little one-sided. Each of these books is the work of a single person (or brothers in the case of Nutshell). Reading them, I've often wanted to give present a different point of view - or alternatively, to give a hearty "hear, hear." I believe that a book giving guidance would benefit greatly from being more of a conversation: where the authors all agree on something, that's great; where they differ, it would be good to hear about the pros and cons of various approaches. The reader can then weigh up those factors as they apply to each particular real-world scenario.

Scope

So what would such a book contain? Opinions will vary of course, but I would like to see:

Effective ways of using language features such as lambda expressions, generic type inference (and indeed generics in general), optional parameters, named arguments and extension methods. Assume that the reader knows roughly what C# does, but give some extra details around things like iterator blocks and anonymous functions.
Guidance around class design (in a similar fashion to the FDG, but with more input from others in the community)
Core framework topics (again, assume the basics are understood):

Resource management (disposal etc)
Exceptions
Collections (including LINQ fundamentals)
Streams
Text (including internationalization)
Numeric types
Time-related APIs
Concurrency
Contracts
AppDomains
Security
Performance

I would prefer to avoid anything around the periphery of .NET (WPF, WinForms, ASP.NET, WCF) - I believe those are better handled in different topics.

Obstacles and format

There's one big problem with this idea, but I think it may be a saving grace too. Many of the leading authors work for different publishers. Clearly no single publisher is going to attract all the best minds in the C# and .NET world. So how could this work in practice? Well...

Imagine a web site for the book, paid for jointly by all interested publishers. The web site would be the foremost delivery mechanism for the content, both to browse and probably to download in formats appropriate for offline reading (PDF etc). The content would be edited in a collaborative style obviously, but exactly how that would work is a detail to be thrashed out. If you've read the annotated C# or CLI specifications, they have about the right feel - opinions can be attributed in places, but not everything has a label.

Any contributing publisher could also take the material and publish it as hard copy if they so wished. Quite how this would work - with potentially multiple hard copy editions of the same content - would be interesting to see. There's another reason against hard copy ever appearing though, which is that it would be immovable. I'd like to see this work evolve as new features appear and as more best practices are discovered. Publishers could monetize the web site via adverts, possibly according to how much they're kicking into the site.

I don't know how the authors would get paid, admittedly, and that's another problem. Would this cannibalize the sales of the books listed earlier? It wouldn't make them redundant - certainly not for the Nutshell type of book, which teaches the basics as well as giving guidance. It would hit Effective C# harder, I suspect - and I apologise to Bill Wagner in advance; if this ever takes off and it hurts his bottom line, I'm very sorry - I think it's in a good cause though.

Dream Team

So who would contribute to this? Part of me would like to say "anyone and everyone" in a Wikipedia kind of approach - but I think that practically, it makes sense for industry experts to take their places. (A good feedback/comments mechanism for anyone to use would be crucial, however.) Here's a list which isn't meant to be exhaustive, but would make me happy - please don't take offence if your name isn't on here but should be, and I wouldn't expect all of these people to be interested anyway.

Anders Hejlsberg
Eric Lippert
Mads Torgersen
Don Box
Brad Abrams
Krzysztof Cwalina
Joe Duffy
Vance Morrison
Rico Mariani
Erik Meijer
Don Symes
Wes Dyer
Jeff Richter
Joe and Ben Albahari
Andrew Troelsen
Bill Wagner
Trey Nash
Mark Michaelis
Jon Skeet (yeah, I want to contribute if I can)

I imagine "principal" authors for specific topics (e.g. Joe Duffy for concurrency) but with all the authors dropping in comments in other places too.

Dream or reality?

I have no idea whether this will ever happen or not. I'd dearly love it to, and I've spoken to a few people before today who've been encouraging about the idea. I haven't been putting any work into getting it off the ground - don't worry, it's not been delaying the second edition of C# in Depth. One day though, one day...

Am I being hopelessly naïve to even consider such a venture? Is the scope too broad? Is the content valuable but not money-making? We'll see.

Faking COM to fool the C# compiler

skeet — Tue, 07 Jul 2009 22:28:46 GMT

C# 4 has some great features to make programming against COM components ~~bearable~~ fun and exciting. In particular:

PIA linking allows you to embed just the relevant bits of the Primary Interop Assembly into your own assembly, so the PIA isn't actually required at execution time
Named arguments and optional parameters make life much simpler for APIs like Office which are full of methods with gazillions of parameters
"ref" removal allows you to pass an argument by value even though the parameter is a by-reference parameter (COM only, folks - don't worry!)
Dynamic typing allows you to remove a load of casts by converting every parameter and return type of "object" into "dynamic" (if you're using PIA linking)

I'm currently writing about these features for the book (don't forget to buy it cheap on Friday) but I'm not really a COM person. I want to be able to see these compiler features at work against a really simple type. Unfortunately, these really are COM-specific features... so we're going to have to persuade COM that the type really is a COM type.

I got slightly stuck on this first, but thanks to the power of Stack Overflow, I now have a reasonably complete demo "fake" COM type. It doesn't do a lot, and in particular it doesn't have any events, but it's enough to show the compiler features:

using System;
using System.Runtime.InteropServices;

// Required for linking into another assembly (C# 4)
[assembly:Guid("86ca55e4-9d4b-462b-8ec8-b62e993aeb64")]
[assembly:ImportedFromTypeLib("fake.tlb")]

namespace FakeCom
{
    [Guid("c3cb8098-0b8f-4a9a-9772-788d340d6ae0")]
    [ComImport, CoClass(typeof(FakeImpl))]
    public interface FakeComponent
    {
        object MakeMeDynamic(object arg);

        void Foo([Optional] ref int x,
                 [Optional] ref string y);
    }

    [Guid("734e6105-a20f-4748-a7de-2c83d7e91b04")]
    public class FakeImpl {}
}

We have an interface representing our COM type, and a class which the interface claims will implement it. Fortunately the compiler doesn't actually check that, so we can get away with leaving it entirely unimplemented. It's also worth noting that our optional parameters can be by-reference parameters (which you can't normally do in C# 4) and we haven't given them any default values (as those are ignored for COM anyway).

This is compiled just like any other assembly:

csc /target:library FakeCom.cs

Then we get to use it with a test program:

using FakeCom;

class Test
{
    static void Main()
    {
        // Yes, that is calling a "constructor" on an interface
        FakeComponent com = new FakeComponent();

        // The boring old fashioned way of calling a method
        int i = 0;
        string j = null;
        com.Foo(ref i, ref j);

        // Look ma, no ref!
        com.Foo(10, "Wow!");

        // Who cares about parameter ordering?
        com.Foo(y: "Not me", x: 0);

        // And the parameters are optional too
        com.Foo();

        // The line below only works when linked rather than
        // referenced, as otherwise you need a cast.
        // The compiler treats it as if it both takes and
        // returns a dynamic value.
        string value = com.MakeMeDynamic(10);
    }
}

This is compiled either in the old "deploy the PIA as well" way (after adding a cast in the last line):

csc /r:FakeCom.dll Test.cs

... or by linking the PIA instead:

csc /l:FakeCom.dll Test.cs

(The difference is just using /l instead of /r.)

When the test code is compiled as a reference, it decompiles in Reflector to this (I've added whitespace for clarity):

private static void Main()
{
    FakeComponent component = (FakeComponent) new FakeImpl();

    int x = 0;
    string y = null;
    component.Foo(ref x, ref y);

    int num2 = 10;
    string str3 = "Wow!";
    component.Foo(ref num2, ref str3);

    string str4 = "Not me";
    int num3 = 0;
    component.Foo(ref num3, ref str4);

    int num4 = 0;
    string str5 = null;
    component.Foo(ref num4, ref str5);

    string str2 = (string) component.MakeMeDynamic(10);
}

Note how the compiler has created local variables to pass by reference; any changes to the parameter are ignored when the method returns. (If you actually pass a variable by reference, the compiler won't take that away, however.)

When the code is linked instead, the middle section is the same, but the construction and the line calling MakeMeDynamic are very different:

private static void Main()
{
    FakeComponent component = (FakeComponent) Activator.CreateInstance(Type.GetTypeFromCLSID
        (new Guid("734E6105-A20F-4748-A7DE-2C83D7E91B04")));

    // Middle bit as before

    if (<Main>o__SiteContainer6.<>p__Site7 == null)
    {
        <Main>o__SiteContainer6.<>p__Site7 = CallSite<Func<CallSite, object, string>>
            .Create(new CSharpConvertBinder
                       (typeof(string),
                        CSharpConversionKind.ImplicitConversion, false));
    }
    string str2 = <Main>o__SiteContainer6.<>p__Site7.Target.Invoke
        (<Main>o__SiteContainer6.<>p__Site7, component.MakeMeDynamic(10));
}

The interface is embedded in the generated assembly, but with a slightly different set of attributes:

[ComImport, CompilerGenerated]
[Guid("C3CB8098-0B8F-4A9A-9772-788D340D6AE0"), TypeIdentifier]
public interface FakeComponent
{
object MakeMeDynamic(object arg);
void Foo([Optional] ref int x, [Optional] ref string y);
}

The class isn't present at all.

I should point out that doing this has no practical benefit in real code - but the ability to mess around with a pseudo-COM type rather than having to find a real one with the exact members I want will make it a lot easier to try a few corner cases for the book.

So, not a terribly productive evening in terms of getting actual writing done, but interesting nonetheless...

A different approach to inappropriate defaults

skeet — Thu, 14 May 2009 17:03:52 GMT

I've had a couple of bug reports about my Protocol Buffers port - both nicely detailed, and one including a patch to fix it. (It's only due to my lack of timeliness in actually submitting the change that the second bug report occurred. Oops.)

The bug was in text formatting (although it also affected parsing). I was using the default ToString behaviour for numbers, which meant that floats and doubles were being formatted as "50,15" in Germany instead of "50.15". The unit tests caught this, but only if you ran them on a machine with an appropriate default culture.

Aaargh. I've been struggling with a similar problem in a library I can't change, which uses the system default time zone for various calculations in Java. When you're running server code, the default time zone is almost never the one you want to use, and it certainly isn't in my case.

A similar problem is Java's decision to use the system default encoding in all kinds of bizarre places - FileReader doesn't even let you specify the encoding, which makes it almost entirely useless in my view.

So I've been wondering how we could fix this and problems like it. One option is to completely remove the defaults. If you always had to pass in a CultureInfo/Locale, TimeZoneInfo/TimeZone, Encoding/Charset when you call any method which might be culturally sensitive.

Making life easier (in .NET)

It strikes me that .NET has a useful abstraction here: the assembly as the unit of deployment. (Java's closest equivalent is probably a jar file, which probably gets messier.)

Within one assembly, I suspect in many cases you always want to make the same decision. For example, in protocol buffers I would like to use the invariant culture all the time. It would be nice if I could say that, and then get the right behaviour by default. Here are the options I'd like to be able to apply (for each of culture, time zone and character encoding - there may be others):

Use a culture-neutral default (the invariant culture, UTF-8, UTC)
Use a specific set of values (e.g. en-GB, Windows-1252, "Europe/London")
Use the system default values
Use whatever the calling assembly is using

Of course you should still have the option of specifying overrides on a per call basis, but I think this might be a way forward.

Thoughts? I realise it's almost certainly too late for this to actually be implemented now, but would it have been a good idea? Or is it just an alternative source of confusion?

Breaking Liskov

skeet — Mon, 16 Mar 2009 17:48:04 GMT

Very recently, Barbara Liskov won the Turing award, which makes it a highly appropriate time to ponder when it's reasonable to ignore her most famous piece of work, the Liskov Substitution (or Substitutability) Principle. This is not idle speculation: I've had a feature request for MiscUtil. The request makes sense, simplifies the code, and is good all round - but it breaks substitutability and documented APIs.

The substitutability principle is in some ways just common sense. It says (in paraphrase) that if your code works for some base type T, it should be able to work with subtype of T, S. If it doesn't, S is breaking substitutability. This principle is at the heart of inheritance and polymorphism - I should be able to use a Stream without knowing the details of what its underlying storage is, for example.

Liskov's formulation is:

Let q(x) be a property provable about objects x of type T. Then q(y) should be true for objects y of type S where S is a subtype of T.

So, that's the rule. Sounds like a good idea, right?

Breaking BinaryReader's contract

My case in point is EndianBinaryReader (and EndianBinaryWriter, but the arguments will all be the same - it's better to focus on a single type). This is simply an equivalent to System.IO.BinaryReader, but it lets you specify the endianness to use when converting values.

Currently, EndianBinaryReader is a completely separate class to BinaryReader. They have no inheritance relationship. However, as it happens, BinaryReader isn't sealed, and all of the appropriate methods are virtual. So, can we make EndianBinaryReader derive from BinaryReader and use it as a drop-in replacement? Well... that's where the trouble starts.

There's no difficulty technically in doing it. The implementation is fairly straightforward - indeed, it means we can drop a bunch of methods from EndianBinaryReader and let BinaryReader handle it instead. (This is particularly handy for text, which is fiddly to get right.) I currently have the code in another branch, and it works fine.

And I would have gotten away with it if it weren't for that pesky inheritance...

The problem is whether or not it's the right thing to do. To start with, it breaks Liskov's substitutability principle, if the "property" we consider is "the result of calling ReadInt32 when the next four bytes of the underlying stream are 00, 00, 00, 01" for example. Not having read Liskov's paper for myself (I really should, some time) I'm not sure whether this is the intended kind of use or not. More on that later.

The second problem is that it contradicts the documentation for BinaryReader. For example, the docs for ReadInt32 state: "BinaryReader reads this data type in little-endian format." That's a tricky bit of documentation to understand precisely - it's correct for BinaryReader itself, but does that mean it should be true for all subclasses too?

When I've written in various places about the problems of inheritance, and why if you design a class to be unsealed that means doing more design work, this is the kind of thing I've been talking about. How much detail does it make sense to specify here? How much leeway is there for classes overriding ReadInt32? Could a different implementation read a "compressed" Int32 instead of always reading four bytes, for example? Should the client care, if they make sure they've obtained an appropriate BinaryReader for their data source in the first place? This is basically the same as asking how strictly we should apply Liskov's substitutability principle. If two types are the same in every property, surely we can't distinguish between them at all.

I wonder whether most design questions of inheritance basically boil down to defining which properties should obey Liskov's substitutability principle and which needn't, for the type you're designing. Of course, it's not just black and white - there will always be exceptions and awkward points. Programming is often about nuance, even if we might wish that not to be the case.

Blow it, let's do it anyway...

Coming back to BinaryReader, I think (unless I can be persuaded otherwise) that the benefits from going against the documentation (and strict substitutability) outweigh the downsides. In particular, BinaryReaders don't tend to be passed around in my experience - the code which creates it is usually the code which uses it too, or it's at least closely related. The risk of breaking code by passing it a BinaryReader using an unexpected endianness is therefore quite low, even though it's theoretically possible.

So, am I miles off track? This is for a class library, after all - should I be more punctilious about playing by the rules? Or is pragmatism the more important principle here?

Quick rant: why isn't there an Exception(string, params object[]) constructor?

skeet — Fri, 23 Jan 2009 21:42:52 GMT

This Stack Overflow question has reminded me of something I often wish existed in common exception constructors - an overload taking a format string and values. For instance, it would be really nice to be able to write:

throw new IOException("Expected to read {0} bytes but only {1} were available",
requiredSize, bytesRead);

Of course, with no way of explicitly inheriting constructors (which I almost always want for exceptions, and almost never want for anything else) it would mean yet another overload to copy and paste from another exception, but the times when I've actually written it in my own exceptions it's been hugely handy, particularly for tricky cases where you've got a lot of data to include in the message. (You'd also want an overload taking a nested exception first as well, adding to the baggage...)

Horrible grotty hack: returning an anonymous type instance

skeet — Fri, 09 Jan 2009 07:52:14 GMT

One of the reasons I don't view anonymous types as being too bad is that they're nicely confined to methods. You can't declare the type that you're returning from a method if it's anonymous (or if one of its type arguments is generic, e.g. a List<T> where T is an anonymous type and T isn't a type parameter to the method itself). However, you can get around this if you're sneaky.

I've always known that it's perfectly easy to return an instance of an anonymous type by declaring that the method will return object. However, it hadn't occurred to me before today that you can actually cast back to that type afterwards. Of course, you can't just use a normal cast expression - that requires the name of the type to be known at compile-time. But you can do a cast in a generic method... and you can use type inference to supply a type argument... and two anonymous type instance creation expressions will use the same type within the same assembly if the order, names and types of the properties are the same.

Behold the evil cheesecake factory!

using System;

static class GrottyHacks
{
    internal static T Cast<T>(object target, T example)
    {
        return (T) target;
    }
}

class CheesecakeFactory
{
    static object CreateCheesecake()
    {
        return new { Fruit="Strawberry", Topping="Chocolate" };
    }

    static void Main()
    {
        object weaklyTyped = CreateCheesecake();
        var stronglyTyped = GrottyHacks.Cast(weaklyTyped,
            new { Fruit="", Topping="" });

        Console.WriteLine("Cheesecake: {0} ({1})",
            stronglyTyped.Fruit, stronglyTyped.Topping);
    }
}

The important thing to note here is that the stronglyTyped variable really is of the same anonymous type as the one used in the CreateCheesecake method. When we use the Fruit and Topping properties in the last statement, that's checked at compile-time.

Of course, it all goes pear-shaped if you make the slightest of errors when giving the Cast method an example of what you want to cast to - if you got the order of the properties wrong, for example, the code would still compile, but the cast would throw an exception at execution time.

How useful is this? Ooh, probably not at all. Please do not use this "technique" in your code. If you do, at least don't mention my name anywhere near it. It's all fun though, isn't it?

Value types and parameterless constructors

skeet — Wed, 10 Dec 2008 17:55:40 GMT

There have been a couple of questions on StackOverflow about value types and parameterless constructors:

I learned quite a bit when answering both of these. When a further question about the default value of a type (particularly with respect to generics) came up, I thought it would be worth delving into a bit more depth. Very little of this is actually relevant most of the time, but it's interesting nonetheless.

I won't go over most of the details I discovered in my answer to the first question, but if you're interested in the IL generated by the statement "x = new Guid();" then have a look there for more details.

Let's start off with the first and most important thing I've learned about value types recently:

Yes, you can write a parameterless constructor for a value type in .NET

I very carefully wrote "in .NET" there - "in C#" would have been incorrect. I had always believed that the CLI spec prohibited value types from having parameterless constructors. (The C# spec used the terminology in a slightly different way - it treats all value types as having a parameterless constructor. This makes the language more consistent for the most part, but it does give rise to some interesting behaviour which we'll see later on.)

It turns out that if you write your value type in IL, you can provide your own parameterless constructor with custom code without ilasm complaining at all. It's possible that other languages targeting the CLI allow you to do this as well, but as I don't know any, I'll stick to IL. Unfortunately I don't know IL terribly well, so I thought I'd just start off with some C# and go from there:

public struct Oddity
{
    public Oddity(int removeMe)
    {
        System.Console.WriteLine("Oddity constructor called");
    }
}

I compiled that into its own class library, and then disassembled it with ildasm /out:Oddity.il Oddity.dll. After changing the constructor to be parameterless, removing a few comments, and removing some compiler-generated assembly attributes) I ended up with this IL:

.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )  
  .ver 2:0:0:0
}
.assembly Oddity
{
  .hash algorithm 0x00008004
  .ver 0:0:0:0
}
.module Oddity.dll
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003
.corflags 0x00000001

.class public sequential ansi sealed beforefieldinit Oddity
       extends [mscorlib]System.ValueType
{
  .pack 0
  .size 1
  .method public hidebysig specialname rtspecialname 
          instance void  .ctor() cil managed
  {
    .maxstack  8
    IL_0000:  nop
    IL_0001:  ldstr      "Oddity constructor called"
    IL_0006:  call       void [mscorlib]System.Console::WriteLine(string)
    IL_000b:  nop
    IL_000c:  ret
  }
}

I reassembled this with ilasm /dll /out:Oddity.dll Oddity.il. So far, so good. We have a value type with a custom constructor in a class library. It doesn't do anything particularly clever - it just logs that it's been called. That's enough for our test program.

When does the parameterless constructor get called?

There are various things one could investigate about parameterless constructors, but I'm mostly interested in when they get called. The test application is reasonably simple, but contains lots of cases - each writes to the console what it's about to do, then does something which might call the constructor. Without further ado:

using System;
using System.Runtime.CompilerServices;

class Test
{
    static Oddity staticField;
    Oddity instanceField;

    static void Main()
    {
        Report("Declaring local variable");
        Oddity localDeclarationOnly;
        // No variables within the value, so we can use it
        // without inializing anything
        Report("Boxing");
        object o = localDeclarationOnly;
        // Just make sure it's really done it
        Report(o.ToString());
        Report("new Oddity() - set local variable");
        Oddity local = new Oddity();
        Report("Create instance of Test - contains instance variable");
        Test t = new Test();
        Report("new Oddity() - set instance field");
        t.instanceField = new Oddity();
        Report("new Oddity() - set static field");
        staticField = new Oddity();
        Report("new Oddity[10]");
        o = new Oddity[10];
        Report("Passing argument to method");
        MethodWithParameter(local);
        GenericMethod<Oddity>();
        GenericMethod2<Oddity>();
        Report("Activator.CreateInstance(typeof(Oddity))");
        Activator.CreateInstance(typeof(Oddity));
        Report("Activator.CreateInstance<Oddity>()");
        Activator.CreateInstance<Oddity>();
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void MethodWithParameter(Oddity oddity)
    {
        // No need to do anything
    }

    static void GenericMethod<T>() where T : new()
    {
        Report("default(T) in generic method with new() constraint");
        T t = default(T);
        Report("new T() in generic method with new() constraint");
        t = new T();
    }

    static void GenericMethod2<T>() where T : struct
    {
        Report("default(T) in generic method with struct constraint");
        T t = default(T);
        Report("new T() in generic method with struct constraint");
        t = new T();
    }

    static void Report(string text)
    {
        Console.WriteLine(text);
    }
}

And here are the results:

Declaring local variable
Boxing
Oddity
new Oddity() - set local variable
Oddity constructor called
Create instance of Test - contains instance variable
new Oddity() - set instance field
Oddity constructor called
new Oddity() - set static field
Oddity constructor called
new Oddity[10]
Passing argument to method
default(T) in generic method with new() constraint
new T() in generic method with new() constraint
default(T) in generic method with struct constraint
new T() in generic method with struct constraint
Activator.CreateInstance(typeof(Oddity))
Oddity constructor called
Activator.CreateInstance<Oddity>()
Oddity constructor called

So, to split these out:

Operations which do call the constructor

new Oddity() - whatever we're storing the result in. This isn't much of a surprise. What may surprise you is that it gets called even if you compile Test.cs against the original Oddity.dll (without the custom parameterless constructor) and then just rebuild Oddity.dll.
Activator.CreateInstance<T>() and Activator.CreateInstance(Type). I wouldn't be particular surprised by this either way.

Operations which don't call the constructor

Just declaring a variable, whether local, static or instance
Boxing
Creating an array - good job, as this could be a real performance killer
Using default(T) in a generic method - this one didn't surprise me
Using new T() in a generic method - this one really did surprise me. Not only is it counterintuitive, but in IL it just calls Activator.CreateInstance<T>(). What's the difference between this and calling Activator.CreateInstance<Oddity>()? I really don't understand.

Conclusions

Well, I'm still glad that C# doesn't let us define our own parameterless constructors for value types, given the behaviour. The main reason for using it - as far as I've seen - it to make sure that the "default value" for a type is sensible. Given that it's possible to get a usable value of the type without the constructor being called, this wouldn't work anyway. Writing such a constructor would be like making a value type mutable - almost always a bad idea.

However, it's nice to know it's possible, just on the grounds that learning new things is always a good thing. And at least next time someone asks a similar question, I'll have somewhere to point them...

Redesigning System.Object/java.lang.Object

skeet — Fri, 05 Dec 2008 17:52:44 GMT

I've had quite a few discussions with a colleague about some failures of Java and .NET. The issue we keep coming back to is the root of the inheritance tree. There's no doubt in my mind that having a tree with a single top-level class is a good thing, but it's grown a bit too big for its boots.

Pretty much everything in this post applies to both .NET and Java, sometimes with a few small changes. Where it might be unclear, I'll point out the changes explicitly - otherwise I'll just use the .NET terminology.

What's in System.Object?

Before we work out what we might be able to change, let's look at what we've got. I'm only talking about instance methods. At the moment:

Life-cycle and type identity

There are three members which I believe really need to be left alone.

We need a parameterless constructor because (at least with the current system of chaining constructors to each other) we have to have some constructor, and I can't imagine what parameter we might want to give it. I certainly find it hard to believe there's a particular piece of state which really deserves to be a part of every object but which we're currently missing.

I really don't care that much about finalizers. Should the finalizer be part of Object itself, or should it just get handled automatically by the CLR if and only if it's defined somewhere in the inheritance chain? Frankly, who cares. No doubt it makes a big difference to the implementation somewhere, but that's not my problem. All I care about when it comes to finalizers is that when I have to write them it's as easy as possible to do it properly, and that I don't have to write them very often in the first place. (With SafeHandle, it should be a pretty rare occurrence in .NET, even when you're dealing directly with unmanaged resources.)

GetType() or (getClass() in Java) is pretty important. I can't see any particular alternative to having this within Object, unless you make it a static method somewhere else with an Object parameter. In fact, that would have the advantage of freeing up the name for use within your own classes. The functionality is sufficiently important (and really does apply to every object) that I think it's worth keeping.

Comparison methods

Okay, time to get controversial. I don't think every object should have to be able to compare itself with another object. Of course, most types don't really support this anyway - we just end up with reference equality by default.

The trouble with comparisons is that everyone's got a different idea of what makes something equal. There are some types where it really is obvious - there's only one natural comparison. Integers spring to mind. There are other types which have multiple natural equality comparisons - floating point numbers (exact, within an absolute epsilon, and within a relative epsilon) and strings (ordinal, culture sensitive and/or case sensitive) are examples of this. Then there are composite types where you may or may not care about certain aspects - when comparing URLs, do I care about case? Do I care about fragments? For http, if the port number is explicitly specified as 80, is that different to a URL which is still http but leaves the port number implicit?

.NET represents these reasonably well already, with the IEquatable<T> interface saying "I know how to compare myself with an instance of type T, and how to produce a hashcode for myself" and IEqualityComparer<T> interface saying "I know how to compare two instances of T, and how to produce a hashcode for one instance of T." Now suppose we didn't have the (nongeneric!) Equals() method and GetHashCode() in System.Object. Any type which had a natural equality comparison would still let you compare it for equality by implementing IEquatable<T>.Equals - but anything else would either force you to use reference equality or an implementation of IEqualityComparer<T>.

Some of the principle consumers of equality comparisons are collections - particularly dictionaries (which is why it's so important that the interfaces should include hashcode generation). With the current way that .NET generics work, it would be tricky to have a constraint on a constructor such that if you only specified the types, it would only work if the key type implemented IEquatable<T>, but it's easy enough to do with static methods (on a non-generic type). Alternatively you could specify any type and an appropriate IEqualityComparer<T> to use for the keys. We'd need an IdentityComparer<T> to work just with references (and provide the equivalent functionaliy to Object.GetHashCode) but that's not hard - and it would be absolutely obvious what the comparison was when you built the dictionary.

Monitors and threading

This is possibly my biggest gripe. The fact that every object has a monitor associated with it was a mistake in Java, and was unfortunately copied in .NET. This promotes the bad practice of locking on "this" and on types - both of which are typically publicly accessible references. I believe that unless a reference is exposed explicitly for the purpose of locking (like ICollection.SyncRoot) then you should avoid locking on any reference which other code knows about. I typically have a private read-only variable for locking purposes. If you're following these guidelines, it makes no sense to be able to lock on absolutely any reference - it would be better to make the Monitor class instantiable, and make Wait/Pulse/PulseAll instance members. (In Java this would mean creating a new class and moving Object.wait/notify/notifyAll members to that class.)

This would lead to cleaner, more readable code in my view. I'd also do away with the "lock" statement in C#, making Monitor.Enter return a token implementing IDisposable - so "using" statements would replace locks, freeing up a keyword and giving the flexibility of having multiple overloads of Monitor.Enter. Arguably if one were redesigning all of this anyway, it would be worth looking at whether or not monitors should really be reentrant. Any time you use lock reentrancy, you're probably not thinking hard enough about the design. Now there's a nice overgeneralisation with which to end this section...

String representations

This is an interesting one. I'm genuinely on the fence here. I find ToString() (and the fact that it's called implicitly in many circumstances) hugely useful, but it feels like it's attempting to satisfy three different goals:

Giving a developer-readable representation when logging and debugging
Giving a user-readable representation as part of a formatted message in a UI
Giving a machine-readable format (although this is relatively rare for anything other than numeric types)

It's interesting to note that Java and .NET differ as to which of these to use for numbers - Java plumps for "machine-readable" and .NET goes for "human-readable in the current thread's culture". Of course it's clearer to explicitly specify the culture on both platforms.

The trouble is that very often, it's not immediately clear which of these has been implemented. This leads to guidelines such as "don't use ToString() other than for logging" on the grounds that at least if it's implemented inappropriately, it'll only be a log file which ends up with difficult-to-understand data.

Should this usage be explicitly stated - perhaps even codified in the name: "ToDebugString" or something similar? I will leave this for smarter minds to think about, but I think there's enough value in the method to make it worth keeping.

MemberwiseClone

Again, I'm not sure on this one. It would perhaps be better as a static (generic!) method somewhere in a class whose name indicated "this is for sneaky runtime stuff". After all, it constructs a new object without calling a constructor, and other funkiness. I'm less bothered by this than the other items though.

Conclusion

To summarise, in an ideal world:

Equals and GetHashCode would disappear from Object. Types would have to explicitly say that they could be compared
Wait/Pulse/PulseAll would become instance methods in Monitor, which would be instantiated every time you want a lock.
ToString might be renamed to give clearer usage guidance.
MemberwiseClone might be moved to a different class.

Obviously it's far too late for either Java or .NET to make these changes, but it's always interesting to dream. Any more changes you'd like to see? Or violent disagreements with any of the above?

C# 4.0: dynamic ?

skeet — Thu, 30 Oct 2008 22:04:10 GMT

I've not played with the VS2010 CTP much yet, and I've only looked briefly at the documentation and blogs about the new C# 4.0 dynamic type, but a thought occurred to me: why not have the option of making it generic as a way of saying "I will dynamically support this set of operations"?

As an example of what I mean, suppose you have an interface IMessageRouter like this:

public interface IMessageRouter
{
void Send(string message, string destination);
}

(This is an arbitrary example, by the way. The idea isn't specifically more suitable for message routing than anything else.)

I may have various implementations, written in various languages (or COM) which support the Send method with those parameters. Some of those implementations actually implement IMessageRouter but some don't. I'd like to be able to do the following:

dynamic<IMessageRouter> router = GetRouter();

// This is fine (but still invoked dynamically)
router.Send("message", "skeet@pobox.com");
// Compilation error: no such overload
router.Send("message", "skeet@pobox.com", 20);

Intellisense would work, and we'd still have some of the benefits of static typing but without the implementations having to know about your interface. Of course, it would be quite easy to create an implementation of the interface which did exactly this - but now imagine that instead of IMessageRouter we had MessageRouter - a concrete class. In this case the compiler would still restrict the caller to the public API of the class, but it wouldn't have to be the real class. No checking would be performed by the compiler that your dynamic type actually supported the operations - given that we're talking about dynamic invocation, that would be impossible to do. It would instead be an "opt-in" restriction the client places on themselves. It could also potentially help with performance - if the binding involved realised that the actual type of the dynamic object natively implemented the interface or was/derived from the class, then no real dynamic calls need be made; just route all directly.

This may all sound a bit fuzzy - I'm extremely sleepy, to be honest - but I think it's a potentially interesting idea. Thoughts?

Update

Apparently this post wasn't as clear as it might be. I'm quite happy to keep the currently proposed dynamic type idea as well - I'd like this as an additional way of using dynamic objects.

Jon Skeet: Coding Blog : C#, Wacky Ideas

Eduasync part 13: first look at coroutines with async

Introduction to coroutines

Round-robin independent coroutines

Client code

The Coordinator class

Initialization and a choice of data structures

The main execution loop and async infrastructure

Conclusion

Of memory and strings

How are we going to measure object size?

Some simple starter measurements

Custom classes containing primitives

So what does an ASCII string look like?

Option 1: A simple class

Option 2: A simple struct

Option 3: Extension methods on byte[]

Conclusion

The curious case of the publicity-seeking interface and the shy abstract class

The rules around interfaces

Abstract classes to the rescue!

Conclusion

Non-iterable collection initializers

Option 1: Class level attribute

Option 2: Method level attributes

Option 3: Compiler and language simplicity

And another thing...

Iterate, damn you!

How can we fix it?

Change the type of the values variable

Change the type of the iterator variable

Pass the iterator variable by reference

Dynamic typing to the anti-rescue

Conclusion

First encounters with Reactive Extensions

Subscription model

Blocking aggregation

Using the non-blocking aggregation operators

Conclusion

"Magic" null argument testing

Contract classes and nested types within interfaces

Iterating atomically

Fix the interface

Conclusion

An object lesson in blogging and accuracy; was: Efficient "vote counting" with LINQ to Objects - and the value of nothing

Generic constraints for enums and delegates

Introducing UnconstrainedMelody

A couple of hours later...

The "dream book" for C# and .NET

Scope

Obstacles and format

Dream Team

Dream or reality?

Faking COM to fool the C# compiler

A different approach to inappropriate defaults

Making life easier (in .NET)

Breaking Liskov

Breaking BinaryReader's contract

And I would have gotten away with it if it weren't for that pesky inheritance...

Blow it, let's do it anyway...

Quick rant: why isn't there an Exception(string, params object[]) constructor?

Horrible grotty hack: returning an anonymous type instance

Value types and parameterless constructors

Yes, you can write a parameterless constructor for a value type in .NET

When does the parameterless constructor get called?

Operations which do call the constructor

Operations which don't call the constructor

Conclusions

Redesigning System.Object/java.lang.Object

What's in System.Object?

Life-cycle and type identity

Comparison methods

Monitors and threading

String representations

MemberwiseClone

Conclusion

C# 4.0: dynamic ?

Update

Change the type of the `values` variable

Change the type of the `iterator` variable

Pass the `iterator` variable by reference