Jon Skeet: Coding Blog : Google

MVP Again

skeet — Fri, 30 Oct 2009 21:14:09 GMT

I'm delighted to be able to announce that I'm now an MVP again.

Google has reconsidered the situation and worked out a compromise: I now receive no significant gifts from Microsoft, and I'm not under NDA with them. While that precludes me from a lot of MVP activities, it removes any concerns to do with Google's Code of Conduct. Basically my MVP status is truly just a token of Microsoft's recognition of what I've done in the C# community - and that's fine by me.

When I announced that I'd been advised not to seek renewal, I was amazed at the scale of the reaction in the comments, other blog posts, Twitter and personal email. I was touched by the response of the community. I really love working at Google, and the fact that we could figure out a solution to this situation is definitely one of the things that makes Google such an awesome place to work. Oh, and did I mention that we're hiring? :)

Anyway, the basic message of this post is: thanks to the community for caring, thanks to Google for reconsidering, and thanks to Microsoft for renewing my award. And they all lived happily ever after...

OS Jam at Google London: C# 4 and the DLR

skeet — Fri, 19 Jun 2009 16:50:00 GMT

Last night I presented for the first time at the Google Open Source Jam at our offices in London. The room was packed, but only a very few attendees were C# developers. I know that C# isn't the most popular language on the Open Source scene, but I was still surprised there weren't more people using C# for their jobs and hacking on Ruby/Python/etc at night.

All the talks at OSJam are just 5 minutes long, with 2 minutes for questions. I'm really not used to this format, and felt extremely rushed... however, it was still a lot of fun. I used a somewhat different approach to my slides than the normal "bullet points in PowerPoint" - and as it was only short, I thought I might as well effectively repeat the presentation here in digital form. (Apologies if the images are an inconvenient size for you. I tried a few different ones, and this seemed about right. Comments welcome, as I may do a similar thing in the future.)

	Introductory slide. Colleagues forced me to include the askjonskeet.com link.
	.NET isn't Open Source. You can debug through a lot of the source code for the framework if you agree to a "reference licence", but it's not quite the same thing.
	.NET isn't Open Source, but the DLR is. And IronRuby. And IronPython. Yay! And of course Mono is Open Source: the DLR and Mono play nicely together, and the Mono team is hoping to implement the new C# 4.0 features for the 2.8 release in roughly the same timeframe as Microsoft.
	This is what .NET 4.0 will look like. The DLR will be included in it, despite being open source. IronRuby and IronPython aren't included, but depend heavily on the DLR. (Currently available versions allow you to use a "standalone" DLR or the one in .NET 4.0b1.) C# doesn't really depend on the DLR except for its handling of `dynamic`. C# is a statically typed language, but C# 4.0 has a new static type called `dynamic` which you can do just about anything with. (This got a laugh, despite being a simple and mostly accurate summary of the dynamic typing support in C# 4.0.)
	The fundamental point of the DLR is to handle call sites - decide what to do dynamically with little bits of code. Oh, and do it quickly. That's what the caches are for. They're really clever - particularly the L0 cache which compiles rules (about the context in which a particular decision is valid) into IL via dynamic methods. Awesome stuff. I'm sure the DLR does many other snazzy things, but this feels like it's the core part of it.
	At execution time, the relevant binder is used to work out what a call site should actually do. Unless, that is, the call has a target which implements the shadowy `IDynamicMetaObjectProvider` interface (winner of "biggest mouthful of a type name" prize, 2009) - in which case, the object is asked to handle the call. Who knows what it will do?
	Beautifully syntax-highlighted C# 4.0 source code showing the `dynamic` type in action. The method calls on lines 2 and 3 are both dynamic, even though in the latter case it's just using a static method. Which overload will it pick? It all depends on the type of the actual value at execution time. If I'd had more time, I'd have demonstrated how the C# compiler preserves the static type information it knows at compile time for the execution time binder to use. This is very cool, but would take far too long to demonstrate in this talk - especially to a bunch of non-C# developers.
	There were a couple of questions, but I can't remember them offhand. Someone asked me afterwards about how all this worked on non-.NET implementations (i.e. Mono, basically). I gather the DLR itself works, but I don't know whether C# code compiled in the MS compiler will work at the moment - it embeds references to binder types in Microsoft.CSharp.dll, and I don't know what the story is about that being supported on Mono.

This is definitely the format I want to use for future presentations. It's fun to write, fun to present, and I'm sure the "non-professionalism" of it makes it a lot more interesting to watch. Although it's slower to create text-like slides (such as the first and the last one) this way, the fact that I don't need to find clip-art or draw boxes with painful user interfaces is a definite win - especially as I'm going to try to be much more image-biased from now on. (I don't want people reading slides while I'm talking - they should be listening, otherwise it's just pointless.)

DotNetRocks interview

skeet — Tue, 07 Oct 2008 16:33:49 GMT

Last Monday evening I had a chat with the guys from DotNetRocks, and today the show has gone live.

I wouldn't claim to have said anything particularly earth-shattering, and regular readers will probably be familiar with many of the themes anyway, but I thoroughly enjoyed it and hope you will too. Amongst other things, we talked about:

Protocol buffers
Implicit typing and anonymous types
Why it doesn't bother me that Office hasn't been ported to .NET
C# 4
My wishlist for C#
Threading and Parallel Extensions
Working for Google
How to learn LINQ
C# in Depth

Feedback welcome. And yes, I know I sound somewhat like a stereotypical upper-class idiot at times. Unfortunately there's not a lot I can do about that. Only the "idiot" part is accurate :)

Lessons learned from Protocol Buffers, part 4: static interfaces

skeet — Fri, 29 Aug 2008 18:50:32 GMT

Warning: During this entire post, I will use the word static to mean "relating to a type instead of an instance". This isn't a strictly accurate use but I believe it's what most developers actually think of when they hear the word.

A few members of the interfaces in Protocol Buffers have no logical reason to act on instances of their types. The message interface has members to return the message's type descriptor (the PB equivalent of System.Type), the default instance for the message type, and a builder for the message type. The builder interface copies the first two of these, and also has a method to create a builder for a particular field. None of these touch any instance data.

In most cases this doesn't actually cause any difficulties - we usually have an instance available when we're in the PB library code, and the generated types have static properties for the default instance and the type descriptor anyway. Even so, it feels messy to have interface members which rely only on the type of the implementation and not on any of the actual data of the instance.

I've wondered before now about the possibility of having static members in interfaces - usually when thinking about plug-in architectures - but there's always been the problem of working out how to specify the type on which to call the members. Variables and other expressions usually refer to values rather than types, and System.Type doesn't help as it provides no compile-time knowledge of the type being referred to.

There's one big exception to this, however: generic type parameters. I don't know why it had never occurred to me before, but this is a great fit for the ability to safely call static methods on types which are unknown at compile-time. Furthermore, it could provide a great way of enforcing the presence of constructors with appropriate signatures, and even operators. Before I get too far ahead of myself, let's tie the simple case to a concrete example.

Creating builders from nothing

In my previous post I gave an example of a method which ideally wanted to return a new message given a CodedInputStream and an ExtensionRegistry (both types within Protocol Buffers, the details of which are unimportant to this example). The current code looks like this:

private static TMessage BuildImpl<TMessage2, TBuilder> (Func<TBuilder> builderBuilder,
                                                        CodedInputStream input,
                                                        ExtensionRegistry registry)
    where TBuilder : IBuilder<TMessage2, TBuilder>
    where TMessage2 : TMessage, IMessage<TMessage2, TBuilder>
{
    TBuilder builder = builderBuilder();
    input.ReadMessage(builder, registry);
    return builder.Build();
}

For the purposes of this discussion I'll simplify it a little, making it a generic method in a non-generic type.

private static TMessage BuildImpl<TMessage, TBuilder> (Func<TBuilder> builderBuilder,
                                                       CodedInputStream input,
                                                       ExtensionRegistry registry)
    where TBuilder : IBuilder<TMessage, TBuilder>
    where TMessage : IMessage<TMessage, TBuilder>
{
    TBuilder builder = builderBuilder();
    input.ReadMessage(builder, registry);
    return builder.Build();
}

The first parameter is a function which will return us a builder. We can't simply add a new() constraint to TBuilder as not all geenrated builders will have a public constructor. However, we do know that the TMessage type has a CreateBuilder() method because it implements IMessage<TMessage, TBuilder>. Unfortunately we don't have an instance of TMessage to call CreateBuilder() on! Really, we'd like to be able to change the code to this:

private static TMessage BuildImpl<TMessage, TBuilder> (CodedInputStream input, ExtensionRegistry registry)
    where TBuilder : IBuilder<TMessage, TBuilder>
    where TMessage : IMessage<TMessage, TBuilder>
{
    TBuilder builder = TMessage.CreateBuilder();
    input.ReadMessage(builder, registry);
    return builder.Build();
}

That's currently impossible, but only because we can't specify static methods in interfaces. Suppose we could write:

public interface IMessage<TMessage, TBuilder>
    where TMessage : IMessage<TMessage, TBuilder>
    where TBuilder : IBuilder<TMessage, TBuilder>
{
    static TBuilder CreateBuilder();

    // Other methods as before
}

Wouldn't that be useful? The value would almost entirely be for generic types or methods where the type parameter is constrained to specify the relevant interface, but that could arguably still be very handy.

Operators and constructors

At this point hopefully the idea I mentioned earlier of being able to specify operators and constructors is quite obvious. For instance, we could make all the existing numeric types implement IArithmetic<T> (where T was the same type, e.g. int : IArithmetic<int>):

public interface IArithmetic<T>
{
    static T operator +(T left, T right);
    static T operator -(T left, T right);
    static T operator /(T left, T right);
    static T operator *(T left, T right);
}

// Used in LINQ to Objects, for example:
public static T Sum<T>(this IEnumerable<T> source) where T : IArithmetic<T>
{
    T total = default(T);
    foreach (T element in source)
    {
        total += element; // Interface says we can do total + element
    }
    return total;
}

Plug-ins for a particular program could implement IPlugin:

public interface IPlugin
{
    static new (PluginHost host);

    // Normal plug-in members here
    string Title { get; }
}

// Used within a PluginHost like this...
public T CreatePlugin<T>() where T : IPlugin
{
    T plugin = new T(this);
    log.Info("Loaded plugin {0}", plugin.Title);
    return plugin;
}

In fact, I'd imagine it would make sense to define a whole family of IConstructable interfaces, along the same lines as the Func and Action delegate families:

public interface IConstructable
{
    static new();
}

public interface IConstructable<T>
{
    static new(T arg)
}

public interface IConstructable<T1, T2>
{
    static new(T1 arg1, T2 arg2);
}

// etc

Inheritance raises its ugly head

There's a fly in the ointment here. Normally, if a base type implements an interface, that means a type derived from it will effectively implement the interface too. That ceases to hold in all cases. You can get away with it for straightforward static methods/properties, in the same way that people often write code such as UTF8Encoding.UTF8 when they really just mean Encoding.UTF8. However, it doesn't work for constructors - they aren't inherited, so you can't guarantee that Banana has a parameterless constructor just because Fruit does.

This is not only a problem for one concrete class deriving from another, but also for abstract implementations of interfaces. This happens a lot in Protocol Buffers; often the interface is partially implemented by the abstract class, with the final "leaf" classes in the inheritance tree implementing outstanding members and occasionally overriding earlier implementations for the sake of efficiency. Should we be able to specify an abstract static method in the abstract class, making sure that there's an appropriate implementation by the time we hit a concrete class? As it happens that would be useful elsewhere in the Protocol Buffer library, but I'll admit it's slightly messy. I suspect there are ways round all of these issues, even if they might sometimes involve restricting the feature to particular, common use cases. However, every niggle would add its own piece of complexity.

There may well be other issues which would prove challenging - and other interesting aspects such as what explicit interface implementation would mean, if anything, in the context of static members. Language experts may well be able to reel off great lists of problems - I'd be very interested to here the ensuing discussions.

Conclusion

I believe that static interface members could prove very useful in generic algorithms, particularly if operators and constructors were allowed as well as the existing member types available in interfaces. There are significant sticking points to be carefully considered, and I wouldn't like to prejudge the outcome of such deliberations in terms of whether the feature would be useful enough to merit the additional language complexity involved. It feels odd to implement an interface member which is effectively only of use when the implementing type is being used as the type argument for a generic type or method, but as developers learn to think more generically that may be less of a restriction than it currently seems.

This post is the last in my current batch talking about Protocol Buffers. There may well be more, but it's unlikely now that the Protocol Buffer port is almost entirely complete. Most of these posts have involved generics, and the current limitations of what C# allows us to express in them. I do not intend to give the impression that I'm dissatisfied with C# - I've just found it interesting to take a look at what lies beyond the current boundaries of the language. The one aspect of these posts which I would definitely like to see addresses is that of covariant return types - the Java implementation of Protocol Buffers is significantly simpler in many ways purely due to this one point.

Lessons learned from Protocol Buffers, part 3: generic type relationships

skeet — Fri, 29 Aug 2008 18:48:48 GMT

In part 2 of this series we saw how the message and builder interfaces were self-referential in order to allow the implementation types to be part of the API. That's one sort of relationship, but in this post we'll see how the two interfaces relate to each other. If you remember from part 1 every generated message type has a corresponding builder type. As it happens, this is implemented with a nested type, so if you had a Person message, the generated types would be Person and Person.Builder (in a specified namespace, of course).

Without any interfaces involved, this would be very simple. The types would just look like this (with more members, of course):

public class Person
{
    public static Builder CreateBuilder() { ... }

    public Builder CreateBuilderForType() { ... }

    public class Builder
    {
        public Builder() { ... }

        public Person Build() { ... }
    }
}

You may well be wondering why there are two methods for creating a builder. The static method is convenient for code which knows it's dealing with the Person message. The instance method ends up being part of the message interface, which makes it useful for code which can work with any message. In addition, the constructor for Person.Builder is accessible in the C# version. In the original Java code the only way of creating a builder is via the methods in the message class; I decided to remove this restriction for the sake of making the oh-so-readable object initializer syntax available in C# 3.

Redesigning the interfaces to refer to each other

In part 2 we created self-referential interfaces for the message and builder interfaces which looked like this:

public interface IMessage<TMessage> where TMessage : IMessage<TMessage>
{
...
}

public interface IBuilder<TBuilder> where TBuilder : IBuilder<TBuilder>
{
...
}

The constraints on the type parameters allow us to make the API very specific, and we can use the same trick again when we relate the builder and message types together. The step where we introduce a new type parameter to each of them is straightforward:

public interface IMessage<TMessage, TBuilder> where TMessage : IMessage<TMessage, TBuilder>
{
...
}

public interface IBuilder<TMessage, TBuilder> where TBuilder : IBuilder<TMessage, TBuilder>
{
...
}

Unfortunately without any restrictions on the "foreign" type parameter in each interface, we don't get enough information to make everything work. We need to tie the two types together more tightly, like this:

public interface IMessage<TMessage, TBuilder>
    where TMessage : IMessage<TMessage, TBuilder>
    where TBuilder : IBuilder<TMessage, TBuilder>
{
    ...
}

public interface IBuilder<TMessage, TBuilder>
    where TMessage : IMessage<TMessage, TBuilder>
    where TBuilder : IBuilder<TMessage, TBuilder>
{
    ...
}

To make this concrete for Person and Person.Builder we end up with implementations like this:

public class Person : IMessage<Person, Builder>
{
    public static Builder CreateBuilder() { ... }

    public Builder CreateBuilderForType() { ... }

    public class Builder : IBuilder<Person, Builder>
    {
        public Builder() { ... }

        public Person Build() { ... }
    }
}

This works, but it's really ugly. Any generic methods wanting to take a TMessage type parameter implementing IMessage<TMessage, TBuilder> have to also have a TBuilder type parameter, and the two constraints need to be expressed each time. It's a real pain. In fact, I've got an IMessage<TMessage> interface which contains almost nothing in it (and which the more generic interface extends). This allows me to get hold of the message type (and use it in the API), inferring the builder type by reflection. That's a pain too, frankly. It's a particular nuisance because when I do infer the builder type, I haven't actually got any compile-time constraint the lets any other code know that it's the right builder type for the message type. In one specific case it's led to this horrific method (in a type generic in TMessage:

Fortunately this is hidden from public view - and the only reason to do it at all is to enable a pleasant API of MessageStreamIterator<TMessage> : IEnumerable<TMessage> where TMessage : IMessage<TMessage>. The result of the evil method above is exactly what the caller is likely to want, otherwise I wouldn't put up with it. However, that sort of excuse has been coming up far too much in the PB implementation, so I've had a quick think about what could be done about it.

Contemplating a more expressive language

I should really prefix this section by saying that I'm not actually suggesting this as a way forward for C# or .NET. (I suspect it would take more work in the CLR as well as just in the language; I don't know enough about CLR generics to say for sure, but I'd be surprised if this were feasible.) I haven't encountered many situations where I've wanted anything like this, and the extra complexity in the language would be quite high, I suspect. Suppose an interface could contain extra type parameters, including constraints, in the body of the interface:

// Purely imaginary syntax!
public interface IMessage<TMessage> where TMessage : IMessage<TMessage>
{
    <TBuilder> where TBuilder : IBuilder<TBuilder>, TBuilder.TMessage : TMessage

    // Normal methods, which could use TBuilder
}

public interface IBuilder<TBuilder> where TBuilder : IBuilder<TBuilder>
{
    <TMessage> where TMessage : IMessage<TMessage>, TMessage.TBuilder : TBuilder

    // Normal methods, which could use TMessage
}

There are various ways in which the interface implementation could indicate the type of TBuilder. The syntax itself isn't particularly interesting - it's the extra information which is conveyed which is the important bit. I've dithered between this being a step forward and it not. At first glance it looks no better than having both type parameters in the interface declaration, but I believe it would genuinely make a difference. For instance, the above evil method could be written as:

private static TMessage BuildImpl(Func<TMessage.TBuilder> builderBuilder,
                                  CodedInputStream input,
                                  ExtensionRegistry registry)
{
    TMessage.TBuilder builder = builderBuilder();
    input.ReadMessage(builder, registry);
    return builder.Build();
}

This time there's no need for the method to be generic, because the type is already generic in the message type. Furthermore, we can call this method with no reflection. All other APIs which have previously had to be specify two type parameters can now just specify the one. Apart from anything else, this leaves more scope for type inference in generic methods - passing either a message or a builder to a generic method happens occasionally, but it's very rare to pass in both.

We've essentially expressed the relationship between the message type and the builder type a little more explicitly, so that we can guarantee it exists (and use it) at compile time. That's at the heart of the problem to start with - without a second type parameter in the initial interface declaration, in the current language there's no way of expressing a close relationship with another type.

Conclusion

I don't think it would be fair to say that C# really lets us down here - it happens not to support a pretty rare scenario, and that's fair enough. I'd be interested to know whether any other languages allow the same sort of concepts to be expressed more pleasantly. The ugly solution I've presented here does at least work, and it's nearly invisible to most users, who are likely to just reference the concrete generated types. I'm not happy with the verbosity which has become necessary in many places, but it's in a good cause. It's interesting to note that the Java API doesn't use this sort of doubly-generic relationship: again, covariant return types allow the concrete message and builder types to express their APIs directly and still implement a more general interface at the same time.

In the next part I'll look at another possibility which would make interfaces and generics a more powerful combination: static interface methods.

Google release protocol buffers as an open source project

skeet — Tue, 08 Jul 2008 18:30:13 GMT

Yesterday the Google open source blog announced a new open source project, to release one of the core pieces of Google infrastructure: protocol buffers. Basically protocol buffers are a way of encoding structured data in a language-neutral and versioning-friendly fashion. Yes, there are a lot of similar ways of doing similar things, but this happens to be the one we use.

Currently the protocol buffer compiler (which takes .proto files and converts them into source for various languages) supports C++, Java and Python. I'm hoping to add C# to that list reasonably quickly. To start with this will just be in my spare time rather than as a 20% project, but who knows...

It's really nice to work in an environment where the question of "is there any reason we can't open source this?" crops up quite frequently. I'm looking forward to the day when a piece of code I've developed at Google ends up as part of an open source project.