Leaning Into Windows

My debugging strategy is to develop a hypothesis or two, pick the easiest most valuable one to test, test it in the easiest manner, and go on. I’m working to disprove the hypothesis because this is far easier to do and helps avoid me getting stuck on a particular notion of what is wrong. A test is more valuable if it cuts out a greater portion of the possibilities for failure, which John Robbins calls the divide and conquer approach.

I bring this up to explain that I rarely use the reporting MEF provides to solve stable composition errors. Maybe one day I will, but its much faster for me to find the missing part and begin adding the “AllowDefault=true” parameter to the Import attribute. I’m not being chaotic or random, it just works for me.

This morning I remained a bit stumped by a stable composition error. I realized it was a stable composition error because it went away when I said “AllowDefault=true”. But, my own reporting and other tests showed that the implementation was clearly in the container. So, what was going on.

A passing hypothesis flitted through my head “what would happen if there were two?”

It was a quick test to put a break point right after composition competed and request the exports for the part. Voila there were two!

Two tips:

- Stable composition fails when cardinality is not correct. A big word say Import fails when there are zero or more than one match.

- If you have multiple discovery directories, it is very easy for common assemblies to wind up in more than one directory. This results in multiple matches, and why I had the cardinality failure.

VS 2010 is nearly out the door, so it’s time to start fantasizing about killer features in the next version of .NET and Visual Studio.

The feature I want to see is “embedded DSL.”

Like many killer features, success comes from doing in a great way something you can already do halfway. So, I’ll demonstrate this feature in relation to T4 and a MEF scenario where it’s useful today. But before I get there…

I want to see DSL embedded in a language – maybe VB since it oddly enough has become the platform for experimentation, and since we’ve already started embedding other stuff in the language (XML). DSL implies particular syntax issues, but ultimately its metadata for code generation. I want to do code generation that is under my control in a fragmented way within the scope of normal code. Here’s a potential syntactic example:

Public Sub New()
‘ Normal Code
End Sub
{PropertyPattern}
{Property Name=”DisplayName” Type=String}
{Property Name=”DataName” Type=String}
{End PropertyPattern}

‘ More normal code

In this simplistic DSL, the Property metadata is just a pretty syntax for filling data into an interface. Wait, wait, you can’t stuff data into an interface. Right. You need an implementation which you can discover in real time (MEF anyone?) and that implementation can provide defaults for all the other values! And an include pattern here should let you reuse metadata defined once in your application.

PropertyPattern refers to a generation template – like an extended version of T4 – that can output real code in response to the metadata that’s passed. It’s an extended version because it’s strongly typed to the metadata defined via the metadata/DSL interface. The pattern is also discovered (MEF anyone?), which, well just trust me on this, allows a full governance model (customizing and governance on templates is one of my specialties, but let’s not geek out on that right in the middle of a hot fantasy). In simple terms, the governance model means a fallback mechanism through project (assembly), to group, to organization, to defaults/in the box.

Before I go further, let me say this is not a replacement for application generation. Application generation and code generation aren’t the same thing. Application generation is either a closely linked set of templates where the interrelations are as important as the templates, or application generation is architecture generation which is a new and emerging field.

Back, to the fantasy, because it’s an important part of a bigger picture of changing how we write applications…

Is this fantasy just Kathleen on too much MEF?

Actually no MEF required at all this morning… let me show you how to do this today – no compiler changes, no new dependencies, you can do this right now in VS 2008 (if you download the DSL Toolkit so you have T4, which you already did anyway, right?).

Well, OK, just a little MEF to get our morning started… the scenario is a MEF scenario, although there is no MEF in the solution. When you work with MEF and you want to create a MEF friendly interface for later discovery, you create an interface that a part will later fulfill, a separate interface of composition metadata, and a custom export attribute, which allows you to define a part just by implementing the first interface and tossing on the attribute. Whew! This means there is an annoying tedious (but technically elegant and necessary) pattern that I’ve implemented perhaps a billion times. What’s worse, the pattern obscures information about the interface, which of course I should also include in XML Documentation so I’m not in the code in the first place checking out the info – but see, more internal redundancy. And oh, by the way, if you screw up the pattern, the bugs can be very hard to track. Let’s fix it…

[NOTE: The implementation jumped back and forth between VB and C#. In VB, I’d implement this as XML literals which makes a more concise syntax with way less code in the included template, but T4 barfed at the XML literals and I didn’t feel like bothering with it today. Without XML literals, C# has better syntax because of the collection initializers which don’t make it into VB until 2010. There is no relation between the syntax of the DSL and the output syntax. I happen to be outputting VB.]

I created a T4 template. If you’re in VS2010, this is a normal Text Template, not a preprocessed one. If you’re in 2008, create a text file and give it a .tt extension. Also copy in the MefInterfaceDsl.t4 file that I’ve attached into your project. (I tested in VS 2008, in VS 2010, you may need to change the extension to .tx) Yes, your DSL will be written in T4 and yes, your DSL T4 template will have a different extension than the supporting one. That’s because Visual Studio outputs code for a file with a .tt extension and does not for one with a .t4 extension. You want output only from your DSL T4.

Here’s the template, then more talk:

<#@ template debug="false" hostspecific="false" language="C#v3.5" #>
<#@ assembly name="System.Core" #>
<#@ output extension=".vb" #>
<#@ include file="MefInterfaceDsl.t4" #>
<#=
         new Interface()
         {
            Name = "ISearchModelBase",
            Scope = Scope.Public,
            CompositionInfo =
            {
               new Property() {Name="TargetType", PropertyType="Type"}
            },
            Members =
            {
               new Property() {Name="DisplayName", PropertyType="string"},
               new Property() {Name="DataName", PropertyType="string"}
            }
         }.Output()
#>

Wow, is that really a DSL? Yes, but I won’t claim it’s a very good one. It’s a hack to allow the concept to work in VS 2008 and VS 2010, in hopes that we can get an elegant syntax in VS 20Next.

The initial four lines are a necessary T4 distraction. I’ve stated that the template language is C# with 3.5 extensions (not necessary in VS 2010). I’ve included core because it makes 3.5 work. I’ve stated that output will be in .vb. The actual syntax for the output is in the include file. Remember this is a working sample, if you wish to use this, you’ll want to enhance the MefInterfaceDsl.t4, including rewriting it to output C# if that’s your current flavor preference.

The include file has a class named Interface, Property and a few others. Initializers and collection initializers build the graph for the ISearchModelBase interface which has two properties, and one composition metadata property. The Interface class has an Output method that returns a string with the code output. Visual Studio places this output in a dependent file (select Show All Files to see this in VB). I included it below so you don’t have to run a project just to see the output.

Since the artifact is generated, I don’t have to remember the pattern and I’ll never be bit by forgetting to set AllowMultiple=false. Since the DSL/metadata is in the project beside the artifact, I can find an work with it (this would not be appropriate for application generation DSL/metadata whose artifacts spanned many projects, solution, and platforms).

So why is a normal T4 template a DSL? Because a DSL is a way to define generation information (metadata) in a way that is friendly to the human, followed by artifact generation.

I want this extended to be a true embedded part of the language to avoid these limitations and supply these features (and probably a bunch more stuff I haven’t thought of):

• - Can be any part of any normal code file
  • Not limited to entire files (although the T4 output can be partials)
  • The DSL/metadata is a holistic part of the code
• - Artifact patterns (templates) are discoverable (can be anywhere by anyone)
• - DSL/metadata patterns are discoverable (default values and pattern extensions
• - Intellisense on the DSL
• - Better syntax (drop the new and other class residuals)
• - No reliance on file location (the T4 support file must be in relation to your project)
• - No ugly opening stuff about T4 unrelated to the task at hand
• - Standard extensible metadata/DSL patterns provided
• - Standard extensible artifact patterns provided

Here’s the output:

Option Strict On
Option Explicit On
Option Infer On

Imports System
Imports System.Collections.Generic
Imports System.Linq
Imports System.ComponentModel.Composition

public Interface ISearchModelBase
   Property DisplayName As string
   Property DataName As string
End Interface
public Interface ISearchModelBaseComposition
   Readonly Property TargetType As Type
End Interface

< MetadataAttribute() > _
< AttributeUsage(AttributeTargets.Class, AllowMultiple:=False) > _
public Class SearchModelBaseAttribute
   Inherits ExportAttribute
   Implements ISearchModelBaseComposition

   Public Sub New( ByVal targetType As Type)
      MyBase.New(GetType(ISearchModelBase))
      _targetType = targetType
   End Sub

   Private _targetType As Type
   Public Readonly Property TargetType As Type Implements ISearchModelBaseComposition.TargetType
      Get
         Return _targetType
      End Get
   End Property

End Class

I got bit with an issue that I’d love to spare you.

For a short time MEF Preview 8 was posted with a fairly serious bug for Silverlight programmers using PartInitializer.

The way PartInitializer should work, and works now, is that all exports within the current XAP should be discovered by default.

When MEF Preview 8 was first posted, it included a bug which resulted in the XAP not being searched for exports – no exports except those explicitly placed into the container were discovered. The incorrect version was up for a handful of days. Of course luck would have it that I downloaded during that time, and a spasm of insecurity led me to believe I’d forgotten how to use MEF.

Here’s Glenn Block’s discussion.

I demo’d code contracts in my session at DevConnections. When I did a Camtasia video of the talk I found that they were not working. This led to a frustrating few hours, as I really wanted to complete the recording. Let me explain what I think happened in hopes of saving you some time. Note that this may be entirely different at VS 2010 RTM.

First, the reason they were not working was because runtime checking was turned off. The normal way to turn runtime checking on is through a tab called Code Contracts which appears on the Project Properties dialog. But alas, I had no such tab.

There is a download for code contracts which looks like this on the code contracts page.

Perhaps you would have thought this was intended for VS 2010 Beta 2, but I did not. I also thought I had already installed this because I’ve previously played with contracts in VS 2008, but apparently this didn’t get reinstalled after a machine rebuild.

At Julie Lerman’s suggestion, I reinstalled the Code Contract development tools and the tab appeared. This allowed me to turn runtime checking back on and voila see the failure.

Code contracts are very cool and I hope they will be implemented a bit more cleanly in the final release of VS 2010.

I got the following error with Beta 2.

I do not know what is relevant, but the project back-targets .NET 3.5 and is a project that includes T4 preprocessed templates. Neither of these should matter in relation to this error:

Error    4    Unable to write to output file ‘[[full project diretory]]/\obj\x86\Debug\T4CompiledTemplates.pdb': Unspecified error     T4CompiledTemplates

(with [full project directory] replaced with the actual directory.

I tried the obvious – unload the project, unload VS, restart the system. None worked.

The solution for me was to create a brand new project (with new name and directory) copy the files, reset the references, etc.

I rarely lobby publically for feature changes. But as we get closer to Beta 2, the feature lockdown becomes tighter. MEF is in the core of the framework, so any changes need to be carefully considered. I know that. But this change is small, testable and very important. 

And Glenn has told me that its unlikely to happen, partly because I seem to be the only one asking for it. I think that’s just because you don’t know about it yet.

You need to help me make this happen.

The attributed model of MEF is the one you will almost certainly use in the VS 2010 timeframe. In this model, you adorn classes that you want in play in your container with the Export attribute. This is MEF specific attribute and a separation concern guideline is that you want to expose as little as possible about your technology, except where you need it. So, MEF Preview 6 came out with a very slick solution. The ExportAttribute class is unsealed. You can inherit from it and create your own attributes, named whatever you want. If you have a class which exports as an IProcess you can have a ProcessAttribute adorn that class. This makes your class more readable and moves the type your exporting (IProcess) into the base class where no one needs to remember it. This can be important when you have more complex derivation stacks.

This is especially valuable with custom metadata attributes. Let’s say my process needs a name, perhaps so it can be displayed in a menu. MEF metadata allows you to add this information, managed at the manifest level so your class isn’t fully loaded until needed. Custom metadata is very valuable and by adding a MetadataAttributeAttribute (yes, that’s the name) to the class derived from ExportAttribute, you give your ProcessAttribute a nice name, and the data required for MEF to do its job.

So far everything is very nice. Nick describes this briefly in his post here.

But it’s about to get very ugly.

Metadata is inserted into a dictionary for use by MEF. Your name is a string, so you expect to be access it as a string. You commonly MEF metadata via an interface, and a best practice prior to Preview 6 was to supply the same interface for your custom attribute (ProcessAttribute) and the call to retrieve the metadata.

This fails when you derive from ExportAttribute.

Let’s first look at why, then the simple fix that I think is so important to add, even if its late in the cycle.

ExportAttribute is a class that has an available attribute of AttributeUsage. AttributeUsage has a parameter called AllowMultiple. ExportAttribute quite reasonably has AllowMultiple set to true so you can export several interfaces at once.

MetadataAttributeAttribute is marking a class as supplying metadata information to MEF for us in a dictionary. This dictionary allows you to find out more information about your part before loading it – for display, filtering, custom prioritization, and other reasons. MetadataAttributeAttribute is applied to a class deriving ultimately from Attribute. It checks the AttributeUsage to see if the attribute allows multiples. If it allows multiples, MEF adds the values to the dictionary as an array of the declared type instead of the declared type.

OK, that’s a bit convoluted. Let me explain why this is really nice behavior. Let’s say you want to add metadata to category to any of your MEF parts. You create a CategoryAttribute class and give this class both the MetadataAttributeAttribute attribute and the AttributeUsage(AllowMultiple=true) attribute. Now you can supply Category as an attribute to any of your MEF parts and you can supply as many categories as you want to each part. This is very nice behavior.

However, we take these two very nice behaviors together we get a rather bizarre result. If you simply inherit from ExportAttribute the way Nick describes, you’ll place your metadata into the metadata dictionary as an array, and you will only be able to successfully retrieve the metadata with an interface that uses an array. The attribute you declare and the way you retrieve the metadata will not be able to use the same interface. This will be totally unexpected, you will get a rather cryptic error, and I hope you find this blog post so you spend somewhat less time than the 20 hours I did. You will also be able to place the attribute on multiple times – this is almost never what you want because you placed the interface you were exporting into the attribute – in other words, the ProcessAttribute is identifying things that export IProcess. You don’t want to repeat this.

Thanks for sticking with me through that. This is arcane knowledge that puts a twisty mountain pass between you and the pit of success. Instead I want it to “just work” the way you expect.

Now here’s the itty bitty fix that to solve this problem….

The MEF core needs to add a new attribute class – let’s say “ExportCustomBaseAttribute” for this discussion. This attribute inherits from ExportAttribute and does only one thing. It resets the AllowMultiple parameter to False. All you need to know is that you generally inherit from ExportCustomBaseAttribute instead of ExportAttribute. You will find this attribute adjacent to ExportAttribute in Intellisense. It gives a very nice place in the documentation to explain this problem for the folks that wish to understand it. Alternatively, this attribute could be called “ExportSingleAttribute” or a variety of other names. It might be possible to make it abstract if avoiding direct usage is desirable.

The benefit is that you do not need to understand the entire scenario described in this blog post to be able to create readable custom attributes that tuck the MEF dependency away behind the scenes.

But this isn’t going to happen unless we tell the team we want it. Comments on this site have been a bit ditzed. If you have a problem with posting a comment here (or you just want to bug Glenn instead), please place a comment here in Glenn’s Preview 6 post or here at Nick’s

I just posted this on my AppVenture blog on preprocessed T4 templates

I love it when Glenn and the MEF team make me wrong! Because that means we get something new that I didn’t think they could pull off

First it was Stable Composition.

Now it’s Open Generic Composition.

A closed generic is something like this air code: Public Class x These are the only types of generics I anticipated MEF would support, at least in .NET 4.0. I’ve said this in a number of talks, and I’m happy to say that information is not wrong..

An open generic allows you to expose Foo(Of T) and allow the specific class to be specified by the import.

Note that this is not intrinsic to MEF itself but an alternate catalog. So, it’s not only cool in and of itself that we can now do Open Generics, but it’s a hint at how powerful MEF really is as we move beyond what’s delivered in the box. I’ll let you read Glenn’s post for the details.

What blows me away is that MEF is just starting. I think that’s what amazes me most. It’s like a seed just sprouting, or a kernel we’re just beginning to imagine how to harness. I’m excited about MEF in NET 4.0, but I also think more cool things like this approach to Open Generics will unfold as more people explore MEF.

I’ll follow up with some cool uses I’ve got in mind after I work through the details another day.

You can check here for a quick description of MEF

I'm giving a half dozen MEF talks this summer and I'm frequently asked "what happens if a part isn't available". The old answer was "the system crashes, how could it do anything else?" This conversation definitely deflates the upbeat mood of a MEF talk. Recently, MEF has changed, making that answer obsolete.

MEF is a composition container which satisfies imports by tracking down associated exports. The correct number of exports to satisfy each import is called the cardinality and can be one, zero to one or zero to many. Thus an import can fail because there are too few or too many matching exports.

In the MEF previews 1-5 (inclusive) and in the Visual Studio CTPs and beta1, MEF throws an exception when a failure occurs.

MEF Preview 6, released last Monday, introduces "stable composition." With stable composition, the container can know about, but not expose parts. If a part fails on a cardinality rule, the MEF container remembers the part, but keeps it hidden. If additional composition occurs (such as through the Refresh method) additional attempts to fully compose the part occur. If its cardinality is fulfilled, the part becomes available for additional composition. You can think of this as "if a missing sub part shows up later, the containing part will become available."

There are both good and bad aspects of this, and it definitely affects how you think of and write your MEF systems. In general, it will make MEF apps more stable. If you design a plug-in model, and the creator of a plug-in fails to properly deploy (or a confused user deletes some but not all of a plug-in via File Explorer), your system will not crash. The containing part, which would probably fail if run, doesn't appear in composition. This makes your system more robust against errors that are beyond your control. It also allows the late composition strategy, although I'm not yet clear on good scenarios for it.

The down side is that you may have more challenges finding certain types of composition errors because you will not receive an exception - you need to catch the current state of the composition container. And if you don't consider this behavior when writing your app, you can get officially bad behavior.

In a plug-in design such as the directory composition model, any part in your system can fail on a cardinality (and could previously have crashed) because all parts can be made of other parts with dependencies you don't know about.

For example, consider creating a main menu as a part and that menu is made up of menu items and sub menus and one of the menu items cannot be composed due to a deeply nested cardinality failure. If each main menu item is a part and you import the menu items as a collection using the ImportMany attribute, you're fine. Your application will simply not display the failed menu item.As deep as the nesting goes, each layer that is a collection is naturally protected because cardinality failures just remove one part from the collection.

If instead you create your main menu to explicitly expect a particular menu item (such as a Tools sub menu) and that menu has a required Import for a part and so on down to a point of cardinality failure, then the failure cascades up the chain. This happens because the leaf cardinality failure means that part is not available, causing a cardinality failure at the next level. In this hypothetical case, the failure cascades all the way up to the main menu which does not now exist. The application either runs without a main menu, or the application fails because the main menu is missing.

You can avoid this by considering the intent of each import and providing appropriate protection. Some parts are optional and your application can run just fine without them. These should either be in a collection (ImportMany in recent previews) or not required. Other parts are important, but not important enough to cause their container to fail. These can be managed via asserts, and communication with support or the user. If parts are critical to the application running, then you need to check that they exist after composition and shut down the application as gracefully as possible.

I think this is a good change, but at least until more patterns emerge, you need to consider what would happen if any import is not successfully satisfied. Should the containing part also fail to be composed?

MEF is a very sharp knife. It cuts up the tomatoes and carrots really well and we can have a dandy stew. Or we can wind up in the emergency room. Understanding MEF is important to safe MEF use.

You can find more on the Preview 6 changes in this post by Nicholas Blumhardt who is a member of the MEF team. I'd like to thank Glenn Block for his discussions with me on this change.

A couple of times recently I've wanted a super short summary of what MEF is and why you care. I thought I'd share what I came up with here. MEF stands for Managed Extensibility Framework.

MEF allows you to compose parts at runtime to alter application behavior based on the current environment. The most common way to alter behavior is to include dll's in a directory. The most common way to code MEF is through attributes. Both of these decisions are flexible because MEF itself is extensible and can work in many different ways. The Visual Studio 2010 IDE uses a good deal of MEF. You can use MEF for plug-ins for your application, improve testability, and to create a system of small replaceable parts. You may find other creative ways to use MEF.

I don't know whether MEF is addicting, but composability definitely is !

A few friends have pointed out that my post discussing my disappointment with Workflow 4.0 was somewhere between difficult and incomprehensible if you have no idea what's actually changed. Maurice posted this which gives some additional background.

Well, to me it is vastly disappointing. Back in January, I said that I thought Workflow 4.0 would be the most important feature of .NET 4.0. I was completely wrong. WF 4.0 will be a rather uneventful part of the .NET 4.0 release.

My initial projection was partly because .NET 4.0 will (thankfully) be a rather quiet release at core with only limited new language features so the competition was never all that steep. But it was also because the WF in the CTP represented an important change to Workflow. Microsoft screwed up WF 3.0 and 3.5. It was excessively complex for the marketplace. They implicitly acknowledged this when deciding to scrap the entire thing and start over with WF 4.0. While some people do have deployed applications, WF 3.0 and 3.5 could not be the ubiquitous WF I think we all need in our toolbox. I was hoping WF 4.0 would be that tool.

The bad news is that WF 4.0 probably won’t be. WF 4.0 suffers from two problems, one implied by the other. The first problem is that it has no state machine. At least it’s not in the beta and I have heard no contradictory announcement. The second is that if they could not even provide basic functional parity with WF 3.0/3.5 how could they possibly have had their development act together/sufficient resources/fill in the blank to produce a great 1.0 product. Remember, with the rewrite, WF 4.0 is really a 1.0 product.

I hope I’m wrong on the second part of this and that they’ve covered an adequate range of real world scenarios that it will at least be solid. And WF 4.0 will probably be an interesting tool in the core scenarios, particularly those around WCF integration. I’m not saying it’s useless by any means. I’m just saying it’s probably not going to change your world.

I’m still angry about the lack of a state machine. Certainly WF 3.0/3.5 state machines will still work as well as it ever has, and if you can manage deployment and avoid the layout being trashed by the IDE (hint, keep your layout files ready for restore from source control or edit them to find your lost states). But I’m angry because you and I can’t give our customers new products that just ignore half the scenarios for the old product. We can’t toss out all the work our customers have done (interop is present but limited). We just can’t do stuff like this!

I know the WF team has been working hard and that there are really smart people on the team. I don’t know what went wrong, but the problems with WF 4.0 became the problems of the early adopters, and shafting early adopters (especially when two years still involves “early adoption”) is very bad business.

In fairness, they offered a flowchart solution which will on paper accommodate some of the scenarios where state machines made sense. It’s just the wrong solution, and a horrible decision to do something new while ignoring previous functionality - state machine should have had a higher priority.

The vast majority of non-trivial workflow scenarios are best solved with a state machine. It’s based on a quick analysis. Theoretically all state machines can be expressed in flowchart terms and all flowcharts can be expressed as state machines. So in theory as long as you have either you’re covered. But it doesn’t quite work out that way. Business systems have non-trivial numbers of states. As the number of states increase, the number of legal paths between them increases. The theoretical increase is huge, but unimportant because many states are illegal. However, a sufficient number of legal states remain that as systems become more complex, the number of paths in a flowchart increases very fast. Paths are added more often than states (paths are the most common unexpected side cases). Thus, the system also remains more stable if the initial analysis is against states, not paths. Finally in many business systems, thinking of the problem in terms of states simplifies analysis.

The good news is that WF isn’t the only new thing in .NET 4.0 and in my next couple of posts, I’ll talk about what I now think are some of the most important features of the next release.

I got a Twitter on MEF resources, so I wanted to at least post the resources slide for my MEF/MAF talk. My big question, is what else do you want. Do you want more technie stuff on the core, more extensibility details, more stuff on the WPF editor, more on scenarios and value proposition, to know why I think composability is the most important thing since OOP/OOD (yep, I rank it above generics), or something else entirely? I think there's at least a half dozen people trying to get the right stuff out on this, but we don't need to shotgun this.

I actually modified this a bit if you were in the talks, but it’s a start – definitely not the final word:

• http://codebetter.com/blogs/glenn.block

• www.msmvps.com/blogs/kathleen

• Ask Kathleen, www.VisualStudioMagazine.com

• April 2009 for MEF/System.ComponentModel/Composition discussion

• April 2008 for MAF/System.AddIn discussion

• Hanselminutes: www.Hanselminutes.com

• #148 About MEF –Glenn Block

• #152 March 5, 2009 – Kathleen Dollard

• #147 VS 2010 WPF editor interview

• Also, good Bob Martin interview

• DotNetRocks: (me #436 April 9, 2009 -also, #304, 171, 121, 63)

• MEF download - www.codeplex.com/MEF

• MEF provider model and other interesting Andreas/TheCodeJunkie work www.codeplex.com/MEFcontrib

• PDC Videos

• TL49 (MEF-ify BabySmash)

• TL33 (Glenn Block) at PDC

• Pipeline Builder download: www.codeplex.com/clrAddin

• Kids, don’t try MAF without this

• Channel 9 video on MAF: Daniel Moth (search for System.AddIn)

It’s not up yet. You may have noticed. And if you downloaded what we’ve currently got up at AppVenture, you probably noticed that it was nothing like what I discussed on Hanselminutes or DotNetRocks.

Why is it delayed?

Primarily because I have had a very hard time drawing a line around what should and should not be in the tool. I’m still having a hard time of that, and the first version won’t include everything I want.

Also, MEF is a new paradigm. It’s been rough at times to learn to rethink and debug differently. This has caused a few redesigns along the way, and lots of consideration for things like the large number of assemblies. These are things I think need to be pretty close to correct in the first go round. This process would have been much more difficult without Glenn Block’s help, and I really appreciate his help and friendship.

I began many parts of this project in VB. I love VB. But the open source aspect of the future of this project will involve primarily C# coders. That’s the world around me. I am happy in C#, I just went fairly far down the VB path and had to do a lot of translation. When I release it, it will be mostly C#, with some remaining VB in parts I want to rewrite (the UI) and places where VB offers benefits (XML literals).

I’ve also run into a half dozen really nasty scenarios, both in my code and the framework that took a few days to resolve.

Parts of this have been raw research. What should such a tool look like? I’m convinced it’s by far the best way to generate code, but the devil’s in the details. I think it will make it easier for you and others that I worked though these details, although as I look back at the simplicity of many parts of the solution I have plenty of self-doubt that I just started out stupid.

All these things take time, a lot of time. And I try occasionally to have a life, although the people in it would certainly argue my capacity for setting this aside.

When? I hope next week.

The purpose of MEF is to hook up parts. Parts are indentified by string contracts.

MEF (and tools like it) empower fully composable systems. I use that term to describe a system in which the system no longer knows what is going on. What you write empowers an ecosystem specific to a problem domain it does not solve the problem.

In my case, MEF empowers a harness where processes, generally code generation templates, are run in a specific manner (such as with pre and post processing). I don’t care what your process does. At one level I don’t care about what services it needs or what metadata it uses (although in another post, I’ll explain why I actually do care, the harness definitely doesn’t care). This is a tremendously powerful environment for certain problems, precisely because it becomes an ecosystem solution, not a proscribed solution.

It is also a very different environment where we’re still working out the rules as an industry. One area where the rules shift is in assembly organization. There are always two core drivers for assembly (project) organization – deployment and isolation. How does this play out in a MEF system?

Let’s assume UI isolation. MEF systems generally solve a complex but finite problem. It makes sense for them to be class libraries that are distinct from any single user interface. In the code generation case, we know we will have a graphical UI and a command line. I suspect I’ll have a kind of sucky graphical UI and someone will make a pretty one and starting in 2010 embedding in Visual Studio becomes more realistic. That’s 2 in any single deployment - one of the UI’s and a class library that is the core of the system.

A core goal of a composable system is to replace direct references with indirect references. This means interactions will generally use interfaces which belong in their own library for isolation. These interfaces will generally have a default implementation. While technically these defaults could reside in either the interface or the core library directory, putting them in either location means they will always be available and you will have to work harder for the user to replace them. Thus I think four class libraries is the minimum complexity for a real world MEF solution.

Almost all fully composed systems should go to at least one additional set of assemblies. Fully composed assemblies contain interfaces which are directly referenced and used by the system core and interfaces that do not have this direct interaction. It’s software entropy if I force you to design an interface for extracted database metadata. I’ll give you one, although the core system could care less about it. For isolation and clarifying intent, I think separation is a better approach.

You could place the default implementations for indirect services (such as metadata extraction) into the same assembly as the default implementations for services directly referenced by the core. Nothing will break. But they will always be in the same catalog and export provider, requiring the complexity of selecting the correct one. Deployment is simpler if these are separate. I also think a parallel between implementation and interface assemblies makes the system overall easier to understand.

In some cases, including the code generation harness, there are specific sets of roles parts can play. A given use of the harness may use any combination of these. If the assemblies are partitioned across these boundaries, people setting up the MEF system can employ only those items they need. Among other things, this speeds the MEF discovery process. For example, in the code generator, there are a couple of categories of metadata, output, naming and miscellaneous services.

The specific roles the parts play is defined via the interface assemblies. If these assemblies are partitioned, programmers extending the system later reference the specific set of interfaces they want. With partitioned assemblies, they might need to reference more than one, but once referenced, the complexity of what the programmer sees is less (which could also be done with interfaces).

Let’s explore the role of the extending programmer a bit more. If there is no extending programmer (you or someone else) there is no point in a composable system. This extending programmer might be part of the original team, if the purpose of composition is simply flexibility in deployment, such as whether particular modules based on customer status. In many cases, including the generation harness, the system is really being created for the convenience of the extending programmer, and it is there scenarios that should drive the application design. You may be able to design your system so certain changes can be made without MEF knowledge (new templates in the code generation case) but the reason you’re using a composable system is to support programmers creating new parts.

This extending programmer works in their own assembly, which among other things allows you to issue updates to default assemblies without breaking their work. These assemblies reference interface assemblies. Is it easier for that programmer to select well partitioned interface assemblies and use them to better understand how the system works. If you partition your implementation assemblies to parallel the interfaces, deployment is simpler and if you fully implement an assembly, you can remove the default one.

There are a few blog posts including this by Chad Myers and this by Davy Brion that speak disparagingly about solutions with many projects. I do not actually agree with either post (because solutions are convenience only features and our architectures should not reflect Visual Studio bugs) but I do think they make some good points. And the fundamental agreement is that you should understand why you have a lot of projects if you do. In this case, it’s for isolation (a core tenant of MEF) and deployment. If you want only a couple of projects in your solution, you don’t want a composable system (although you might still use MEF for other reasons).

So much of what I do with technology turns out ten times harder than I think it should be that I’ll admit I walked very softly and fearful of the quicksand when I wanted to implement my own prioritization strategy for MEF parts. In the end, the MEF interaction was dirt simple.

For review, MEF has a prioritization strategy. When you are looking for parts, the first export provider that fulfills the request (contract and cardinality) ends the part search process (the GetExport(s) variation). However, the generation harness has more complex requirements that this allows.

To create an application, you’ll use a set of templates. Templates will have a specific purpose, such as a Select stored procedure. I’ll supply a default for this, but you may not like my default and will want to replace it.

I want to do this while maintaining complete ignorance in the template about what contracts you are working with. I can’t use the multiple export provider strategy because I want all the templates in all of the catalogs considered.

You also will probably want to replace metadata and service parts. This scenario is different because the template knows the specific contract. Because there is a specific contract request, the MEF native prioritization could potentially work, except that I want the simplicity of a single export provider to manage the templates and do not want multiple export providers/containers.

Parts, including templates, have a contract, which in MEF is a string. There will be many templates doing multiple jobs in the generation process. MEF just grabs everything that implements IProcessWrapper when it grabs templates. I want another level of selection.

All contracts involved in the prioritization strategy have an Id and a priority as MEF metadata. MEF metadata is just a dictionary of values so I can check whether an Id or priority exist in the export definition. I want to run only the highest priority template for each Id. An Id might be “SelectStoredProc” or some other descriptive name. For convenience I’ll supply constants for the common templates. My templates will have a priority of 0 and will only run if no other templates are available. I’ll include a different post about why I think we can all play nice with priorities and about how to override priorities, after I figure out how I’m going to do that.

To accomplish this…

I created a new aggregate catalog that derived from the MEF AggregateCatalog and override the GetExports method. 

Imports System

Imports System.Linq

Imports System.ComponentModel.Composition

Imports System.Collections.Generic

Public Class PrioritizedPartCatalog

   Inherits Hosting.AggregateCatalog

 Public Overrides Function GetExports( _

            ByVal definition As Primitives.ImportDefinition) _

            As IEnumerable(Of TempTuple(Of Primitives.ComposablePartDefinition, Primitives.ExportDefinition))

[[ In case I confused the C# folks, VB has “partial namespaces” which means it tries to tack namespace pieces like “Primitives” onto the stated namespaces and applies a little more clarity to namespace organization. I could have alternatively imported System.ComponentModel.Composition.Primitives]]

I want to grab all of the matching parts and do some additional filtering.

      Dim allExports = MyBase.GetExports(definition)

The next part involves some KinkyLinq (I should probably grab that url). When I’m doing hard stuff in Linq, I split things up into multiple queries. Linq will optimize this if it can, and my brain won’t explode from one overly complex statement. The first query creates groups by Id. The second grabs the highest priority item in each group:

      Dim q1 = From y In allExports _

               Group y By key = GetId(y) _

               Into Group _

               Select key, Group

      Dim q2 = From g In q1 _

                Select _

                ( _

                From y In g.Group _

                Where GetPriority(y) = _

                g.Group.Max(Function(z) GetPriority(z)) _

                ).FirstOrDefault

      Return q2.ToList()

   End Function

Not all parts will opt into prioritization. This is entirely optional. Managing this with priority is relatively easy. Any part without a priority has a priority of zero. However, managing Id’s for things that opt out of prioritization is a little more complex.

   Private Function GetId( _

            ByVal tuple As TempTuple( _

                Of Primitives.ComposablePartDefinition, _

                   Primitives.ExportDefinition)) _

            As String

      ' The guid means any export that opted out of prioritization is included

      Dim idObject As Object = Nothing

      tuple.Second.Metadata.TryGetValue("Id", idObject)

      Dim id = CStr(idObject)

      If String.IsNullOrEmpty(id) Then

         Return Guid.NewGuid().ToString()

      End If

      Return id

   End Function

If the ID exists, it’s used. If the Id does not exist, a Guid takes its place for the purpose of the filter. That means anything without an Id will always be considered the sole member of a unique group and will always be included since one member is selected for each Id.

The amusing thing about this solution is looking back in retrospect at where I spent my time. I spent a lot of time figuring out what could be done with MEF. I spent some time doing something wrong with Id’s at the metadata attribute level (which does not need to be complex) and spend some time on that KinkyLinq statement. Thanks to Glenn Block, Jay Harlow, Rob Teixaira, and Andreas for their help.

What I didn’t spend much time on is actually fitting the solution into LINQ. Adding this additional filtering involved only the code here (and the helper function for GetId() which you can predict). How simple can you get? The extensibility of MEF is what really makes playing with it worthwhile. If you hit a problem, like my unique filtering needs, you can probably find a way to work around it.

First, hugs and kisses to the people at Microsoft that made the decision to include T4 in Visual Studio 2008 instead of having it only available within the separately downloaded DSL toolkit. This indicated an important commitment by Microsoft to T4 and was a really positive step in code generation over all. Thanks guys!

I think the option wasn’t to have T4 done right or wrong in VS 2008, but to have T4 done wrong or not to have it. I’m happy the decision to include it was made, but I also think people need to understand what’s wrong with it.

First, the T4 template engine runs code in place and puts the result in the same project as the template in a dependent location. Huh? When would you ever want this? A core purpose of templates is to reuse templates across many projects, and once you get to working with and debugging the project, the template should be in the background, not the foreground. Oh, and this design strongly implies a single output file when single file output has been pursued by no one outside Microsoft (and by the way, Microsoft, we hate single file output). The folks working on the T4 Toolbox have shown us ways around this, but they are hacks giving templates inappropriate responsibilities for organizing the generation process. Templates should have the single responsibility of defining what to output.

Second, templates do not have a mechanism for passing parameters. Huh? We always pass parameters to templates, and we are almost always looping when we do it. There are hooks to pass parameters in T4, but you must understand some pretty complex plumbing to write into these hooks and it takes a few hundred lines of ugly code and internal, on the fly code generation.

Third, templates represent two interleaved sets of code. It’s the nature of templates and in T4 this is C# or VB code embedded within the artifact definition, which is often the definition of C# or VB code. Ad in the Microsoft provided editor its all the same color (black). Huh? Did you ever notice that these are nearly unreadable when displayed in a single color?

Sure you can fix these problems on your own, but everyone doing things their own way both wastes time and increases software entropy. Software entropy is the converse of feedback systems, and I’ll be talking more about that in relation to architecture expression in the next few days.

There’s good news here. Clarius provides a colorizing editor as a community edition to get you started. If you want fancier features, their main product is just $100. We’re releasing a community generation tool (also for free) as the AppVenture Community Generation Harness. This harness is a fully composable generation harness that will initially deliver with support for T4 and class based generation (such as VB 9 XML literal generation). It’s got the basic feature list in this entry. That’s in review at the moment and I’ll post here on my blog when it’s available at www.AppVenture.com. Using these two tools together, you’ll have a very workable generation environment based on what is a really a very sweet template language in T4 or VB 9 XML literal code generation, or any other type of generation someone writes a wrapper for. And thanks to Clarius and AppVenture, it’s all free to you.

I’ve been contemplating how to organize MEF assemblies. I think the processing I did establishing the first cut at organization, and the shake down of that strategy, may be interesting to other people designing MEF systems.

As a quick review, MEF lets you throw parts into a MEF container and sort out how parts work together at runtime. Parts are recognized by a string identifier. I’m almost always using interfaces as the contract and the interface names as the identifiers. Parts reside within assemblies and in the common case assemblies are discovered because they are grouped into anticipated directories.

With this approach, only the part implementing the interface and the part that is using the interface need to understand the interface or explicitly reference the interface’s assembly. And since parts are discovered and loaded at an assembly level, the granularity of implementing assemblies also controls the granularity of the load. I care about the assembly granularity of contract/interface assemblies so excess stuff can be avoided and naming conflicts (resolved via namespaces) are minimized. I care about the granularity of implementation assemblies because until I attain a priority system with additional granularity, prioritization/defaults are only as granular as their containing assemblies.

At one extreme, all interfaces reside in one assembly and all implementations reside in another. It doesn’t make sense to put them into the same assembly as then hard coded references exist and ensuring isolation is difficult. At the other extreme, every interface and every implementation resides in its own assembly. I think both of these extremes are a terrible solution. That’s because this composable system (and I would think any composable system) have parts with very different roles and lineages/history. In the simplest sense for a generator - metadata providers and templates are fundamentally different and could easily be provided by different teams.

Initially I thought the primary consideration should be the implementation deployment, but Phil Spidey pointed out in the MEF discussions that the interface organization is more important, because once released to the wild it might be hard to fix.

I decided on six contract assemblies:

I’ve used a few criteria for this design:

Interfaces that are used by the system and therefore can’t easily be changed reside together in CommonContracts. The template harness also references CommonOutputServiceContracts but this is in a separate assembly because it has a distinct purpose, may evolve on a different time frame and you are far more likely to provide alternate implementations for output than for the core interfaces.

The naming service is also a separate assembly because it is a distinct purpose and some people will certainly supply alternate implementations to manage human languages other than US English. Both the output service and naming service are a few distinct interfaces that work together. I also had a few odd ball interfaces and decided to go with a grab bag of miscellaneous interfaces rather than a separate assembly for each interface. Time will tell whether that is a good decision.

I initially put the two metadata interfaces into a single assembly, but I think it’s quite likely that these interfaces will evolve separately and almost certain that they will be implemented independently.

I’d like to note that the first version of the harness, which is almost, almost done (a separate blog post) will be a CTP /alpha level release. I will take feedback on the interfaces and I do expect them to change. A core part of the composable design is that you can spin off your interfaces/implementations so while these changes will be breaking, you can uptake them at your own pace.  

Scott and I sat down at the Rocky Mountain Trifecta 1 to discuss code generation and other fun things. Check out show 170 at www.Hanselminutes.com

If you aren’t currently creating T4 templates, skip this post as a rather geeky exploration of something you should never have to touch. Hopefully in another few days you’ll harness to take care of this ickiness.

T4 templates should know their own output file name! This is not a function of the host or the harness or anything else. We want to wrap up the responsibility of creating a template in one location (with any redirection necessary by the class to get the job done and support single responsibility).

How to do this? My first idea was way nerdy wacko cool – use MEF! It’s new, it’s cool, everyone loves something with such a cute name!

The first step to MEF-ifywas moderately easy – add a class in the T4 template that has an export (or a property, or whatever). This just involves usual ickiness of setting T4 references, imports etc. Messy, but if you are working with T4 hosts and engines, you know this stuff. Got that done.

The code for the MEF export approach might look like:

<#+

[Export(typeof(IT4Template))]

public class TemplateSupport : IT4Template

{

     public string GetFileName()

     {

         return "Fred";

     }

}

#>

Then, comes the harder part – where do you deal with the MEF container? Some place in looking through this, I realized why this is NOT a good use for MEF. Two reasons actually – First I know exactly where the value should come from. I suppose I could argue for some sort of default service that uses an embedded naming pattern, but I already know that embedded naming patterns get ugly over time. XSLT will require a naming pattern (or call backs which are problematic during testing), but MEF and XML literals, etc will not. We need a single location, the template, to tell us the output file name. And it needs to run code to determine the name. I can’t predict for a pattern language everything someone will want to do, leaving people to extend a pattern service – well complexity goes up.

The second reason is that there is another way that is demonstrably less complex. To meet this bar, we need code simpler than what I showed above.

The second approach I considered was a value holder service for the host. Basically the template could request a service (which could the template itself) that supported the service and simply set the value. Then after the template ran, the engine could retrieve the filename from the host, which was holding it for everyone’s convenience:

<# (this.Host as IValueHolderService).SetValue("FileName", "Fred"); #>

That’s still pretty ugly. It occurred to me that we already have a value holder service, we just call it properties and it contains parameters to the T4 template. I’m just going to swipe a slot in the properties dictionary for my use, which by implication means you can’t use the property name I use. I decided it would not be nice to force you to avoid having a property named FileName with some other purpose than the output file name. The name needs to be exquisitely precise, because I am messing in your logical symbol space: TemplateOutputFileName does the trick for me (I’ll be happy to take other suggestions).

Now, you can just set the name of the output file in code, using whatever additional functionality you need. At the level of the T4 template code, you are just setting the value of a variable. The result is something like:

<# TemplateOutputFileName = "Fred"; #>

This fulfills my goal of a very simple way for you to set the filename inside your T4 template while allowing you to run any code you. There will be some T4 geeks that do a double take at the fact this is not a directive, but directives do not allow internal template code to run. If this gives people heartburn they can either declare the property (I won’t redeclare it) or I’ll also add support for a directive if people really want it.