Leaning Into Windows : Code Generation

MEF Assembly Granularity

Kathleen — Sun, 15 Mar 2009 15:25:00 GMT

I’ve been contemplating how to organize MEF assemblies. I think the processing I did establishing the first cut at organization, and the shake down of that strategy, may be interesting to other people designing MEF systems.

As a quick review, MEF lets you throw parts into a MEF container and sort out how parts work together at runtime. Parts are recognized by a string identifier. I’m almost always using interfaces as the contract and the interface names as the identifiers. Parts reside within assemblies and in the common case assemblies are discovered because they are grouped into anticipated directories.

With this approach, only the part implementing the interface and the part that is using the interface need to understand the interface or explicitly reference the interface’s assembly. And since parts are discovered and loaded at an assembly level, the granularity of implementing assemblies also controls the granularity of the load. I care about the assembly granularity of contract/interface assemblies so excess stuff can be avoided and naming conflicts (resolved via namespaces) are minimized. I care about the granularity of implementation assemblies because until I attain a priority system with additional granularity, prioritization/defaults are only as granular as their containing assemblies.

At one extreme, all interfaces reside in one assembly and all implementations reside in another. It doesn’t make sense to put them into the same assembly as then hard coded references exist and ensuring isolation is difficult. At the other extreme, every interface and every implementation resides in its own assembly. I think both of these extremes are a terrible solution. That’s because this composable system (and I would think any composable system) have parts with very different roles and lineages/history. In the simplest sense for a generator - metadata providers and templates are fundamentally different and could easily be provided by different teams.

Initially I thought the primary consideration should be the implementation deployment, but Phil Spidey pointed out in the MEF discussions that the interface organization is more important, because once released to the wild it might be hard to fix.

I decided on six contract assemblies:

CommonContracts	Interfaces referenced by the template harness itself
CommonDatabaseMetadataContracts	Interfaces sharing database structure
CommonDomainMetadataContracts	Interfaces sharing business object structure
CommonNamingServiceContracts	Interfaces for a naming service
CommonOutputServiceContracts	Interfaces for outputting data, including hashing
CommonServiceContracts	Miscellaneous interfaces that don’t fit elsewhere

I’ve used a few criteria for this design:

Interfaces that are used by the system and therefore can’t easily be changed reside together in CommonContracts. The template harness also references CommonOutputServiceContracts but this is in a separate assembly because it has a distinct purpose, may evolve on a different time frame and you are far more likely to provide alternate implementations for output than for the core interfaces.

The naming service is also a separate assembly because it is a distinct purpose and some people will certainly supply alternate implementations to manage human languages other than US English. Both the output service and naming service are a few distinct interfaces that work together. I also had a few odd ball interfaces and decided to go with a grab bag of miscellaneous interfaces rather than a separate assembly for each interface. Time will tell whether that is a good decision.

I initially put the two metadata interfaces into a single assembly, but I think it’s quite likely that these interfaces will evolve separately and almost certain that they will be implemented independently.

I’d like to note that the first version of the harness, which is almost, almost done (a separate blog post) will be a CTP /alpha level release. I will take feedback on the interfaces and I do expect them to change. A core part of the composable design is that you can spin off your interfaces/implementations so while these changes will be breaking, you can uptake them at your own pace.

Another View of Code Generation

Kathleen — Mon, 03 Mar 2008 14:12:00 GMT

Karl on WPF posted this on code generators.

Karl appears to be largely talking about issues within existing MS code generation tools. The underlying problem, is that Microsoft does not have a code generation group that understands code generation from an application perspective and acts as a core resource to demand standard metadata, force Visual Studio support, explore problems like the attribute issue Karl raises and a host of other issues. There’s certainly a team around the CodeDOM, but it is entirely inappropriate for our use. And there’s one around DSL, but it’s not language neutral and it’s not incorporated by teams.

For the seven years I’ve been doing code generation and talking about it outside Microsoft I have never found a single person inside Microsoft that understands code generation the way Karl, Steele, Miguel, and the dozens of other people that actually fight through using code generation understand it. While I think I’ve had a few important ideas, the overwhelming majority of what I’ve thought and written is the same thing almost anyone who spent as much time as I have with code generating real applications would come to. This is NOT magic; it’s NOT core research anymore. It’s a well known discipline with well known problems and it’s about jolly time Microsoft stopped playing at the edges and got serious about it.

Metadata is my second core principal and the lack of standard metadata that Karl talks about is a real problem. If I a genie gave me three wishes, I would stop world hunger, turn half the teachers in elementary schools into men, and poof! create a rich robust standard metadata structure. OK, maybe I’d pick my priorities a little different if the genie actually shows up.

We sort of have standard metadata now. The problem is its too wimpy and it was designed without an extensibility model. The standard metadata is the Entity Framework edmx, and it’s a huge step. The problem is the combination of wimpiness and lack of extensibility model. As soon as we add something simple like plurals, you and I do it differently and the metadata is no longer standard.

You and I will have some extreme side cases, but 99% of our metadata will be the same content and must use the same approach. For example, Karl wants to put the “pull-down” data in the table (the high school graduated from in the college student table). I think he’s wrong. Foreign keys are always combo boxes or another lookup style and the primary key table (the high school table) should be responsible for telling the outside world what it needs to display:

· I’m a static lookup – you can cache me

· The short name for each row is in this field

· The long name for each row is in these fields concatenated like this

· I have about x records so you know what style of UI makes sense to you

· I can be filtered on these fields (optional- country/region problem)

This illustrates that metadata always looks simpler at first blush than when you spend years with it. But there are a dozen people in this country that could come together and crate a metadata standard that we could live with for a very long time. And it wouldn’t look like the gobbledygook that OMG created. Simple clear rich metadata as the bar we all reach and metadata would never be the same. Tools would quickly arise to fill that structure - including UI tools and storage models for the parts of it you can't extract from databases. The only way this can happen is for Microsoft to get serious about the problem, call a summit, then be willing for the result to differ markedly from anything they currently have leaving a lot of work for all of us to do.

As a side effect, this would open the door for model driven (as opposed to database driven) design.We're using databases as our prime source because a) we know how to make them and b) it's the only structure we can guarantee will have meaning in five or ten years. No metadata structure today has that guarantee. Standard metadata would remove that major hurdle in improving the overall we create applications. Code generation is one little cog - but since its broken the whole machine doesn't work. Fixing it will expose the next cog that's broken, but there is a sane way to develop applications and writing 2 million lines, or even 10,000 lines of semi-static code is not it.

In the meantime, look for the GenDotNet generator to isolate metadata in a separate pluggable layer to allow use with a variety of metadata formats. This insulates you from many types of metadata changes. It's the best we can do today.

Hopefully I’ve finally found a puppy and I am desperately trying to tie up loose ends to drive to Ogallala Nebraska so I’m putting this post up without my usual edits. Please let me know if I garbled any of it.

Rethinking “Nearly VB”

Kathleen — Thu, 28 Feb 2008 15:56:00 GMT

Bill McCarthy added a comment to my blog which I wanted to answer:

So why not use VB for the templates but C# for the initial output rather than some "Nearly VB" . Doesn't C# address every issue you've raised ?
But I am curious as to what about issues that are language specific, such as declarative event wiring, optional parameters etc ?

C# fixes the majority of the issues I raised, except ambiguity in closing brackets. If you assume that the closing of a structure will always be at the same level and outside embedded expressions such that you maintain symmetry in relation to the evaluation stack, you can resolve the closing brackets. Retaining symmetry in closing brackets means that the following will work:

Return _

<code>

if (x == 1)

{

<%= MoreStuffFunction() %>

}

</code>.Value

Any variation of the following will not work:

Return _

<code>

if (x == 1)

{

<%= MoreStuffFunction() %>

<%= "}" %>

</code>.Value

Which means among other things you cannot do:

Return _

<code>

if (x == 1)

{

<%= StuffFunction() %>

<%= If(z, "}", MoreStuffFunction() & "}") %>

</code>.Value

But you can rearrange it to:

Return _

<code>

if (x == 1)

{

<%= StuffFunction() %>

<%= If(z, String.Empty, MoreStuffFunction()) %>

}

</code>.Value

Bill believes this symmetry restriction is less onerous than the restrictions I placed on VB, especially the open/close parentheses on method calls. Another significant value to the C# first approach is that it’s much easier to recognize equals comparisons in assignment statements, and some of the null comparison problems I’m currently ignoring will be lessened because C# does not allow certain comparisons with nullable that VB allows.

While Bill convinced me that the C# first template was not nearly as difficult as I imagined it, by convincing me the restriction on the location of the close brackets in symmetry with the open was reasonable. However, he didn’t convince me to change my current work. VB first is the best scenario for my current client and I think if we have the possibility we should try to supply both so people can write and maintain their templates with the output code that they prefer, and prefer to debug the first version of the output in. Hopefully this can happen, but the most important thing to me at the moment is getting a working version out to you to play with – I don’t want to derail that with a second template converter/preprocessor. If someone else wants to work on that…J let me know.

Template Languages and "Nearly VB"

Kathleen — Mon, 25 Feb 2008 14:12:00 GMT

The templates I’ve been talking about require very specific language features of the VB compiler and language neutral templates do not allow any ambiguity in the code output in the initial template.

The template itself must be in VB because it’s required for embedded XML – the code blocks. The code blocks are essential for understanding which code to translate when creating an alternate language template in a pre-processor. Code in strings would be impossible (or nearly so) to translate at the template level and translating at the output level would have many issues including debugging and performance. There are tools available that translate normal source code, and you could do that, but I’m not sure why. It’s a lot of extra variables, when translating the template offers faster performance and more reproducible results. Sticking with template translation - the code block clearly indicates to the template preprocessor where to switch into translation mode.

The language output by the initial template must be VB, or “nearly VB.” Even if your primary interest is C#, a language neutral solution requires that the initial template have no ambiguity. Sticking with familiar and well supported languages is helpful because the initial output can be tested in VB, isolating problems in the template from any in the template translator/precompiler. This requires a non-ambiguous language I’ll call “Nearly VB”. If you’re strictly interested in C#, and have no interest in language neutrality, you can, of course, use VB’s XML literal code blocks to directly output C# code.

Ambiguity breaks the ability to build language neutral templates because the preprocessor has very little idea of the current context. It cannot understand whether a particular close curly bracket is an End If, a Next, an End Get or something else. Unfortunately, Visual Basic is not totally ambiguity free either, which forces the concept of “Nearly VB” rather than just normal VB. Nearly VB has one syntax change and a couple of extra rules when compared to VB.

VB is ambiguous on parentheses. It uses parentheses to include both method parameters and indices. VB is also ambiguous when it comes to case. To solve this in templates, use square brackets to indicate indices and parentheses for normal method calls. The C# compiler will help you find the problems when your C# output files fail to compile. The VB output can easily replace the square brackets with parentheses when outputting VB files.

At the moment I’m not convinced that the other meaning of square brackets – allowing identifiers to match keywords – need to be supported very well. There aren’t that many keywords and simply avoiding them seems an easier solution. You can support them if you escape the character via the \x20 escape pattern and the ASCII character (/x28). OK, that’s not very pretty, so a shorter escape sequence may make sense if people run into this very often.

Case insensitive is really another way to say “case ambiguous”. Language neutral templates require that you correctly case all symbols, the preprocessor can manage the keywords it’s translating, but you’ve got to get the symbols correct. Consider a Symbols class with constants, which also provides Intellisense while you’re creating your templates.

VB is sloppy in not forcing you to include the open/close brackets after a call to a method that does not have parameters. In a broader perspective this is ambiguous because in C# the presence or absence of the parentheses indicate whether you want to call the method or grab the delegate. While that particular ambiguity is resolvable because VB would require the AddressOf operator (or a lambda expression), I’m not tracking symbols. So I don’t know whether your symbol is a method, variable, or property. Thus, I don’t know whether the parentheses is needed. For language neutral templates, you add the parentheses on all method calls.

NOTE: I actually explored whether this problem is solvable, and I believe it is not. I don’t think it’s that much to ask you to include the parentheses correctly – it’s just a place we VB coders have historically been lazy.

So, to allow language neutral templates:

Use basic VB syntax
Use square brackets instead of parentheses for indexes
Maintain consistent case for all symbols
Include open/close parentheses for all method calls
Avoid keywords as symbol names or escape the surrounding brackets with the XML escape sequence
Rather obviously, avoid features unique to VB

I’ll do another post in a few weeks on issues around spots the two languages inherently work differently. There will be more items on this list, particularly around the management of nulls in relation to operators.

I do not dream that I’ve covered everything. The only way to ensure language neutral templates is to create them, ensure the code is are syntax correct, compile and run in VB and then create the similar code in C# and make sure you valid syntax, clean compile, and can run the finished applications. After the upcoming preprocessor has been out for a while we will have a better idea how you can break it and *** the holes where you can. But issues that involve ambiguity will have to be solved by the template author.

Validation Information in Metadata

Kathleen — Tue, 19 Feb 2008 14:17:00 GMT

Mike asks:

Just curious if your metadata also contains validation rules or not? Things like property is required or range of valid values.

It could include them in three possible ways – it currently uses one and I’ve had two others working in the past that may be resurrected.

The metadata that the database inherently knows is automatically transferred - this would be nulls and string length. How well nulls are handled is up to the architecture, but the metadata definitely knows what's nullable.

I've experimented with two additional approaches: using extended properties and parsing the TSQL of check constraints. The first would work for simple ranges and other predictable data sets, but it puts information in an unexpected place. I currently can’t justify it over placing validation in known places in the handcrafted code.

Using check constraints leverages existing information so is a "good" thing. Unfortunately, no one ever seemed to care about the months of work I put into that five years ago so I let it stagnate. Since I know more now, I could resurrect that work, but honestly I don't think I'll get time soon.

The problem is that most people just don't put check constraints in the database very often. I find that unfortunate for many reasons, but it becomes a chicken and the egg problem. People don’t put the constraints in the database because they’ll have to restate them in the business layer for decent usability. This initiative doesn’t get attention to solve that problem, because the check constraints aren’t already there. Perhaps the time is ripe now. I would love to include check constraint based validation in the Open Source version that we plan to start up on Code Plex this week or next (public within thirty days after) – at least a framework for it.

Check constraints are closely related to defaults because both require parsing TSQL. Turns out, over the years folks have been primarily interested in defaults of “now”, new guids, and raw values. Today or Now are pretty easy because it’s just a straight up translation between a SQL function and a .NET function. Any straight up translations like that can be defined in sort of a metametadata (hate that phrase) layer. I handle all three of these scenarios in my metadata extraction tool (a metadata extraction tool will be part of the CodePlex project).

I think validation should be stated in the business layer in rules. I wasn’t doing this five years ago so the whole process of incorporating validation from check constraints will be vastly simpler. Instead of code to code, you need to recognize a category – such as a bound range (the most important) and parse out the bounds into a structure usability by a specific rule. Then another rule is “there’s a check constraint and I think you need to validate based on it, but I can’t write it so you need to.” The architecture could enforce some code being written in response to that rule. To state the change from five years ago, the metadata wouldn’t contain code but the statement of which rule and its parameters.

Validation in triggers would seem, at least to my weak TSQL mind, to be exceedingly difficult.

So, the basic answer to Mike’s question is “some, but not all of the really important scenarios are covered, and I don’t think you’ll ever cover all scenarios”

Runtime vs. Design/Compile Time

Kathleen — Thu, 14 Feb 2008 17:21:00 GMT

Chris asks:

At what point with code gen / templating do you start to think about doing all this codegen at runtime instead of compile time?

And if we were to be doing it at runtime would be be better served by using a dynamic language such as ruby to program in?

That's a good point. In a perfect world, there would be no need for code generation. We would write nothing but business/domain specific code and everything else would just happen. But for well over twenty years we've been aiming for that perfect world and we seem only a few baby steps closer than in 1987.

In an imperfect world, we have two basic choices, manage an architecture and run a lot of code at design time or manage an architecture and run a lot of code at runtime. Both require a fair amount of configuration.

And both can offer the real long term benefit of switching away from coding code – which is transitory and dying before you even finish coding. We want to switch away from coding code and toward creating metadata which is a true business representation. Of course metadata changes. But it changes at the speed the business changes – not due to artificial technology shifts.

For my effort – I want the extra code run at design time to offer the best possible runtime behavior, including performance. I can extend an architecture I’m expressing in templates far easier than I can extend an architecture I am expressing via an OR/M tool. I also want to debug directly into code specific to the problem at hand – I want to debug through generated code, not an OR/M engine.

The next round of effort at making plumbing simpler from Microsoft is also code generation – Entity Framework. When someone gets the plumbing correct and we truly never need to care, we can turn off the code gen and go direct to whatever structure, however its’ done and completely ignore the problems code gen is primarily used for today. In the long term, we shouldn’t care about anything except the business problem we’re solving.

Dynamic languages certainly change the architecture. I'm looking forward to exploring them as the overall knowledge base expands. But we've been sitting with a pretty dynamic language in our laps for many years, the majority of us have programmed in it, and with the exception of precisely one person - everyone I know varies between mild distaste and downright hatred for it. I think a lot of Javascript's problems have been related to debugging and platform issues, and I realize that there are differences between Ruby/Python and Javascript. However, if we didn't fall in love with the dynamic aspects of Javascript in the last ten years, I remain slightly skeptical about dynamic languages in the next few years. Some of the differences, and our attitudes and skills may change as a result, are that the new languages have a broader platform base and we're increasing our understanding of how to actually use them as opposed to hacking them enough to solve some trivial website detail.

But at the core, the technique for expressing metadata into a working application is not half as important as metadata at the core of the application, however its expressed.

Open Source

Kathleen — Wed, 13 Feb 2008 15:31:24 GMT

It’s occurred to me that if you are following this and my DNR TV show a logical reaction would be “OK, so that’s a lot of hot air, where do I get it?” I intend for all of this to be released Open Source, on whichever site s hot when I release it. I hope I’ll start releasing pieces in just a matter of weeks. It will help a lot if it becomes “we” instead of “me”. So, if you’re interested n this stuff and you want to help, let me know and you can get this stuff directly as its rapidly evolving too much to publically post right now.

My current expectation of the order of release:

Metadata interfaces
GenDotNet metadata load
EDMX metadata load
Template infrastructure – what’s needed for the language neutral templates
Simple harness
Activity based harness

If you’re interested, you can contact me directly at Kathleen@mvps.org

Isolating Metadata

Kathleen — Wed, 13 Feb 2008 15:26:09 GMT

In code generation, metadata is the information about your application, generally about your database and definitions to express your data as business objects. If you use Entity Framework, your metadata is the edmx file which is displayed via the designers. If you’re using CodeSmith, the metadata is more subtle. Metadata can also be about the process itself. CodeBreeze in particular has a very rich set and extensible set of information about your application.

Since metadata itself is data – information - we can store it many ways. I’ve used XML for years. CodeSmith has used a couple of mechanisms including XML. Entity Framework uses XML. Metadata can also come directly from a database, although I think this is a remarkably bad idea and one of my code generation principles is not to do that – you need a level of indirection and isolation surrounding your database.

What I haven’t talked about before how valuable it is to have another layer of indirection between your metadata storage structure – your XML schema – and your templates. In my XSLT templates I could provide this only through a common schema – you can morph your XML into my schema so that’s indirection – right?

No, that’s not really indirection. It’s great to be back in .NET classes with real tools for isolation and abstraction. Now I use a set of interfaces for common metadata constructs such as objects, properties and criteria. I can then offer any number of sets of metadata wrappers that implement these interfaces via a factory.

The template programs only against the interfaces. The template could care less whether I am using entity framework, my own metadata tools, or something entirely different. I can write the same template and use it against Entity Framework’s edmx file or any other metadata format. That’s powerful stuff. Especially since you already heard that the template will run against C# or VB. That means in my world the only reason to have more than one set of templates against an architecture like CSLA is that they are pushing the boundaries and actually doing different things.

But if you don’t like this new templating style, you can use classes based on exactly the same interfaces in CodeSmtih (at least) and again free your framework and metadata extraction. You’ll still need VB/C# versions there, but you’re metadata input can use the same interfaces.

The interfaces is expressed in sets of classes that know how to load themselves from a data source. Each set uses a different metadata source – different XML structures or other format.

Isolated metadata removes your templates from caring what the metadata source is – beyond being something that could successfully fill a set of classes that implement the data interfaces. This is a very important step and one we need to work together to get right. What do you think I've left out of the current design?

CurrentTypeInfo and the Context Stack

Kathleen — Tue, 12 Feb 2008 15:46:00 GMT

Creating templates requires a lot of access to the thing you’re currently creating. That’s the current output type, which as I discussed in yesterday I suffix with “Info.” The CurrentTypeInfo is thus what you’re currently outputting.

I neglected to clarify in that post that the Data and the Info classes are in entirely different assemblies. The Data classes could be used with any .NET generation mechanism, including (at least) XML Literal code generation and CodeSmith. The Info objects are relatively specific to my style of code generation.

The CurrentTypeInfo may not be the same throughout the file.

There are a few reasons to combine multiple classes or enums in a file. In some cases, that’s to nest them, and in some cases it’s just to logically combine them in a file. While FxCop has declared them unfashionable, I find nested classes tremendously valuable, especially for organizing Intellisense and keeping well structured classes and naming. If you’re working with nested classes, there is a good chance you’ll need to access not only the current type, but also the encapsulating type. I use a stack for this, and give control of pushing and popping TypeInfo objects from the stack to the templates themselves.

The base item on the stack is the current outer most class in the current file. Once you’ve pulled a class off the stack, such as by popping the base and adding a new base TypeInfo, you can’t access the previous version unless you’ve saved it.

Here’s where you see the flaw I mentioned yesterday – these classes could be in separate namespaces, and I don’t allow for that – yet. I’ll fix it later.

Remember the TypeInfo is the thing you’re outputting. The entity definition it’s built from is ObjectData in my semantics.

The stack is an extremely useful construct for this scenario. You have quick access to information about the class you’re currently outputting. You’ll frequently need this for type information and perhaps calling shared/static methods. You don’t want to recreate its name every time you use it because that would be redundant and hard to maintain. You can also access any of the containing classes, which again is useful in defining types and calling shared/static methods.

While I haven’t done this in CodeSmith, I expect this technique to be viable there. I’m not sure on other platforms, but it’s not specific to XML literal code generation.

Two Parallel Entities - Metadata and Info

Kathleen — Mon, 11 Feb 2008 17:08:00 GMT

One of the confusing things about templating is that you are writing two programs simultaneously and there is no way around it. My brilliant son may write a templating language for a class project, and this is exactly what he wants to address – that and the issue of escaping. You can’t avoid it, he just thinks they should look different – I’m hoping he doesn’t write one in Magyar.

One of the programs you’re writing (or maintaining) is the template program. There is logic in any non-trivial template, regardless of the template style. The other program is the one you’re outputting that will eventually run on your clients computer. In ASP.NET style templates, the output program is the overall template, and the logic is inserted. In XSLT and XML literal generation, the logic of the template program is the overall code and the output template is nested inside.

You are also working with two sets of data – the metadata you’re using as source and what you’re currently outputting. I call the elements in the XML I use as source metadata for .NET templates Object and Property for clarity in the XML. If I called them that in the templates, I’d encounter great confusion between the object definition in metadata and the class I am outputting. That’s particularly painful when it comes to properties.

I solve this by calling suffixing all metadata with the word “Data”. Thus I have ObjectData, PropertyData, CriteriaData, etc. Each of these classes contains the metadata on the corresponding item. I might have an ObjectData for a Customer that had a PropertyData for FirstName. The PropertyData would include things like the maximum length of the field and its caption.

I also need to describe the output – at least the way I’m building templates. I need information about the file I’m creating, the type I’m creating, and in certain locations in the template, the property, function, constructor, etc I’m creating. I identify these by suffixing them with the word “Info”. Thus I have a ClassInfo, PropertyInfo, FunctionInfo, etc. I do not have a CriteriaInfo because I am never outputting a .NET thing called a Criteria. It’s strictly metadata for .NET features.

In the template design I’m describing in this series, the DotNetBase class contains information on the file being output. To be picky, the namespace can actually be associated with part of a file, and I may add this flexibility, but I don’t do that very often in business code, so I have included namespaces at the file level.

Regardless of the template mechanism you use, you need to maintain a clear separation between the template logic and the output logic, and between the input metadata and state about what you’re creating.

Next post: CurrentTypeInfo and the Context Stack

Inheritance and Templates

Kathleen — Mon, 11 Feb 2008 16:25:00 GMT

One of the most valuable things about templates written in .NET code of any style is the ability to use inheritance. This is classic inheritance where local state allows you to push functionality into base class methods that would become cumbersome and have excess dependencies if treated as external utility methods.

Inheritance also allows polymorphism, which is nothing more than a big word for exchangeability – OK, that’s actually a bigger word. Templates are called by something. Life is simpler and harnesses practical if that something is known and predictable, thus it’s logical to have an interface in a common location as a contract to the template interface. If the base class implements this interface and provides functionality via MustInherit/abstract members, there’s further capacity for evolution.

There are three assemblies involved. The template interface is in a common assembly available to all other assemblies. The base classes are in a support assembly which can be reused across many template sets. The templates themselves are in a separate assembly. For now, I have all templates associated with a project in one assembly, but I may change that to isolate the SQL and the .NET templates. In any case, these template assemblies are sets of templates that work together.

Inheritance let’s you push common functionality into the base class. Now that we have extension methods they offer an additional mechanism to remove common functionality from the leaf class. I know Scott Hanselman has said he’ll cut off the pinkies of anyone who uses extension methods on classes you own, and I’ll show you when I get to that in a few days why I disagree (although it’s in the DRN TV show). But that explains why my base classes are focused on state more than functionality. The key state issue is what you’re currently working on. In a template, state is metadata. In a .NET template, the important metadata is the definition for the entity you’re currently creating. In my templates, the ObjectData member of the DotNetBase contains this information.

Stay tuned for the distinction between the input data you’re working on and the thing you’re creating, which explains some of those other members.

.NET Template Organization

Kathleen — Sun, 10 Feb 2008 23:55:00 GMT

So, now that you know where I’m going with this – language neutral templates – I want to step back to the basics. Even if you don’t want to build language neutral templates, there are things to learn along the way about making good XML literal templates, and why this approach might be better than your current code gen mechanism. Note, the examples in this post are NOT language neutral. Not surprisingly, there are specific requirements for language neutral templates and I want to show basic templating with XML literals, then the language neutral templates.

If you look at the XML literal sample I posted a few days ago you’ll notice that the entire contents of the output DataPortalFetch method is created within the template method “MemberDataPortalFetch.” This provide important organization as the most common challenging task in code generation is “I have a problem in my output code right here, how do I find where that is in the template. In a simple template this isn’t too hard, but in more complex templates such as CSLA it can be quite challenging.

Creating templates with an extremely predictable structure is very valuable in creating maintainable templates. To start with each template should have a one to one correspondence with an output class.

Within this class, the mechanism of the predictable structure is one of the primary differences from codes that are code with template segments (XML literal code generation) and templates with code segments (ASP.NET style template). In an ASP.NET template, the template must parallels the output and this organizes the template. In an XML literal template, you provide the organization with Visual Studio then providing navigation. Code is organized my regions, nested classes and member. Each is its own method preceded by Region, NestedClass or Member. This organizes the template. The entry points create a hierarchy working down to the local code (navigation mechanism #1):

   Protected Overrides Function GenerateFile() As String
      Return _
<code>
   <Serializable()> _
Public Class <%= mObjectData.ClassName %>
   Inherits BusinessBase(Of <%= mObjectData.ClassName %>)
   <%= RegionBusinessMethods() %>
   <%= RegionValidationRules() %>
   <%= RegionAuthorizationRules() %>
   <%= RegionFactoryMethods() %>
   <%= RegionDataAccess() %>
   <%= RegionExists() %>
End Class
</code>.Value
   End Function

And

#Region "Business Methods"
Private Function RegionBusinessMethods() As String
Return _
<code>
#Region " Business Methods "

   <%= From prop In mObjectData.Properties Select MemberPropertyFields(prop) %>
   <%= From child In mObjectData.Children Select MemberChildFields(child) %>
   <%= From prop In mObjectData.Properties Select MemberPropertyAccess(prop) %>
   <%= From child In mObjectData.Children Select MemberChildAccess(child) %>
   <%= MemberIsValid() %>
   <%= MemberIsDirty() %>
   <%= MemberGetId() %>

#End Region
</code>.Value

End Function

The carefully named methods also allow you to use alphabetical tools in Visual Studio including the combo box in the upper right of the editor and a class diagram (navigation mechanism #2) which works only because you can predict the name of the member you want from the name of the output member and no other members begin with “Member”, “Region” or “NestedClass”

The third navigation approach is that the template matches the structure of the output as closely as practical. Thus if the output has a region names “Business Methods” the template does as well. The order of items in the template is top to bottom closely paralleling the order of the output. This allows you to cruise down in the file.

Template organization is the first step to great templates.

The Punch Line

Kathleen — Sun, 10 Feb 2008 23:21:00 GMT

I mentioned a few days ago that there was a punch line for the XML Literal Code Generation. I planned to unveil this slowly, but it just sprung out of the box when Carl posted episode #102 (which I thought was due for next Friday.

And I’m afraid that I need to add that I was sick during the taping and my brain running at half capacity. If anything isn’t clear, please let me know.

No matter.

You get to hear sooner.

I’ll be unveiling the details in about the same time frame, so you’ll have the big picture by next week.

So, the drumroll please…

I can create excellent readable templates that output code in Visual Basic or C# - no CodeDOM involved.

I’ll show you how to do that. Watch for more here.

XML Literal Code Genaration - Code again again

Kathleen — Sun, 10 Feb 2008 16:53:00 GMT

Crap.

For now, I'm just removing the coloring. Paste this into VS for coloring. It's much prettier:

   Private Function MemberDataPortalFetch() As String
      ' TODO: Add special handing for timestamp
      Return _
<code>
   Private Overloads Sub DataPortal_Fetch(ByVal criteria As Criteria)
      Using cn As New SqlConnection(<%= mObjectData.ConnectionStringName %>)
         cn.Open()
         Using cm As SqlCommand = cn.CreateCommand
            cm.CommandType = CommandType.StoredProcedure
            cm.CommandText = "get<%= mObjectData.ClassName %>"
            <%= From prop In mObjectData.PrimaryKeys Select _
               <code>
            cm.Parameters.AddWithValue("@<%= prop.Name %>", criteria.<%= prop.Name %>)
               </code> %>

            Using dr As New SafeDataReader(cm.ExecuteReader)
               dr.Read()
               With dr
                  <%= From prop In mObjectData.Properties Select _
                     <code>
                  m<%= prop.Name %> = <%= GetReadMethod(prop) %>("<%= prop.Name %>")
               </code>.Value %>

                  ' load child objects
                  .NextResult()
                  <%= From child In mObjectData.Children Select _
                     <code>
                  m<%= child.Name %> = <%= child.Name %> .Get<%= child.Name %> (dr)
                     </code>.Value %>
               End With
            End Using
         End Using
      End Using
   End Sub
</code>.Value
   End Function

XML Literal Code Generation - Code again

Kathleen — Sun, 10 Feb 2008 16:32:00 GMT

OK, that code listing got friend on load. Let me try again.

Private Function MemberDataPortalFetch() As String Return _<code> Private Overloads Sub DataPortal_Fetch(ByVal criteria As Criteria) Using cn As New SqlConnection(<%= mObjectData.ConnectionStringName %>) cn.Open() Using cm As SqlCommand = cn.CreateCommand cm.CommandType = CommandType.StoredProcedure cm.CommandText = "get<%= mObjectData.ClassName %>" <%= From prop In mObjectData.PrimaryKeys Select _ <code> cm.Parameters.AddWithValue("@<%= prop.Name %>", criteria.<%= prop.Name %>) </code> %> Using dr As New SafeDataReader(cm.ExecuteReader) dr.Read() With dr <%= From prop In mObjectData.Properties Select _ <code> m<%= prop.Name %> = <%= GetReadMethod(prop) %>("<%= prop.Name %>") </code>.Value %> ' load child objects .NextResult() <%= From child In mObjectData.Children Select _ <code> m<%= child.Name %> = <%= child.Name %>.Get<%= child.Name %> (dr) </code>.Value %> End With End Using End Using End Using End Sub</code>.Value End Function

XML Literal Code Generation

Kathleen — Fri, 08 Feb 2008 17:20:00 GMT

You must use VB for XML Literal Code Generation. Sorry, that’s life. I don’t have a crystal ball on this, but for now even if you’re a C# programmer you’ve got to suck it up and use VB. The code you output can be VB or C# however, but the template itself has to be written in VB. If you’re allergic to VB, get shots.

So, what does a microcosmic view of a XML literal template look like? I’ve translated a section of a CSLA template. CSLA is Rocky Lhotka’s architecture and this is a version 2.1 because I think it clarifies the template process:

Private Function MemberDataPortalFetch() As String Return _<code> Private Overloads Sub DataPortal_Fetch(ByVal criteria As Criteria) Using cn As New SqlConnection(<%= mObjectData.ConnectionStringName %>) cn.Open() Using cm As SqlCommand = cn.CreateCommand cm.CommandType = CommandType.StoredProcedure cm.CommandText = "get<%= mObjectData.ClassName %>" <%= From prop In mObjectData.PrimaryKeys Select _ <code> cm.Parameters.AddWithValue("@<%= prop.Name %>", criteria.<%= prop.Name %>) </code> %> Using dr As New SafeDataReader(cm.ExecuteReader) dr.Read() With dr <%= From prop In mObjectData.Properties Select _ <code> m<%= prop.Name %> = <%= GetReadMethod(prop) %>("<%= prop.Name %>") </code>.Value %> ' load child objects .NextResult() <%= From child In mObjectData.Children Select _ <code> m<%= child.Name %> = <%= child.Name %> .Get<%= child.Name %> (dr) </code>.Value %> End With End Using End Using End Using End Sub</code>.Value

End Function

This is .NET code, which you can see because the template output is inside a .NET function named MemberDataPortalFetch which returns a string constructed with XML literals. By offsetting with XML literals, the code of the function is easily seen – it’s in gray. All of the gray code will be in the output, none of the yellow, blue or black code will be in the output, although it will certainly affect the output.

Where replacements are necessary, XML literals have “embedded expressions” which I have the habit of calling “expression holes.” I’ll talk in another post about what mObjectDatais, but an embedded expression can contain whatever expression you need. An expression is something that returns a value within a single line. So, you can call methods and properties, but you can’t call a subroutine.

Embedded expressions can contain LINQ as shown in the third embedded expression. Here we want to add parameters for all the primary keys. Within the LINQ Select clause, I can use any string. Including the nested code block creates a more readable template and consistency, among other things.

A quick glance may leave you thinking that this is just a variation of the ASP.NET style template syntax used by CodeSmith, CodeBreeze, MyGeneration and others. However, it’s quite different. To begin with its organized into members of a template class. Then, the logic is expressed in expressions, not direct template logic. Finally, you can nest back and forth between output text and code to any depth you require. The code blocks within the LINQ statements above contain embedded expressions. These embedded expressions could be further LINQ statements with further code blocks, etc.

I think there are right and wrong ways to build templates with XML literals. I’ll spend the next couple of posts talking about template organization and metadata handling.

Let’s Talk Templates

Kathleen — Thu, 07 Feb 2008 22:07:00 GMT

Code generation templates have several competing pressures – they need to be easy to use and they need to be powerful. They need to be flexible but encourage best practices. If you ever say “my templates got me started then I had to customize them” you’re templating can be better. I’m not aware of any situation where you can’t isolate the handcrafted code from the generated code sufficiently to keep code generation active for the entire life of your application.

When I say full life cycle, I mean from initial prototypes to finally pulling the switch in a decade. Yep a decade. Code generation should aim for evolution and extremely long project life cycles. The metadata of the project can far outlive the technology, even outlive the database. That requires a deep commitment to code generation, and if you don’t have that commitment, the flip side is that code generation can pay back today. At that moment when it feels easier to pull a file out of code generation, or to repeat a similar block of code in two places instead of pushing it into your code gen templates, that’s where the commitment comes in. With or without it, you get payback; it’s just greater the deeper your commitment is to the metadata and templates.

My current client pushes me because he is even more committed to the decade long lifecycle based on code gen than I am. And he’s right. Every time he’s pushed to get something into templates or do code gen better, it’s taken less work and gotten us further in the short term than I expected.

What if you could combine the simplicity of CodeSmith with the power of XSLT?

And what if you could do it primarily in a language and architecture you already know, or should be learning anyway?

Interested?

This is the first post in a blog series to present this templating style. I also covered in a.NET Rocks show I recorded yesterday. I’m very excited about it, but I want to introduce the basics before I tell you the real punch line. This template style is the best way to generate code today and it’s based entirely on tools you’ve got on your desktop right now.

Look for more here and in the .NET Rocks TV episode scheduled to appear Feb. 15^th. The beautiful thing about the techniques I’ll show is that you can smoothly transition from simple to complex templates dialing in the exact complexity appropriate for your organization and application. And, just wait for the punch line.

Why You Care About System.AddIn

Kathleen — Thu, 03 Jan 2008 19:31:00 GMT

When I was fighting with AppDomains to support XML Linq code generation in my new Workflow based code generator, Bill McCarthy said “Hey did you look at System.AddIn” and I said “No, silly I’m not writing add-ins.”

Well, a few months later, I’m still trying to make it work, and have come to think it’s worth the trouble. So, first, what System.AddIn namespace offers, then why it’s so painful, then what I’m doing to fix your pain.

Simply put, System.AddIn provides abstracted/isolated app domain access. AppDomains are the boundary at which some security stuff happens, and the unit which must be unloaded as a group. You can load individual assemblies into an app domain, but to unload them, you need to unload the entire app domain.

There are a few scenarios where this is important – sandboxing code you’re running as a plug-in to your application being the one the designers had in mind. I want to use it so I can load my code generator and have it recognize changes in .NET assemblies that are generating code. With my first tool, I never solved this problem because I didn’t think brute force code generation prior to XML literals in Visual Basic made very much sense. You had a lot of the problems with XSLT (whitespace) and a nearly complete inability to search your templates (since we cannot search separately in quoted text. XML literal code generation is the best way yet to generate code – as powerful as XSLT and as easy as CodeSmith. Anyway, I can get carried away on that – it’s why I was willing to invest heavily in System.AddIn.

Along the way, I gained great respect for the complex model that supplies isolation/abstraction. If you’ve ever played with plug-ins you know that the first version of your app and its plug-ins is OK, but keeping things in sync while multiple synergistic applications evolve is nearly impossible. The isolation model means the host only speaks to an adapter, and the add-in only needs to speak to an adapter. The adapter’s functionality and the contract can change in whatever manner is needed. This model, combined with the app domain management may lead System.AddIn to have an important role in your application if your application needs to provide variants for individual clients.

Hopefully you have a good idea what sorts of things clients are going to want to customize, and you place this into an API you hit via the add-in model. If you got it 75% correct out of the chute, it would be a miracle, so the capacity for change built into the isolation model is what actually makes this work

Literally, you load code on the fly, with whatever security limitations you want, with the ability to unload at your convenience, and pick the correct code from what’s available in a specific directory location. Cool huh!

In WinForms, the WinForms threading model prohibits UI’s in the add-in. I understand this is fixed in WPF, although I haven’t yet written a WPF add-in user interface.

So, now that you have some idea why System.AddIn is worth the trouble, why is it so painful. How could I have possibly spent so long getting it running in a sample (I just output a single quoted string right now). To provide the isolation there is a minimum of seven projects/assemblies involved. These must be deployed in a very specific directory structure for the AddIn system to find the pieces it needs when it needs it. Then there is the error reporting problem – I’ve blogged about a particularly nasty “The target application domain has been unloaded” error. So, once you hold your mouth just right, and all your code is perfect, it’s cool. But how many of you right perfect code? And what’s this about an easy maintenance model if you have to change SEVEN assemblies to alter the API.

I’m working on an article and tool for my column in Visual Studio Magazine that will take either metadata for the API, or the interface and build the simple pass through model. This gets you started. Later when you have interface changes, the isolation model pays for itself, but at that point you understand what’s happening.

It’s going to be a pretty cool example of the “overwrite until edited” mode that my tool supports. Before I’ve used this for editable files that were pretty much empty. Now, I want to separate changes due to metadata changes – that could be significant – from those for actual mapping you did in the adapters. With luck partial methods will lead to a pretty robust set of code you can alter as you need, while still generating the main API stream.

I find it very cool to see so many fragments coming together.

What? No Code?!?

Kathleen — Sun, 28 Oct 2007 21:26:00 GMT

When I say things like “writing code bad, other stuff good” it comes out sounding radical and there a couple natural points of pushback. The most significant one is “yeah right, we put our business information in something else and that changes too - so we’re trapped forever in technology and like being addicted to food we just can’t stop”

Let’s nip this in the bud. The key difference between code and storing expressions of the intent of our app is that you can recognize, categorize and morph declarations. Sure, the syntax of any declaration is based on the technology its intended for. Just look at the massive difference between GenDotNet metadata and the metadata behind entity framework which solves a critical subset of the same problem. But, any metadata worth its salt can be transformed into any other metadata syntax. That’s why I’m excited about seeing the entity framework tools work tools work well. They create a standard mapping for any code gen anywhere. It’s just messy transformations that we know how to do.

Code is the opposite of isolation. It’s such a convenient mechanism that everything is thrown together. It’s ghoulash when we need a wedding cake.

If I write code for CSLA, the details of my classes are heavily influenced by the superstructure of Rocky’s library. I’m going to bury my stored procedure, parameters, data reader assignments, authorization, validation, etc into VB or C# code related to a specific library. All of these specifics are also required for a different library and a different set of libraries or C#, VB, Ruby, Python, F#, Post#. By separating out the known aspects of the job we’re doing – we can use it or not use it in the next great thing. I may not use the stored procs in Entity Framework.

While I have a lot to say trying to jar your world view into seeing code differently, it’s exactly that: kicking your world view. Code remains critical for two reasons. I generate my code, but I have to create code, not some other magic. If I don’t have code, I can’t debug. That’s one of the many, many things we forget about the 4GL disasters of the late 1980’s – you could no more debug the crap than fly to the moon. Imagine writing an application designed to run a refinery and hearing the programmers say “I don’t know why it does that, but try something that worked another time I didn’t know what to do”. A refinery! In a generated system, code becomes how the system tells you what it’s doing more than the reverse.

The other important thing about code is that as we move away from code as core way of expressing intent - the code that matters cannot be moved. You find business logic simply because it remains when all the potentially declarative stuff is removed. How exquisite we can make the code we have to write when it stands alone and we address it as a primary purpose of our existence. And how much more if it we’ll write when we aren’t busy plunging the toilets of our application plumbing.

Legacy Lessons

Kathleen — Sun, 28 Oct 2007 13:27:00 GMT

Visual Basic 3 to Visual Basic 6 had many lessons to teach us. We did not as an industry learn these lessons, so we are in the midst of repeating them. One of the most important lessons was not to combine business logic and user interface code. Well DUH!

But, why not? I don’t mean to be rude, but I’d really like to shout in your face and make you truly understand. Fly to Tibet, climb a mountain and meditate in a hut for a month until you can answer that question. Why was it bad to combine business logic and user interface code?

Which of course you probably can’t answer because we would not have been so stupid, as an industry, to make these mistake second time around if we’d gotten it the first time. Combining business logic with any technology dooms it to be rewritten from the ground up when we get a new technology – and code is technology. DOH!

You cannot combine technology with business logic. You cannot combine code with business logic.

The shiniest, prettiest toys and languages are tomorrow’s legacy. Except for movements away from technology and code, everyone at Microsoft if building tomorrow’s legacy tools.

Don’t tell me it’s impossible to change, .NET had all the underpinnings to do metadata based code generation, rules engines, and other techniques in the first release. I’m not the only one who has been using alternative development techniques. Sure it’s hard. It’s really, really hard. It is too hard for you to do individually on your project. Changing our development efforts so fundamentally that we treat code as a necessary evil is such a fundamental shift that we need combined efforts, tens of thousands of man hours and significant leadership. In today’s world, leadership means either Microsoft or standardized, combined third party efforts. The third party has made great advances in niches, but no full-scale strategy. It’s time for Microsoft to step in.

Several things are happening right now. The current crop of technologies will teach your team the painful lesson that .NET is not an inoculation protecting your project from technology change. We have legacy .NET code. That sucks. The solid foundation of .NET actually does the opposite of inoculating your application from change. It speeds up the pace. Microsoft can build amazing new toys - so they do.

Another thing happening right now is that forces are moving behind the scenes to make it easier, or at least more standardized. LINQ to XML in Visual Basic offers XML literals which have amazing possibilities for code generation. Today, it’s better in some ways and worse in others to CodeSmith, but there is a real possibility it will unseat XSLT for complex generation. We need to work out a boatload of patterns which I’ll take many posts to cover, but you can get a peek in Beth Massi’s blog. Also on the horizon is Entity Framework which offers a standardized way to map to metadata. I’m spending time customizing the architectures for a talk at DevConnections, which will make a great blog post later. Other things just below the horizon will also make your generation easier in the future. The earth is turning.

Before letting the details of new technologies and code generation techniques start swirl around you like a blinding swarm of butterflies, take a deep breath and go to a mountain, or whatever you need to do to focus your breath on what business code is. It’s not plumbing, and it IS aspects of your UI. It’s not technology; it’s not anything repetitive between your classes. Encourage conversations about finding all of it, and let the rest of this decade be about isolating real business knowledge in all the places it resides: database structure, service contracts, test definitions, logic rules, workflow, business object code, validation rules, authorization guidelines, user interfaces, etc.