Leaning Into Windows : Visual Basic, C#, Code Generation

Open Source

Kathleen — Wed, 13 Feb 2008 15:31:24 GMT

It’s occurred to me that if you are following this and my DNR TV show a logical reaction would be “OK, so that’s a lot of hot air, where do I get it?” I intend for all of this to be released Open Source, on whichever site s hot when I release it. I hope I’ll start releasing pieces in just a matter of weeks. It will help a lot if it becomes “we” instead of “me”. So, if you’re interested n this stuff and you want to help, let me know and you can get this stuff directly as its rapidly evolving too much to publically post right now.

My current expectation of the order of release:

Metadata interfaces
GenDotNet metadata load
EDMX metadata load
Template infrastructure – what’s needed for the language neutral templates
Simple harness
Activity based harness

If you’re interested, you can contact me directly at Kathleen@mvps.org

Isolating Metadata

Kathleen — Wed, 13 Feb 2008 15:26:09 GMT

In code generation, metadata is the information about your application, generally about your database and definitions to express your data as business objects. If you use Entity Framework, your metadata is the edmx file which is displayed via the designers. If you’re using CodeSmith, the metadata is more subtle. Metadata can also be about the process itself. CodeBreeze in particular has a very rich set and extensible set of information about your application.

Since metadata itself is data – information - we can store it many ways. I’ve used XML for years. CodeSmith has used a couple of mechanisms including XML. Entity Framework uses XML. Metadata can also come directly from a database, although I think this is a remarkably bad idea and one of my code generation principles is not to do that – you need a level of indirection and isolation surrounding your database.

What I haven’t talked about before how valuable it is to have another layer of indirection between your metadata storage structure – your XML schema – and your templates. In my XSLT templates I could provide this only through a common schema – you can morph your XML into my schema so that’s indirection – right?

No, that’s not really indirection. It’s great to be back in .NET classes with real tools for isolation and abstraction. Now I use a set of interfaces for common metadata constructs such as objects, properties and criteria. I can then offer any number of sets of metadata wrappers that implement these interfaces via a factory.

The template programs only against the interfaces. The template could care less whether I am using entity framework, my own metadata tools, or something entirely different. I can write the same template and use it against Entity Framework’s edmx file or any other metadata format. That’s powerful stuff. Especially since you already heard that the template will run against C# or VB. That means in my world the only reason to have more than one set of templates against an architecture like CSLA is that they are pushing the boundaries and actually doing different things.

But if you don’t like this new templating style, you can use classes based on exactly the same interfaces in CodeSmtih (at least) and again free your framework and metadata extraction. You’ll still need VB/C# versions there, but you’re metadata input can use the same interfaces.

The interfaces is expressed in sets of classes that know how to load themselves from a data source. Each set uses a different metadata source – different XML structures or other format.

Isolated metadata removes your templates from caring what the metadata source is – beyond being something that could successfully fill a set of classes that implement the data interfaces. This is a very important step and one we need to work together to get right. What do you think I've left out of the current design?

CurrentTypeInfo and the Context Stack

Kathleen — Tue, 12 Feb 2008 15:46:00 GMT

Creating templates requires a lot of access to the thing you’re currently creating. That’s the current output type, which as I discussed in yesterday I suffix with “Info.” The CurrentTypeInfo is thus what you’re currently outputting.

I neglected to clarify in that post that the Data and the Info classes are in entirely different assemblies. The Data classes could be used with any .NET generation mechanism, including (at least) XML Literal code generation and CodeSmith. The Info objects are relatively specific to my style of code generation.

The CurrentTypeInfo may not be the same throughout the file.

There are a few reasons to combine multiple classes or enums in a file. In some cases, that’s to nest them, and in some cases it’s just to logically combine them in a file. While FxCop has declared them unfashionable, I find nested classes tremendously valuable, especially for organizing Intellisense and keeping well structured classes and naming. If you’re working with nested classes, there is a good chance you’ll need to access not only the current type, but also the encapsulating type. I use a stack for this, and give control of pushing and popping TypeInfo objects from the stack to the templates themselves.

The base item on the stack is the current outer most class in the current file. Once you’ve pulled a class off the stack, such as by popping the base and adding a new base TypeInfo, you can’t access the previous version unless you’ve saved it.

Here’s where you see the flaw I mentioned yesterday – these classes could be in separate namespaces, and I don’t allow for that – yet. I’ll fix it later.

Remember the TypeInfo is the thing you’re outputting. The entity definition it’s built from is ObjectData in my semantics.

The stack is an extremely useful construct for this scenario. You have quick access to information about the class you’re currently outputting. You’ll frequently need this for type information and perhaps calling shared/static methods. You don’t want to recreate its name every time you use it because that would be redundant and hard to maintain. You can also access any of the containing classes, which again is useful in defining types and calling shared/static methods.

While I haven’t done this in CodeSmith, I expect this technique to be viable there. I’m not sure on other platforms, but it’s not specific to XML literal code generation.

Two Parallel Entities - Metadata and Info

Kathleen — Mon, 11 Feb 2008 17:08:00 GMT

One of the confusing things about templating is that you are writing two programs simultaneously and there is no way around it. My brilliant son may write a templating language for a class project, and this is exactly what he wants to address – that and the issue of escaping. You can’t avoid it, he just thinks they should look different – I’m hoping he doesn’t write one in Magyar.

One of the programs you’re writing (or maintaining) is the template program. There is logic in any non-trivial template, regardless of the template style. The other program is the one you’re outputting that will eventually run on your clients computer. In ASP.NET style templates, the output program is the overall template, and the logic is inserted. In XSLT and XML literal generation, the logic of the template program is the overall code and the output template is nested inside.

You are also working with two sets of data – the metadata you’re using as source and what you’re currently outputting. I call the elements in the XML I use as source metadata for .NET templates Object and Property for clarity in the XML. If I called them that in the templates, I’d encounter great confusion between the object definition in metadata and the class I am outputting. That’s particularly painful when it comes to properties.

I solve this by calling suffixing all metadata with the word “Data”. Thus I have ObjectData, PropertyData, CriteriaData, etc. Each of these classes contains the metadata on the corresponding item. I might have an ObjectData for a Customer that had a PropertyData for FirstName. The PropertyData would include things like the maximum length of the field and its caption.

I also need to describe the output – at least the way I’m building templates. I need information about the file I’m creating, the type I’m creating, and in certain locations in the template, the property, function, constructor, etc I’m creating. I identify these by suffixing them with the word “Info”. Thus I have a ClassInfo, PropertyInfo, FunctionInfo, etc. I do not have a CriteriaInfo because I am never outputting a .NET thing called a Criteria. It’s strictly metadata for .NET features.

In the template design I’m describing in this series, the DotNetBase class contains information on the file being output. To be picky, the namespace can actually be associated with part of a file, and I may add this flexibility, but I don’t do that very often in business code, so I have included namespaces at the file level.

Regardless of the template mechanism you use, you need to maintain a clear separation between the template logic and the output logic, and between the input metadata and state about what you’re creating.

Next post: CurrentTypeInfo and the Context Stack

Inheritance and Templates

Kathleen — Mon, 11 Feb 2008 16:25:00 GMT

One of the most valuable things about templates written in .NET code of any style is the ability to use inheritance. This is classic inheritance where local state allows you to push functionality into base class methods that would become cumbersome and have excess dependencies if treated as external utility methods.

Inheritance also allows polymorphism, which is nothing more than a big word for exchangeability – OK, that’s actually a bigger word. Templates are called by something. Life is simpler and harnesses practical if that something is known and predictable, thus it’s logical to have an interface in a common location as a contract to the template interface. If the base class implements this interface and provides functionality via MustInherit/abstract members, there’s further capacity for evolution.

There are three assemblies involved. The template interface is in a common assembly available to all other assemblies. The base classes are in a support assembly which can be reused across many template sets. The templates themselves are in a separate assembly. For now, I have all templates associated with a project in one assembly, but I may change that to isolate the SQL and the .NET templates. In any case, these template assemblies are sets of templates that work together.

Inheritance let’s you push common functionality into the base class. Now that we have extension methods they offer an additional mechanism to remove common functionality from the leaf class. I know Scott Hanselman has said he’ll cut off the pinkies of anyone who uses extension methods on classes you own, and I’ll show you when I get to that in a few days why I disagree (although it’s in the DRN TV show). But that explains why my base classes are focused on state more than functionality. The key state issue is what you’re currently working on. In a template, state is metadata. In a .NET template, the important metadata is the definition for the entity you’re currently creating. In my templates, the ObjectData member of the DotNetBase contains this information.

Stay tuned for the distinction between the input data you’re working on and the thing you’re creating, which explains some of those other members.

.NET Template Organization

Kathleen — Sun, 10 Feb 2008 23:55:00 GMT

So, now that you know where I’m going with this – language neutral templates – I want to step back to the basics. Even if you don’t want to build language neutral templates, there are things to learn along the way about making good XML literal templates, and why this approach might be better than your current code gen mechanism. Note, the examples in this post are NOT language neutral. Not surprisingly, there are specific requirements for language neutral templates and I want to show basic templating with XML literals, then the language neutral templates.

If you look at the XML literal sample I posted a few days ago you’ll notice that the entire contents of the output DataPortalFetch method is created within the template method “MemberDataPortalFetch.” This provide important organization as the most common challenging task in code generation is “I have a problem in my output code right here, how do I find where that is in the template. In a simple template this isn’t too hard, but in more complex templates such as CSLA it can be quite challenging.

Creating templates with an extremely predictable structure is very valuable in creating maintainable templates. To start with each template should have a one to one correspondence with an output class.

Within this class, the mechanism of the predictable structure is one of the primary differences from codes that are code with template segments (XML literal code generation) and templates with code segments (ASP.NET style template). In an ASP.NET template, the template must parallels the output and this organizes the template. In an XML literal template, you provide the organization with Visual Studio then providing navigation. Code is organized my regions, nested classes and member. Each is its own method preceded by Region, NestedClass or Member. This organizes the template. The entry points create a hierarchy working down to the local code (navigation mechanism #1):

   Protected Overrides Function GenerateFile() As String
      Return _
<code>
   <Serializable()> _
Public Class <%= mObjectData.ClassName %>
   Inherits BusinessBase(Of <%= mObjectData.ClassName %>)
   <%= RegionBusinessMethods() %>
   <%= RegionValidationRules() %>
   <%= RegionAuthorizationRules() %>
   <%= RegionFactoryMethods() %>
   <%= RegionDataAccess() %>
   <%= RegionExists() %>
End Class
</code>.Value
   End Function

And

#Region "Business Methods"
Private Function RegionBusinessMethods() As String
Return _
<code>
#Region " Business Methods "

   <%= From prop In mObjectData.Properties Select MemberPropertyFields(prop) %>
   <%= From child In mObjectData.Children Select MemberChildFields(child) %>
   <%= From prop In mObjectData.Properties Select MemberPropertyAccess(prop) %>
   <%= From child In mObjectData.Children Select MemberChildAccess(child) %>
   <%= MemberIsValid() %>
   <%= MemberIsDirty() %>
   <%= MemberGetId() %>

#End Region
</code>.Value

End Function

The carefully named methods also allow you to use alphabetical tools in Visual Studio including the combo box in the upper right of the editor and a class diagram (navigation mechanism #2) which works only because you can predict the name of the member you want from the name of the output member and no other members begin with “Member”, “Region” or “NestedClass”

The third navigation approach is that the template matches the structure of the output as closely as practical. Thus if the output has a region names “Business Methods” the template does as well. The order of items in the template is top to bottom closely paralleling the order of the output. This allows you to cruise down in the file.

Template organization is the first step to great templates.

The Punch Line

Kathleen — Sun, 10 Feb 2008 23:21:00 GMT

I mentioned a few days ago that there was a punch line for the XML Literal Code Generation. I planned to unveil this slowly, but it just sprung out of the box when Carl posted episode #102 (which I thought was due for next Friday.

And I’m afraid that I need to add that I was sick during the taping and my brain running at half capacity. If anything isn’t clear, please let me know.

No matter.

You get to hear sooner.

I’ll be unveiling the details in about the same time frame, so you’ll have the big picture by next week.

So, the drumroll please…

I can create excellent readable templates that output code in Visual Basic or C# - no CodeDOM involved.

I’ll show you how to do that. Watch for more here.

XML Literal Code Genaration - Code again again

Kathleen — Sun, 10 Feb 2008 16:53:00 GMT

Crap.

For now, I'm just removing the coloring. Paste this into VS for coloring. It's much prettier:

   Private Function MemberDataPortalFetch() As String
      ' TODO: Add special handing for timestamp
      Return _
<code>
   Private Overloads Sub DataPortal_Fetch(ByVal criteria As Criteria)
      Using cn As New SqlConnection(<%= mObjectData.ConnectionStringName %>)
         cn.Open()
         Using cm As SqlCommand = cn.CreateCommand
            cm.CommandType = CommandType.StoredProcedure
            cm.CommandText = "get<%= mObjectData.ClassName %>"
            <%= From prop In mObjectData.PrimaryKeys Select _
               <code>
            cm.Parameters.AddWithValue("@<%= prop.Name %>", criteria.<%= prop.Name %>)
               </code> %>

            Using dr As New SafeDataReader(cm.ExecuteReader)
               dr.Read()
               With dr
                  <%= From prop In mObjectData.Properties Select _
                     <code>
                  m<%= prop.Name %> = <%= GetReadMethod(prop) %>("<%= prop.Name %>")
               </code>.Value %>

                  ' load child objects
                  .NextResult()
                  <%= From child In mObjectData.Children Select _
                     <code>
                  m<%= child.Name %> = <%= child.Name %> .Get<%= child.Name %> (dr)
                     </code>.Value %>
               End With
            End Using
         End Using
      End Using
   End Sub
</code>.Value
   End Function

XML Literal Code Generation - Code again

Kathleen — Sun, 10 Feb 2008 16:32:00 GMT

OK, that code listing got friend on load. Let me try again.

Private Function MemberDataPortalFetch() As String Return _<code> Private Overloads Sub DataPortal_Fetch(ByVal criteria As Criteria) Using cn As New SqlConnection(<%= mObjectData.ConnectionStringName %>) cn.Open() Using cm As SqlCommand = cn.CreateCommand cm.CommandType = CommandType.StoredProcedure cm.CommandText = "get<%= mObjectData.ClassName %>" <%= From prop In mObjectData.PrimaryKeys Select _ <code> cm.Parameters.AddWithValue("@<%= prop.Name %>", criteria.<%= prop.Name %>) </code> %> Using dr As New SafeDataReader(cm.ExecuteReader) dr.Read() With dr <%= From prop In mObjectData.Properties Select _ <code> m<%= prop.Name %> = <%= GetReadMethod(prop) %>("<%= prop.Name %>") </code>.Value %> ' load child objects .NextResult() <%= From child In mObjectData.Children Select _ <code> m<%= child.Name %> = <%= child.Name %>.Get<%= child.Name %> (dr) </code>.Value %> End With End Using End Using End Using End Sub</code>.Value End Function

XML Literal Code Generation

Kathleen — Fri, 08 Feb 2008 17:20:00 GMT

You must use VB for XML Literal Code Generation. Sorry, that’s life. I don’t have a crystal ball on this, but for now even if you’re a C# programmer you’ve got to suck it up and use VB. The code you output can be VB or C# however, but the template itself has to be written in VB. If you’re allergic to VB, get shots.

So, what does a microcosmic view of a XML literal template look like? I’ve translated a section of a CSLA template. CSLA is Rocky Lhotka’s architecture and this is a version 2.1 because I think it clarifies the template process:

Private Function MemberDataPortalFetch() As String Return _<code> Private Overloads Sub DataPortal_Fetch(ByVal criteria As Criteria) Using cn As New SqlConnection(<%= mObjectData.ConnectionStringName %>) cn.Open() Using cm As SqlCommand = cn.CreateCommand cm.CommandType = CommandType.StoredProcedure cm.CommandText = "get<%= mObjectData.ClassName %>" <%= From prop In mObjectData.PrimaryKeys Select _ <code> cm.Parameters.AddWithValue("@<%= prop.Name %>", criteria.<%= prop.Name %>) </code> %> Using dr As New SafeDataReader(cm.ExecuteReader) dr.Read() With dr <%= From prop In mObjectData.Properties Select _ <code> m<%= prop.Name %> = <%= GetReadMethod(prop) %>("<%= prop.Name %>") </code>.Value %> ' load child objects .NextResult() <%= From child In mObjectData.Children Select _ <code> m<%= child.Name %> = <%= child.Name %> .Get<%= child.Name %> (dr) </code>.Value %> End With End Using End Using End Using End Sub</code>.Value

End Function

This is .NET code, which you can see because the template output is inside a .NET function named MemberDataPortalFetch which returns a string constructed with XML literals. By offsetting with XML literals, the code of the function is easily seen – it’s in gray. All of the gray code will be in the output, none of the yellow, blue or black code will be in the output, although it will certainly affect the output.

Where replacements are necessary, XML literals have “embedded expressions” which I have the habit of calling “expression holes.” I’ll talk in another post about what mObjectDatais, but an embedded expression can contain whatever expression you need. An expression is something that returns a value within a single line. So, you can call methods and properties, but you can’t call a subroutine.

Embedded expressions can contain LINQ as shown in the third embedded expression. Here we want to add parameters for all the primary keys. Within the LINQ Select clause, I can use any string. Including the nested code block creates a more readable template and consistency, among other things.

A quick glance may leave you thinking that this is just a variation of the ASP.NET style template syntax used by CodeSmith, CodeBreeze, MyGeneration and others. However, it’s quite different. To begin with its organized into members of a template class. Then, the logic is expressed in expressions, not direct template logic. Finally, you can nest back and forth between output text and code to any depth you require. The code blocks within the LINQ statements above contain embedded expressions. These embedded expressions could be further LINQ statements with further code blocks, etc.

I think there are right and wrong ways to build templates with XML literals. I’ll spend the next couple of posts talking about template organization and metadata handling.

Let’s Talk Templates

Kathleen — Thu, 07 Feb 2008 22:07:00 GMT

Code generation templates have several competing pressures – they need to be easy to use and they need to be powerful. They need to be flexible but encourage best practices. If you ever say “my templates got me started then I had to customize them” you’re templating can be better. I’m not aware of any situation where you can’t isolate the handcrafted code from the generated code sufficiently to keep code generation active for the entire life of your application.

When I say full life cycle, I mean from initial prototypes to finally pulling the switch in a decade. Yep a decade. Code generation should aim for evolution and extremely long project life cycles. The metadata of the project can far outlive the technology, even outlive the database. That requires a deep commitment to code generation, and if you don’t have that commitment, the flip side is that code generation can pay back today. At that moment when it feels easier to pull a file out of code generation, or to repeat a similar block of code in two places instead of pushing it into your code gen templates, that’s where the commitment comes in. With or without it, you get payback; it’s just greater the deeper your commitment is to the metadata and templates.

My current client pushes me because he is even more committed to the decade long lifecycle based on code gen than I am. And he’s right. Every time he’s pushed to get something into templates or do code gen better, it’s taken less work and gotten us further in the short term than I expected.

What if you could combine the simplicity of CodeSmith with the power of XSLT?

And what if you could do it primarily in a language and architecture you already know, or should be learning anyway?

Interested?

This is the first post in a blog series to present this templating style. I also covered in a.NET Rocks show I recorded yesterday. I’m very excited about it, but I want to introduce the basics before I tell you the real punch line. This template style is the best way to generate code today and it’s based entirely on tools you’ve got on your desktop right now.

Look for more here and in the .NET Rocks TV episode scheduled to appear Feb. 15^th. The beautiful thing about the techniques I’ll show is that you can smoothly transition from simple to complex templates dialing in the exact complexity appropriate for your organization and application. And, just wait for the punch line.