Using T4 to Create an AppSettings Wrapper, Part 7

In this final article in the series we’re finishing up the deployment projects started last time.  When we’re complete we’ll have a set up projects that can be used to build and deploy T4 templates where needed.  The projects will be able to support any number of templates so maintenance will be simple even as more templates are added.  In the previous article we moved the core template functionality into a library that we can expand upon.  We also set up an item template project to store the root T4 template that someone would use.  In this article we’re going to create the project to store the nested templates (we’ll discuss why later) along with the Visual Studio Extension (vsix) file that will be used to install everything.

Corrections

There are a couple of corrections that need to be made from the source in the last article to prepare for the setup project.

In the AppSettingsTemplate.tt file the assembly reference for the shared assembly needs to include the .dll otherwise the T4 host will assume it is coming from the GAC.  An assembly reference for EnvDTE is also missing so it needs to be added.

<#@ assembly name="TemplateLib.dll" #>
<#@ assembly name="EnvDTE" #>

In the nested template there were still some hard-coded references to the original class name rather than using the ClassName property in the template.  Do the replacement so the template behaves properly based upon the name that is ultimately used.

Nested Template Project

For the nested templates we need to create a new VS Project Template project.  When a developer uses one of the item templates they will only add the core template to their project.  The nested template, and support assembly, will need to be stored in a shared location that the T4 host can find.  The easiest way to do this is to use a VS Project Template project.  All the shared templates will be stored in this project.  We could technically even store the base template class code in this project but I find it easier to keep them separate. 

Add a new project to the solution (Visual C#\Extensibility\C# Project Template) called TextTemplates (the name is not really relevant).  Once the project has been added remove all the files from the project as they will not be needed.  Add a reference to the shared assembly project that was created earlier.

This project mirrors the item template project so set up the same folder structure for each template as needed.  The nested template file will be moved to this project.  Every time a new nested template is added the following steps need to be followed.

  1. Create a subfolder in the project
  2. Add the nested .tt file to the subfolder
  3. For each .tt file set the following properties for the item
    • Build Action = Content
    • Copy to Output = Copy Always
    • Custom Tool = (blank)
    • Include in VSIX = True

There is only one issue with the approach we’re taking – T4 does not know where to find our nested templates and custom assembly.  Therefore we need to update the search path that T4 uses to include the installation path of the package.  The simplest way to do that is to add it to the package definition (.pkgdef) file.  When a package is installed the package definition is processed to allow the package to do any customization it needs.  We can use this feature to update the T4 include path.

Add a new text file to the project.  It’s name must match the name of the project and it should have an extension of .pkgdef (i.e. TextTemplates.pkgdef).  Change the following properties for the item.

  • Build Action = Content
  • Copy to Output = Copy Always
  • Include in VSIX = True

When T4 runs across an assembly or include directive that it cannot find then it uses the registry to search for additional paths.  Each time we add a new template we need to add the path to the template to the list using the .pkgdef file.  Note that the subfolder we use in the project will correspond to the subfolder that the template is installed to so we need to include the full path.  Here’s the code we’ll add to the package definition file.

[$RootKey$\TextTemplating\IncludeFolders\.tt]

"IncludeMyAppSettings"="$PackageFolder$\TextTemplates\AppSettings"

This will add a new key to the registry with the given name and value.  The installer will replace $PackageFolder$ with the installation path for the package.  The project name follows that and the last name is the folder that was used when adding the template to the project.  It is critical that the key name start with “Include” otherwise the T4 host will ignore it.  It must also be unique.  Each new subfolder of nested templates will need a corresponding name-value pair in the definition file.

Some discussion of this can be found in MSDN.  But credit for pointing me in the correct direction when I was trying to figure this out has to go here.

VSIX Project

 

We’re on the home stretch.  We have set up the project for the shared assembly code, defined the item templates that will be available to the developers and got the support templates hooked up to T4.  The final step is to create a VSIX file to install everything.  By far this is the most frustrating part because VSIX is very picky about how things have to work and it can be a bear to work with.

Create a new VSIX Project (Visual C#\Extensibility\VSIX Project) called MyTemplateSetup.  The manifest editor will open up.  The manifest controls several important aspects of the setup including the information the user sees, the files to be installed and the version of the setup.

  1. Set Product Name to the name you want to the extension to appear as in the gallery.
  2. The Product ID must be unique and should already be set.
  3. Set Author to an appropriate value.
  4. The Version should default to 1.0.
  5. Provide a description of the extension.
  6. Since this is a template installer the Tags should probably be set to Templates.
  7. Optionally set the other attributes such as licensing, icon and release notes.

Switch to the Install Targets tab.  This tab indicates what version(s) and edition(s) of Visual Studio the extension supports.  The default is Visual Studio 2012 Pro or higher which should be fine.  Microsoft licensing does not allow third-party extensions to the Express products.  If a newer version of Visual Studio comes out then the version number can be updated.

Switch to the Assets tab.  This is where we identify the files to be installed.  This is also where things can get difficult if the projects were not created using the right project type.

Add a new asset for the shared assembly. 

  1. Type is Microsoft.VisualStudio.Assembly
  2. Source is a project in the solution
  3. Project will be the shared assembly project
  4. Click OK

Add a new asset for the item templates.

  1. Type is Microsoft.VisualStudio.ItemTemplate
  2. Source is a project in the solution
  3. Project will be the item template project
  4. Click OK

Notice that the path that is displayed includes an output group.  This property is set inside the item template project.  If the referenced project actually isn’t an item template then the build will fail.

Add a new asset for the nested templates.

  1. Type is Microsoft.VisualStudio.VsPackage
  2. Source is a project in the solution
  3. Project will be the text template project 
  4. Click OK

An interesting (and sometimes problematic) thing that happens is that project assets are added as references to the project.  For the nested templates this is problematic because by default the setup project expects an assembly to be generated.  For the nested templates there is no assembly so open the properties for the nested template reference and set Reference Output Assembly to False.  If this is not done then a compilation error will occur.

Switch to the Dependencies tab.  This is where any dependencies can be defined.  The .NET framework is already included but we depend upon T4 Toolbox so that needs to be added as well.  Assuming it is already installed on the machine you can do the following. to add the dependency.

  1. Source is Installed Extension
  2. Name is T4 Toolbox
  3. Version range will be set to the current version but you can adjust this as needed
  4. Click OK

One issue with dependencies is that if the user does not have the dependency installed they may get a generic error message before they get information about the missing dependency.  Hence it is useful to put dependencies in the description of the extension.

Almost done.  The last thing we need to do is add another package definition file (.pkgdef) matching the name of the shared assembly project.  As was done earlier, configure the item properties.

  • Build Action = Content
  • Copy to Output = Copy Always
  • Include in VSIX = True

The package definition file used earlier had an entry for each template.  This file will contain an entry for the shared assembly.

[$RootKey$\TextTemplating\IncludeFolders\.tt]

"IncludeMyTemplateAssemblies"="$PackageFolder$"

Compilation

Time to compile the setup but before you do I recommend that you change the VSIX properties in the setup project to not deploy to the experimental instance of VS.  The experimental instance of VS is designed to allow you to test your packages without impacting your main development environment.  Unfortunately in my experience it does not work well when you have additional packages or extensions installed.  You end up sitting through lots of error dialogs. 

If you do not use the experimental instance of VS then you won’t be able to easily debug your template.  However there is a simple approach that I find useful.  In most cases you will develop your template in a stand alone project so you can tweak the template.  Once it is added to the extension though you can still change it by finding the directory where the extension was installed.  By default it will be a randomly generated directory under <VSDir>\Common7\IDE\Extensions for shared extensions or <profiledir>\AppData\Local\Microsoft\VisualStudio\11.0\Extensions for user extensions.  Once you find the directory you can edit and/or replace the files until you’ve resolved any issues you’re having.  You can then update the project and redeploy.

To test the setup simply run the .vsix file.  Start VS, create or open a project and add the new item template to the project.  Confirm the generated code is correct.  In general if you are adding or removing extensions you should ensure that all instances of VS are closed first.  VS doesn’t full install/remove extensions until it is restarted and having multiple instances running can cause problems with that.

Versioning

Versioning of the extension is incredibly important.  When VS is looking for an updated extension it uses the version as defined in the manifest.  Ensure that whenever you build a new version of the setup that you increment the version number.  You should also consider keeping the major/minor version number of the shared assembly in sync with the manifest file.

Do not change the product ID.  If you do then this becomes a brand new extension that can be installed side-by-side with the old one.  In most cases this can cause  conflicts that you would want to avoid.

Troubleshooting

Troubleshooting T4 templates is an art more than a science but here’s a few thoughts based upon my experience.

If you get a compilation error while compiling the setup project saying it cannot find the assembly for the nested templates project then you forgot to change the reference’s properties.  Refer to the earlier section on fixing that.

If the root template cannot find the nested template then the registry was not updated properly.  This can occur if the extension was installed but VS was not restarted, multiple instances of VS were running or if something went wrong.  Uninstall the extension, restart VS to clean everything up, shut down VS and reinstall the extension.

If the shared assembly file cannot be found then the above comments about the root template also apply.  Additionally ensure that the assembly reference in the nested template includes the .dll or the assembly is in the GAC.

If the generated file contains an error token then use Error List to determine the actual error(s) that occurred.  In most cases it is a compilation issue with the nested template.  Load up the nested template in VS and edit it until it compiles.  Then rebuild the deployment package.

If the item templates do not show up then ensure that they were properly added to the setup project and that they appear in the extension directory.  The path to the extension directory is contained in the install log that is accessible at the end of the extension installation.

When adding a new template get it working in a standalone console application and then move it into the templates solution.  This will make it easier to make changes and you can still rely on the shared assembly and break up the template into the item and nested components.

Enhancements

There are quite a few enhancements that could be made with all this code.  One area that I personally recommend is using data annotations for validating configuration properties.  Apply annotations to the configuration properties where appropriate.  Alternatively implement IValidatableObject in the template class.  Then modify the Validate method in the base class to validate the annotations as part of the core validation.  This eliminates the need for per-template validation of things like required parameters.

Another area of improvement is breaking out all the non-template functionality.  I personally like extension methods so you could move all the non-template functionality into extension methods. 

Yet another area is defining some common template generation methods that derived types can override.  For example most T4-generated code should be marked as non user code so they won’t step into it.  Wrapping this up in a method allows derived types to call it where appropriate and, optionally, add their own attributes.

For deployment you can provide the developers with the VSIX package.  If you’ve read my earlier article on hosting your own private VS extension gallery then you could deploy the VSIX to that as well.

Attachment: MyTemplates.zip
Posted Sun, Apr 28 2013 by Michael Taylor | no comments
Filed under:
Using T4 to Create an AppSettings Wrapper

This is the table of contents for the series on using T4 to create an AppSettings wrapper for use in any code.

  1. Creating a static T4 Template for AppSettings
  2. Creating a dynamic T4 Template for AppSettings
  3. Reading Settings from the Configuration File
  4. Separating Customizable Parts from Static Parts
  5. Customizing the Template
  6. Creating a Deployment Package (Part 1)
  7. Creating a Deployment Package (Part 2)
Posted Sun, Apr 21 2013 by Michael Taylor | 1 comment(s)
Filed under:
Using T4 to Create an AppSettings Wrapper, Part 6

In this (almost) final post in the series on creating an AppSettings wrapper in T4 we’re going to package all the work we’ve done into an easily deployable and reusable package that we can extend to include other templates.  We’ll also make sure that adding the template to a project is as easy as Add New Item.

Here’s the list of things we’ll accomplish

  • Create a custom base class and move all the shared template functionality to it.
  • Create a VS extension (VSIX) to install the parts of our template.
  • Install the shared template components into a centralized directory so projects only need to include the core template.

Preparation

There will be 4 projects by the time we’re done so now is a good time to create a new solution for all the template work.  Create a new blank solution in Visual Studio (Other Project Types\Visual Studio Solutions).  This solution will be able to host any number of T4 templates so you should use a generic name (i.e. MyTemplates).

We’ll be building a VSIX so the Visual Studio 2012 SDK needs to be installed.  If the T4 Toolbox isn’t installed then that will be needed as well.

Shared Template Assembly

Let’s start by moving all the common functionality out of the nested template and into a reusable base class.  The base class will contain any logic that can be shared across T4 templates.  Create a new Class Library (i.e. TemplateLib) to store the base class and any additional functionality that will be needed by all templates.  Since this will be hosted by VS it needs to target the .NET 4.5 framework.  Delete any created class file.

We have been using a lot of assemblies so we need to add references to all of them.  Note that when referencing VS assemblies there may be several.  Prefer the versions in the Program Files directories over the VS directory to make it easier to move solutions.

  • Assemblies\Extensions
    • EnvDTE (version 8.0.0.0)
    • Microsoft.VisualStudio.OLE.Interop
    • Microsoft.VisualStudio.Shell.11.0
    • Microsoft.VisualStudio.Shell.Design
    • Microsoft.VisualStudio.Shell.Interop.11.0
    • Microsoft.VisualStudio.TextTemplating.11.0
    • Microsoft.VisualStudio.TextTemplating.Interfaces.11.0
    • VSLangProj
  • Assemblies\Framework
    • System.Configuration
  • T4 Toolbox installation directory
    • T4Toolbox
    • T4Toolbox.VisualStudio

For the T4 Toolbox binaries you’ll unfortunately have to reference them from the installation directory where VS puts them.  By default this will be a folder under your AppData\Local\Microsoft\VisualStudio\11.0\Extensions directory.  If you like you could also copy the files into a shared assemblies folder under the solution to make it easier to use.

When adding EnvDTE and VSLangProj be sure to open the reference properties and set Embed Interop Types to False.  Otherwise you’ll get several compilation warnings.

With all the references in place create a new class that will represent the base template class (i.e. MyTemplate).  It should be public, abstract and derive from CSharpTemplate in the T4 Toolbox. 

/// <summary>Base class for T4 templates.</summary>
public abstract class MyTemplate : CSharpTemplate
{
}

Move all the members not specifically related to the app settings template into the base class.  Make them protected so they are not exposed outside hierarchy.  Here’s the members I see that should be moved.

  • ActiveProject (and field)
  • DteInstance (and field)
  • FindConfigProjectItem
  • FindItem
  • FindProject
  • GetFriendlyName
  • GetProjectItemFileName

With the members moved the nested template can be cleaned up to remove unused namespaces and assembly references.  All it contains now is the functionality specifically needed for dealing with the settings.

The nested template will need a reference to our assembly and an import for the namespace.  Additional the nested template needs to derive from the new class.


<#@ assembly name="TemplateLib" #> <#@ include file="T4Toolbox.tt" #> …
<#@ Import namespace="TemplateLib" #> <#+ public class AppSettingsTemplate : MyTemplate

At this point the template will not work correctly because T4 cannot find the custom assembly.  To temporarily confirm all the changes work you can copy the assembly to the directory containing the templates.

Item Template Project

In order to use the template developers will need to be able to add the template from Add New Item.  To simplify deployment we can create a new template project that installs the items into the appropriate location.  This project can install any number of item templates so as more T4 templates are created they can be added here.

Create a new Item Template project (Visual C#\Extensibility\C# Item Template) called ItemTemplates.  Since we might have several different templates in this project I like to create a folder for each template (i.e. AppSettings).  Each new T4 template can be placed into its own subfolder so there is no worries about collisions.  Every time a new item template is added the following steps need to be followed.

  1. Create a subfolder in the project
  2. Add the .tt file to the subfolder along with the .vstemplate and optional .ico files
  3. For each .tt file set the following properties for the item
    • Build Action = Content
    • Custom Tool = (blank)
    • Include in VSIX = True
  4. For the .vstemplate file set the following properties
    • Build Action = VSTemplate
    • Include in VSIX = False
    • Category = (see below)
  5. In the VSTemplate file
    • Set RequiredFrameworkVersion as appropriate
    • Set a unique TemplateID
    • Set the name and description
    • Set DefaultName accordingly
    • Reference the icon included in the project
    • Add any referenced assemblies
    • Change the existing ProjectItem entry to reference the .tt file instead
    • For each ProjectItem set the ItemType to the appropriate Build Action (for .tt files it should be set to None)

Move the .vstemplate and .ico files that were generated into the template folder.  The .cs file can be deleted.  Rename the copied files to match the template name.  Finally copy the root T4 template from previous articles into the folder.  Then follow the steps above to complete the process.

The Category property for the .vstemplate allows you to group your templates into a folder in Add New Items.  It should be used to keep your custom templates separate from VS or others.

Time Out

This post is getting long so it’s time to take a break.  In the next (and final) post we’ll create the other 2 projects to finish up the deployment of the T4 template.

Attachment: MyTemplates.zip
Posted Sun, Apr 21 2013 by Michael Taylor | no comments
Filed under:
Using T4 to Create an AppSettings Wrapper, Part 5

In the last article we broke out the template into 2 separate template files: main and nested .  A developer adds the main template to their project but it includes the nested template where the actual work is done.  In this article we’re going to update the included template to allow a developer to alter the generated code without having to touch the nested template.

In the original article we listed 6 requirements for the template.  At this point we have met half of them.  By adding customization we will meet the final three.  They are:

  • Exclude some settings
  • Change the type of a setting
  • Use the configuration file of a different project

Customizing a Template

Customizing a template is actually not that hard now that it’s broken up.  The key is in understanding what we can and cannot change when we release updates.  The main template can be updated but in order to get the updates a developer would need to remove the main template from their project and add it back.  The nested template can be updated as needed.  Provided the developer reruns the template they will get any changes.

To customize a template we add public members (properties or methods) to the template class in the nested template.  As with a normal class we can then use the properties/methods when it comes time to execute the template.  We can even add new customizations in future updates without breaking the developer’s existing customizations.

In the main template a developer can call the public members after the template class instance is created but before it renders.  Since they will not have access to Intellisense it makes sense to provide some commented code demonstrating the available customizations.  Additionally we can provide some basic customizations if desired.  Since the main template is never updated once it is added to the project (unless the developer does it explicitly) the customizations are preserved even if the nested template is updated.  Now let’s focus on adding each of the customizations we listed earlier.

Excluding Settings

With certain projects we often find extra settings that we do not care about included in the appSettings section.  ASP.NET is a good example because it stores several options there.  We probably do not want to generate code for these.  For this customization we will allow the developer to exclude a setting based upon its name. 

In the nested template add a public method called ExcludeSetting that accepts the name of a setting.  Some frameworks have a lot of settings so it would be painful to exclude each one.  Therefore we’ll make the exclusion functionality a little smarter by allowing an asterisk on the end (i.e. webpages*).  The asterisk will be a wildcard for anything that follows the name when doing matching.  To simplify the code define 2 private lists to store the excluded settings (one for regular and one for wildcard exclusions).  Also define a private method called IsExcluded that takes a setting name and returns whether it should be excluded or not based upon the lists.

public AppSettingsTemplate ExcludeSetting ( string settingName )
{   
   if (settingName.Contains("*"))
   {
      var token = settingName.Substring(0, settingName.IndexOf('*'));
      m_exclusionMasks.Add(token);
   } else
      m_exclusions.Add(settingName);

   return this;
}

private List<string> m_exclusions = new List<string>();
private List<string> m_exclusionMasks = new List<string>();

private bool IsExcluded ( string settingName )
{
    return m_exclusions.Where(x => String.Compare(x, settingName, true) == 0).Any() ||
            m_exclusionMasks.Where(x => settingName.StartsWith(x, StringComparison.OrdinalIgnoreCase)).Any();
}

In the code where the settings are enumerated call the IsExcluded method to determine if the setting should be generated or not.

<#+
   foreach (KeyValueConfigurationElement setting in GetAppSettings()) { 
      if (IsExcluded(setting.Key))
continue;

   var type = GetSettingType(setting.Value); #>

Now we can add some default entries for some commonly used frameworks.  Notice that we can chain calls together because the public method returns the template instance.

<# var template = new AppSettingsTemplate();
    
// Exclude any settings that are not needed, use * for a wildcard
template.ExcludeSetting("aspnet:*")
           .ExcludeSetting("webpages:*);
  
template.Render(); 
#>

Changing a Setting’s Type

By default the template will determine the type of a setting based upon its value.  But sometimes the value doesn’t properly indicate the type (i.e. 123 for a phone extension) or a specific type is needed (i.e. a long even if the value is 123).  For this we will allow the developer to specify the type to use for a setting rather than relying on the default behavior.

Add a public method to the nested template called OverrideSettingType.  It should accept a setting name and a type.  Store the setting name in a private dictionary for later.

public AppSettingsTemplate OverrideSettingType ( string settingName, Type settingType )

{
   m_settingsTypes[settingName] = settingType;

   return this;
}

private Dictionary<string, Type> m_settingsTypes = new Dictionary<string, Type>(StringComparer.OrdinalIgnoreCase);

Modify the GetSettingType method to check the override dictionary before trying to determine the type of the setting based upon its value.  Note that the setting name needs to be added to the parameter list so this check can be done.  Update the calling code accordingly.

private Type GetSettingType ( string name, string value )

{    
   Type explicitType;
  
if (m_settingsTypes.TryGetValue(name, out explicitType) && explicitType != null)
      
return explicitType;

   //Use heuristics
}

Here is how it might look in the main template.

//Override a setting's type

template.OverrideSettingType("IntValue", typeof(long));

Using a Different Project’s Configuration

Normally you would agree that a project should only rely on its own settings but there are a few cases where this doesn’t make sense.  One example is a WCF service host.  The host generally consists of only the configuration file for the service and the .svc file.  The actual implementation is stored in a separate project so the service can be re-hosted with ease.  In this case we would want to add the template to the implementation project but have it rely on the host project’s configuration file.  For this customization we will allow the developer to specify a project other than the active project from which to read the configuration file.

Add a new public property to the nested template called ConfigurationProject.  If it is set then it is the name of the project where the config file resides that will be read.

public string ConfigurationProject { get; set; }

The hard part is changing the nested template to use a different project.  Fortunately we already have almost all the code.  In an earlier article we wrote a function to get the settings given a ProjectItem.  Up until now we’ve been using ActiveProject.  To use a different project we need only find the project in the solution and then pass it to FindConfigProjectItem instead.

private KeyValueConfigurationCollection GetAppSettings ()
{
    var project = ActiveProject;

    //If a custom configuration file is specified then find the project
    if (!String.IsNullOrEmpty(ConfigurationProject))
        project = FindProject(ConfigurationProject);

    //Get the config file
    var configItem = FindConfigProjectItem(project);

    return GetAppSettings(configItem);
}

public EnvDTE.Project FindProject ( string projectName )
{
    if (String.IsNullOrEmpty(projectName))
        return null;

    foreach (EnvDTE.Project project in DteInstance.Solution.Projects)
    {
        if (String.Compare(project.Name, projectName, true) == 0)
            return project;
    };

    return null;
}

Here is how a developer would specify it in the main template.

//Change the project to use
template.ConfigurationProject = "";

Validation

Up until now we haven’t really needed any validation.  But as we allow developers to customize the template it makes sense to ensure the template can be generated.  If something goes wrong the template host generally spews out useless messages.  Before we finish up lets add some basic validation to the template.  With T4 you can generate either warnings or errors.  With the T4 Toolbox these can be generated using the Warning and Error methods.

For this template the following validation seems appropriate.

  1. If a custom configuration project is set then the project must exist otherwise it is an error.
  2. If no config file can be found then this would be a warning.
  3. If the config file has no app settings then this would be a warning.

I personally like to do validation before the template runs so we’ll add a Validate method that wraps the validation logic.  T4 Toolbox already defines a base validation method so we can override it.  As an optimization we will store off the found settings so we do not have to search for them again when we render.

Add a Validate method to the nested template that handles all the validation rules.

private KeyValueConfigurationCollection AppSettingsInfo { get; set; }

protected override void Validate ()
{
    base.Validate();
KeyValueConfigurationCollection info = null; var project = String.IsNullOrEmpty(ConfigurationProject) ? ActiveProject : FindProject(ConfigurationProject); if (project != null) { var configItem = FindConfigProjectItem(project); if (configItem != null) { info = GetAppSettings(configItem); if (info == null || info.Count == 0) Warning("No appSetting entries found."); } else Warning("Unable to locate configuration file."); } else Error("Unable to locate configuration project '{0}'.", ConfigurationProject); AppSettingsInfo = info ?? new KeyValueConfigurationCollection(); }

The original GetAppSettings method can go away now and inside the template text we can replace the call to it with AppSettingsInfo.  When the template runs the validation method will execute and we’ll get pretty errors and warnings if appropriate.

Next Steps

At this point we have a template that a developer can add to their project and customize to meet their needs.  The template relies on a nested template to do the actual heavy lifting of generating the final file.  There are some downsides to this approach including the fact that the project needs to include multiple files even though only 1 is added to the project.  Another is the fact that even though we have nested the core functionality we would still need to replicate the code if we wanted to create a different template.  Even worse is the fact that if we update the core template (add enhancements or fix bugs) then every project that relies on the template would need to get an updated version.  In the next article we’ll package the template up into a reusable component that is easy to deploy and update and will have minimal impact on the end developer.

Posted Sun, Apr 7 2013 by Michael Taylor | no comments
Filed under:
NuGet with Active Directory Support

In a previous article I discussed how to host a private NuGet repository.  If you aren't familiar with NuGet then please refer to that article.  If you're hosting a private gallery then chances are you're on a network (probably an Active Directory one).  One downside to hosting a private NuGet gallery is that it is tied to Forms authentication.  For a public repository this makes sense but for a private one it would be better to have NuGet use Windows authentication.  This article will discuss the changes that have to be made to the NuGet source to support Windows authentication.  The changes are scattered but minor. 

NuGet Authentication

Official NuGet uses Forms authentication.  When a user is browsing the site or downloading packages then they have not logged in yet.  When a user attempts to do something like manage a package or upload one then NuGet prompts for login.  It does this using the standard Authorize attribute on the appropriate controller actions.  If the user does not have an account yet then they are prompted to register.  The registration process associates the user name, password and email address.  If configured then the user must confirm their email address before their account is recognized.  If the user owns packages then emails sent from NuGet will go to their registered email address.

NuGet defines a single role - Admin.  A user is either an admin or they are not.  There is no UI for assigning roles.  Instead a database call needs to occur.  A normal user can manage their own packages and send emails to other owners.  An admin can manage any packages and assign ownership.

NuGet View Modes

Out of the box it would seem that adding Windows authentication to NuGet would be a simple matter of changing the authentication mode but unfortunately it isn't that easy.  NuGet can be accessed several different ways.  Some support Windows auth and some don't.

  • Website viewing - In this mode a user should be able to view the packages and even download them without having to be authenticated.  In terms of Windows auth any user should probably have permissions.
  • Website uploading - Only users who are authenticated should be able to upload packages. 
  • Visual Studio - VS will connect to NuGet to get the list of packages and to download them using the web API.  This shouldn't require any special privileges.  More importantly VS will not respond to a challenge/response from the server so Windows auth cannot be used.

Because of the different ways of accessing NuGet it seems that the best approach will be to allow anybody to access the site (assuming they have the right Windows group membership).  The Admin role can remain and be used as needed.  A new role, Authors, is needed to control who can upload packages.  This role is only used when accessing the website for uploads.

Before going any further you need to get the NuGet source and ensure it compiles correctly.  I walked through that process in the previous article.  We need to make some changes to the NuGet files and we need to add some additional files.  I'll walk through the process step by step.

Preparing NuGet

Before continuing further be sure that Windows authentication has been installed for IIS.  Also ensure that the app pool that is hosting NuGet has read/write permissions to the App_Data directory.  All other IIS changes will be handled by the web.config file.  If you are using a version of IIS prior to Server 2008 then you might need to make some of these changes to IIS directly.

We will be adding new files to NuGet and making changes to existing ones so we want to try to keep them separated.  In the NuGet website add a new folder to store the new files (i.e. Custom).  Whenever we add new files they will go here.

Replacing User Service

NuGet uses IUserService to create and manage users.  It is clear that IUserService was written based upon a Forms auth approach because it doesn't really encapsulate anything other than Forms auth.  For the most part we just need to replace a few methods that don't make sense in an AD environment.  To do that we'll create a new type called ADUserService and have NuGet use it instead.  Because we aren't providing a full implementation we'll derive from the existing type.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web.Security;

namespace NuGetGallery
{
    public class ADUserService : UserService
    {
        public ADUserService ( GallerySetting settings,
                               ICryptographyService cryptoService,
                               IEntityRepository<User> userRepository )
            : base(settings, cryptoService, userRepository)
        {
        }
    }    
}

When a user registers with NuGet they must enter their user name, password and email.  Once the user has entered the information it is stored in the database by calling Create.  The user is associated with packages through the user table in the database.  Irrevelant of authentication we need to ensure that the entry is created.  For AD we don't need to do anything different except for the confirmation email and password.  By default a confirmation email is sent that the user must respond to.  For AD we don't need to send a confirmation email.  It is a setting in NuGet to disable confirmation emails but we'll go ahead and ensure that the email is confirmed when the user is created.

public override User Create ( string username, string password, string emailAddress )
{
    var user = base.Create(username, password, emailAddress);

    //Confirm the email address
    ConfirmEmailAddress(user, user.EmailConfirmationToken);

    return user;
}

Whenever NuGet needs user information it will find the user by calling one of several find methods.  A couple of these require a password.  Since we're using AD we don't need the password (in fact we won't have it).  When we create the user we'll use a dummy password.  When searching for a user we will use only the user's name and ignore any password.

public override User FindByUsernameAndPassword ( string username, string password )
{
    //Just search by user name
    return base.FindByUsername(username);
}

public override User FindByUsernameOrEmailAddressAndPassword ( string usernameOrEmail, string password )
{
    //Ignore the password
    return base.FindByUsername(usernameOrEmail) ?? base.FindByUsername(usernameOrEmail);
}

That completes the change for the user service.  Now we just need to register it by modifying ContainerBindings.Load (App_Start\ContainerBindings.cs).

//MODIFIED: Use ADUserService
Bind<IUserService>()
    .To<ADUserService>()
    .InRequestScope();

 

Updating the User View

The view UserDisplay.cshtml is responsible for displaying the log on, register and log off buttons.  None of these make sense with Windows auth so modify the view to remove them.  I went ahead and displayed the full domain name of the user.  If you wanted to get fancy you could either display only the user name or even query AD to get the user's display name.

<div class="user-display">
   <span class="welcome">@User.Identity.Name</span>
</div>

 

Adding Authors Role

By default a user going to NuGet won't require authentication so the user service won't be called.  However when a user does anything that requires authentication such as trying to upload a package they will get redirected to the login page.  This happens because NuGet uses the Authorize attribute on the controller action.  Since we'll be using Windows auth the user will already be authorized so we don't need to check for authentication so we could remove the attribute.  But at some point a user has to be added to the NuGet database in order to upload packages.  We could do that when the user comes to the site but presumably most users don't need an account.  Instead it is probably better to only create the NuGet account when the user tries to do something that requires one (i.e. an author action).  So instead of removing the authorization attribute we'll create a new action filter attribute that verifies the user is in the Authors role.  As part of this check the filter will ensure that the user has an account in the NuGet database.  Add a new action filter called MustBeAuthorAttribute.

public class MustBeAuthorAttribute : AuthorizeAttribute
{
    public MustBeAuthorAttribute ()
    {
        this.Roles = "Authors";
    }

    public string ViewUrl
    {
        get { return "~/Users/Account/AccessDenied"; }
    }

    public override void OnAuthorization ( AuthorizationContext filterContext )
    {
        base.OnAuthorization(filterContext);

        if (filterContext.Result is HttpUnauthorizedResult)
        {
            filterContext.Result = new RedirectResult(ViewUrl);
            return;
        };

        EnsureUserIsRegistered(filterContext.HttpContext.User.Identity.Name);
    }

    private void EnsureUserIsRegistered ( string username )
    {
        var svc = GetUserService();
        var user = svc.FindByUsername(username);
        if (user == null)
        {
            //Get the user name without the domain
            var member = Membership.GetUser(username.Split('\\').Last());

            //Create the user in NuGet
            svc.Create(username, "", member.Email);
        };
    }

    private IUserService GetUserService()
    {
        return (IUserService)Container.Kernel.GetService(typeof(IUserService));
    }
}

In order to authenticate the user must be in the Authors role (we'll set this up later).  If the user is authenticated then we confirm they have a user account with NuGet by looking them up.  If we don't find them then we call the Membership API to get their email address (we'll see how this works later) and then creating their user account.  Notice the password is empty because we don't need it.  The final step is to replace all instances of [Authorize] with the new attribute.

If the user is not authenticated then they are redirected to a new access denied page.

@model SignInRequest
@
{
   ViewBag.Title = "Access Denied";
}

<h1>Access Denied</h1>

<p> You have insufficient privileges to acces this page.</p>

 

public partial class AuthenticationController : Controller
{
    public virtual ActionResult AccessDenied ()
    {
        return View();
    }
    ...
}

 

Switching to AD Authentication

It's time to modify the config file to use AD authentication instead of Forms authentication.  .NET already ships with a provider that will use AD and implement the functionality needed by the Membership API.  All we need to do is configure it.  Replace the existing authentication element with this.

<authentication mode="Windows" />

Add an authorization element that allows all users access to the site.  You could technically limit access to specific users (such as developers) but that might cause problems with Visual Studio.  Also note that the config file already contains an element under a location element.  Do not change that element!!

<authorization>
   <allow users="*" />
</authorization>

Add the necessary role and membership providers to use AD authentication.

<roleManager enabled="truecacheRolesInCookie="falsedefaultProvider="RoleManagerAzManProvider">
    <providers>
        <clear />
        <add name="RoleManagerAzManProvidertype="System.Web.Security.AuthorizationStoreRoleProvider,System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                connectionStringName="LocalPolicyStoreapplicationName="NuGet" />
    </providers>
</roleManager>
<membership defaultProvider="ActiveDirectoryProvider">
    <providers>
        <add name="ActiveDirectoryProvidertype="System.Web.Security.ActiveDirectoryMembershipProvider,System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                connectionStringName="ActiveDirectoryConnectionconnectionProtection="SecureenablePasswordReset="falserequiresQuestionAndAnswer="falseenableSearchMethods="trueattributeMapUsername="sAMAccountName" />
    </providers>
</membership>

This hooks up AD to the Membership API allowing us to query for basic AD information such as user name and email address.  The above configuration relies on AzMan which we'll discuss next. 

Note: Ensure that the web site has Windows authorization enabled but not Anonymous otherwise IIS will not work properly.

Authorization Manager

Since we're using application roles and AD understands Windows groups we need a translation layer.  Fortunately newer versions of Windows has us covered with Authorization Manager (AzMan).  AzMan allows us to define our application roles and store them in one of several different data stores.  IT administrators can then configure what AD users and groups are a member of each role.  For NuGet I'm going to use AzMan backed by an XML file but you could just as easily use a SQL database.  Getting the data store set up is pretty straightforward.

  1. Start Microsoft Management Console (mmc.exe).
  2. Add or browse to Authorization Manager.
  3. Create a new data store (i.e. AzManNuGetStore.xml)  Note: You must be in Developer Mode (menu Action\Option) to set up the store but you can switch back to Administrator Mode after the roles have been defined.
  4. Under the store add a new application called NuGet.  This value must match the role manager's applicationName entry in the config file.
  5. Under Definitions\Role Definitions define the application roles - Authors, Admins
  6. Save the data store and optionally switch it back to Administrator Mode

At this point the application specific AzMan settings are completed but no AD users or groups are associated with the roles.  To assign an AD user or group to an application role open the store in MMC and go to the Role Assignments section.  In general dev leads and managers will likely be Admins while the developer groups will be Authors.

Place the store file in the root of the web site so it can be found.  The app pool identity will need read access to this file.  Now the membership provider needs to be pointed to the file.  To do that we'll add two entries to the connectionStrings section.  The first entry (LocalPolicyStore) provides the path to the AzMan store.  This must match the role manager's connectionStringName entry.  The second entry will be the AD connection information.  This must match the membership provider's connectionStringName entry.

<add name="LocalPolicyStoreconnectionString="msxml:{path}/AzManNuGetStore.xml" />
<add name="ActiveDirectoryConnectionconnectionString="LDAP://{domain}/DC={subdomain},DC={subdomain}" />

Note that the path for the AzMan store uses URL slashes (/) between directory names.  That's it.  Now ASP.NET is hooked up to AD for authentication and AzMan is providing the roles based upon the AD groups that are configured.

Additional Enhancements

As I mentioned in the earlier article there are some things about NuGet that don't work well in a private environment, at least for me.  Therefore I went ahead and updated the code for them as well.  The next few paragraphs discuss some of these enhancements.  Refer to the original article for more information.

I wanted to make the option of using HTTPS configurable so the code didn't have to keep changing.  Therefore I added a new gallery setting (Gallery.RequireHttps) to allow it to be toggled on and off.  It is set in the config file.  I updated the RequireRemoteHttpsAttribute to read the app setting and do nothing if it is disabled.  I didn't bother caching the result but you could if you wanted to. 

public void OnAuthorization(AuthorizationContext filterContext)
{
    //MODIFIED: Don't require HTTPS if it is disabled
    var requireHttps = Configuration.ReadAppSettings("requireHttps", x => (x != null) ? Boolean.Parse(x) : false);
    if (!requireHttps)
        return;

Email, by default, uses SSL.  I wanted to be able to turn this off as well.  Most of the email settings are stored in the database so that would be the logical place to change it but NuGet uses EF and updating the models and whatnot would be too much work so I added yet another gallery setting (Gallery.EnableSmtpSsl) to control whether or not to use SSL for email.  To use this setting I modified the ContainerBindings.Load (App_Start\ContainerBindings.cs) method to read the setting when it is creating the mailSenderThunk object.  Here's the relevant code for reference.

var mailSenderThunk = new Lazy<IMailSender>(
    () =>
        {
            var settings = Kernel.Get<GallerySetting>();
            if (settings.UseSmtp)
            {
                var mailSenderConfiguration = new MailSenderConfiguration
                    {
                        DeliveryMethod = SmtpDeliveryMethod.Network,
                        Host = settings.SmtpHost,
                        Port = settings.SmtpPort,

                        //MODIFIED: Pull from the settings
                        EnableSsl = Configuration.ReadAppSettings("enableSmtpSsl", x => (x != null) ? Boolean.Parse(x) : false)
                    };

The newer versions of NuGet displays statistics on the home page.  The stats are handled via stats.js.  Unfortunately this file has a bug in the path that it uses to get the stats.  If the site is hosted as a full website then it will work but if it is an application under a website then the path is wrong.  Since the stats are handled in a Javascript file the change isn't as simple as I'd like.  After talking with one of my Javascript experts we came up with this.

  1. Add a new Javascript variable to the shared layout page that stores the root path of the site.
  2. Modify stats.js to use the Javascript variable rather than using a rooted directory.
<script type="text/javascript">
var rootUrl = "@Url.Content("~")";
</script>

//Inside stat.js:getStats
$.get(rootUrl + 'stats/totals', function(data) {

That should resolve any issues with the stats not displaying on the main page.  The site should now be configured to use AD authentication.  You should confirm the website is working properly as well as being able to view packages from Visual Studio and during builds.

What's Not Covered

Some areas of NuGet have not been covered and may need to be changed if you use them.

  • Uploading packages from the web API might work since the web API doesn't require any special privileges.  The user credentials that are passed should authenticate against the custom user service but some tweaks may be necessary.
  • Accessing any of the existing user APIs will probably not fail but won't work as expected.  I originally started removing all the code but NuGet uses T4MVC which puts way too many hooks into the system for my taste.  In the end I left the existing API in place so I wouuldn't have to touch every file.
  • Non-Active Directory networks won't work directly since the user information is not available.  Specifically the email address would need to be obtained through some other means.
Using T4 to Create an AppSettings Wrapper, Part 4

In the previous article we finished the basic appsettings template that allowed us to generate a strongly typed class from the settings in a configuration file.  We now want to expand the template to allow it to be customized depending upon the needs of the project, a later article.  Before we get there though it is time to refactor the code to make it more reusable and easier to maintain.  That is the focus of this article.

T4 Toolbox

Under the hood a template is nothing more than a compiled class that implements a method to render the template.  Up to this point we have been using the default class because our template wasn't that complicated but we want to be able to customize the template so it is time to move away from the default settings.  The T4 Toolbox is a handy set of extensions to T4 that allows us to work with the template as a class rather than as a text file.  By switching to a class approach we can take advantage of all the features we are used to in normal code but still ultimately generate a template.  There is nothing we are going to implement that you couldn't do by hand or with another library but the T4 Toolbox makes things easier so we'll use it.  You'll need to download and install it before moving on.

To make the best use of T4 Toolbox we need to move our template into a class that derives from Template.  We can then implement the TransformText method to generate our template.  First ensure that the T4 Toolbox is installed through Extension Manager.  Normally if you're creating a new template you will add a new template file using T4 Toolbox\Template but since we already have one we'll just modify our existing template file.  

Including Files

T4 allows you to include text files inside templates using the include directive.  It works similar to C++ #includes in that the contents of the file are inserted into the template file whereever the directive resides.  This is useful for sharing common functionality across multiple templates.  The included file must be either in the same directory as the template, a directory relative to the template or in a path that T4 will search.

Open the template file and add an include for the T4 Toolbox.

<#@ include file="T4Toolbox.tt" #>

This include brings in the infrastructure needed by T4 Toolbox.  I normally place it after the assembly directives but before the import directives.  A word of warning, T4 complains if it finds multiple assembly or import statements for the same string so try to keep these directives down to a single file.

Creating the Template Class

Now it is time to move the template into a Template class. Right after the import directives start a new class block.  Define a new class that derives from CSharpTemplate.  The class name is not that relevant.

...
<#@ output extension=".generated.cs" #>
<#+ 
public class AppSettingsTemplate : CSharpTemplate
{
}
#>

Move any methods from the original template inside the class body.  They should probably be private since they are used only for generating the template.  The last method in the class should be the override for the TransformText.  You can generate the template body by making method calls but the easier approach is to simply end the class block and starting the template text.  After the template text will be another class block that finishes the TransformText method.  The call to GenerationEnvironment is what returns the template text from the method.  The template text outside of the class block is automatically added to the returned text. 

    public override string TransformText ()
    {
        var className = MakeIdentifier(Path.GetFileNameWithoutExtension(Host.TemplateFile));
#>
/*
 * This file is auto-generated from a config file.   BR /> * Do not modify this file.  Modifications will be lost when the file is regenerated.
 */

...
<#+
        return this.GenerationEnvironment.ToString();
    }
}
##>

Remember that indentation and blank lines can be confusing in templates so it might be necessary to play around with the template to get them right.  The goal isn't to make the template be styled correctly but rather the generated code.

Notice that the className variable that was in a statement block before has been moved inside the method.  Variables defined in the method are accessible to the template text.  Also note that any statement blocks used inside the template text need to be switched to class blocks because a statement block cannot follow a class block.

Removing Unneeded Code

When we were writing the template in previous articles we defined a few helper methods.  Some of these can go away because T4 Toolbox provides them for us.

MakeIdentifier() was used to create a valid identifier given a string.  It can be replaced by a call to PropertyName() if you want a Pascal cased identifier or FieldName() if you want a camel cased identifier.  Replace the calls and remove the method.

GetNamespaceForTemplate() was used to determine the namespace to use for the type.  It can be replaced by a call to the property DefaultNamespace which should provide the same information.  Replace the calls and remove the method.

Host was used to access the host information.  T4 Toolbox wraps the host to better provide isolation from the underlying implementation.  Replace references of Host with TransformationContext.Current.Host.

Nested Templates

If you run the template at this point you'll see that the generated file is empty.  What's going on?  If you look at your template file you'll realize that all you've done is defined a template class.  There isn't anything that is actually causing the template to run.  Time to fix that.

A nested template is a template inside a template.  It is one of the key techniques that we can use to both encapsulates a template and make it configurable.  The issue with the current template is that someone would need to know how the template works before they could customize it.  Additionally they would need to change the template file to make any customizations.  If we later release a new version of our template then they would either need to manually merge the changes in or redo their customizations.  A nested template allows us to expose a simple template that exposes the customizable parts of the template (generally through properties) while calling the real template to do the actual work.  This allows us to separate the real template from the customizable parts.  Let's break our template up into the customizable part and the template part. 

I like for my customizable template to bear the name of the template as will be shown in the Add New Items dialog and the real template to match the template class name so rename the existing template file to AppSettingsTemplate.tt.  Then create a new template file called AppSettings.tt which will be the customizable part.

The customizable template represents what will get generated so it needs the template directive, the output directive and an include of the real template.

<#@ template debug="false" hostspecific="true" language="C#" #>
<#@ output extension=".generated.cs" #>
<#@ include file="AppSettingsTemplate.tt" #>
<#
#>

The real template doesn't need the template or output directives anymore.  In fact it would be an error to leave them in.  Instead it just defines the assemblies and namespaces needed to generate the template.  Additionally we need to tell VS not to treat it as a template file anymore so view the properties of the file and remove the custom tool setting.  A caveat to this is that VS will not automatically regenerate the output if you modify this template anymore.  If you make any changes to it you will need to regenerate it manually.

The customizable template needs to create an instance of the real template, call any customization members that it needs to and then render the template.  Since we have no customization yet it would boil down to this.

<#
   var template = new AppSettingsTemplate();

   //Do customization here

   template.Render();
#>

We've now set the stage to allow our template to be customized.  In the next article we'll expose customization points to make our template truly useful in real world projects.

Posted Sun, Mar 24 2013 by Michael Taylor | 18 comment(s)
Filed under:
Using T4 to Create an AppSettings Wrapper, Part 3

In the previous article we expanded the template to dynamically set the type and namespace names based upon the project the template is used in.  Now we are going to turn our focus to generating properties for each of the settings in the configuration file.  For this we'll be adding more code to the class block to open and read the settings from the project's configuration file (which we'll have to write code to retrieve).  We'll also have to decide how to convert string values to typed equivalents. 

Generating Property Code

Before we read the configuration file let's go ahead and modify the template to generate the properties that back the settings.  We can do this by creating a simple method that returns a pre-defined list of setting key and value pairs.  We can then enumerate them and generate the corresponding properties.  Since this is template generation code we'll place it in the class block where the other methods are.  We'll also make it private as it is specific to this template.  Later when we generalize the template private methods will remain with the template.

private Dictionary<stringstring> GetAppSettings ()
{
    //Stub some values
    return new Dictionary<stringstring>() {
            { "IntValue""10" },
            { "StringValue""Hello" },
            { "DoubleValue""45.6" }
};

Now we can remove the static public properties we initially added and replace them with code that enumerates each setting and generates the corresponding public property.  Notice that we mix code blocks with static content.  This is handy to do (reminds me of ASP.NET or MVC) but can get confusing if used too much.  The setting key becomes the property name and is used to retrieve the value at runtime.  The setting key is inserted using an expression block. 

<#
        foreach (KeyValuePair<stringstring> setting in GetAppSettings()) { 
#>

        public string <#= MakeIdentifier(setting.Key) #>
        {
            get { return GetConfigSetting("<#= setting.Key #>"); }
        }
<#      }
#>

Saving the template should cause the three properties to be generated.  At this point we could reference the properties in some code, compile and run the code and the values from the config file should be pulled used.  We're close to being done with the basic implementation but let's clean up a few things before we start reading values from the config file.  The first thing to note is that GetSafePascalName has been renamed to MakeIdentifier.  It is shorter and more clear.

Determining the Property Type

We need to determine the correct property type.  If the setting is an integer, for example, we'll generate an Int32 type.  For most settings the following types should work: int, double, bool and string.  We'll use simple test parsing to find the correct type.  If we cannot match any type we'll use string.  The GetSettingType method accepts the setting the value and attempts to parse it.  The Type is returned back so we can use it to generate the correct code later.

private Type GetSettingType ( string value )
{
    //Try to convert to int first
    int iValue;
    if (Int32.TryParse(value, NumberStyles.Integer, nullout iValue))
        return typeof(int);

    //Double is next
    double dValue;
    if (Double.TryParse(value, NumberStyles.Float | NumberStyles.AllowCurrencySymbol, nullout dValue))
        return typeof(double);

    //Boolean
    bool bValue;
    if (Boolean.TryParse(value, out bValue))
        return typeof(bool);

    return typeof(string);
}
<#
        foreach (KeyValuePair<stringstring> setting in GetAppSettings()) { 
            var type = GetSettingType(setting.Value);
#>

        public <#= GetFriendlyName(type) #> <#= MakeIdentifier(setting.Key) #>
        {
            get { return GetConfigSetting("<#= setting.Key #>"); }
        }
<#      }
#>

I like for my code to use the standard aliases for built in types so I wrote a quick and dirty function to get the alias, if any, given a type.  It is not efficient but it works without me having to write a mapping table myself.  We need to include the System.CodeDom, System.Globalization and Microsoft.CSharp namespaces before this will compile.

public static string GetFriendlyName ( Type source )
{
    var provider = new CSharpCodeProvider();

    var tr = new CodeTypeReference(source);

    var str = provider.GetTypeOutput(tr);

    //Ref and out parameters have an & on the end so strip it off
    if (str.EndsWith("&"))
        return str.Substring(0, str.Length - 1);

    return str;
}

Type conversion

The properties now have the correct type but we're returning strings from the properties so the code won't compile.  We now need to convert the string values as obtained from ConfigurationManager to the correct type.  To keep this relatively clean we'll create yet another private method that accepts a type as a parameter and returns the name of the conversion function to call.  We'll wrap the entire call to ConfigurationManager with a call to the conversion routine.  In the case of a string there is no conversion so the generated property will just be wrapped in parenthesis.

private Lazy<Dictionary<Type, string>> m_settingsConverter = new Lazy<Dictionary<Type, string>>(InitializeSettingsConverter);

private string GetSettingConverter ( Type type )
{
    string converter;
    if (m_settingsConverter.Value.TryGetValue(type, out converter))
        return converter;

    return "";
}

private static Dictionary<Type, string> InitializeSettingsConverter ()
{
    return new Dictionary<Type, string>() {
            { typeof(sbyte), "SByte.Parse" },
            { typeof(short), "Int16.Parse" },                    
            { typeof(int), "Int32.Parse" },
            { typeof(long), "Int64.Parse" },           

            { typeof(byte), "Byte.Parse" },
            { typeof(ushort), "UInt16.Parse" },
            { typeof(uint), "UInt32.Parse" },
            { typeof(ulong), "UInt64.Parse" },

            { typeof(float), "Single.Parse" },
            { typeof(double), "Double.Parse" },        
            { typeof(decimal), "Decimal.Parse" },        
                    
            { typeof(bool), "Boolean.Parse" },                                                                                
            { typeof(char), "Char.Parse" },
            { typeof(DateTime), "DateTime.Parse" },
            { typeof(Guid), "Guid.Parse" },
            { typeof(TimeSpan), "TimeSpan.Parse" },

            { typeof(string), "" }
            };
}

The type to converter mapping is stored in a dictionary to optimize performance.  For this code I'm using the standard Parse methods but you can replace it with any method you want.  Here's the updated property block.

public <#= GetFriendlyName(type) #> <#= MakeIdentifier(setting.Key) #>
{
    get { return <#= GetSettingConverter(type) #>(GetConfigSetting("<#= setting.Key #>")); }
}

We've finished the propery generation code.  Depending upon the value stored in the config file (when the template is generated) a strongly typed property will be generated for each app setting.  Type conversion is used to convert the value in the config file at runtime to the correct type.  We have effectively replaced the Settings-based code added to .NET with our own implementation that works similarily but without having to add anything to our config files.  But we aren't quite done yet as we're still using hard coded values.  It is time to retrieve the settings from the config file directly.

Reading the Configuration File

In order to get the settings from the configuration file we have to locate the config file in the project (keeping in mind it could be a web or Windows app).  We already have the code to find a specific project item so we just need to use it to find the config file.  For now we'll assume the config file exists.  In a later article we'll see how to handle a missing config file.

public EnvDTE.ProjectItem FindConfigProjectItem ( EnvDTE.Project source )
{
    return FindItem(source, "web.config"true) ?? FindItem(source, "app.config"true);
}

Now it becomes a simple matter of loading the settings from the config file using ConfigurationManager.  We'll modify our stub method of earlier to return the information.  Note we need to add another import for System.Configuration.  We'll also need to ensure we have an assembly directive for System.Configuration because it is not included by default.

private KeyValueConfigurationCollection GetAppSettings ()
{
    //Get the config file
    var configItem = FindConfigProjectItem(ActiveProject);

    return GetAppSettings(configItem);
}

private KeyValueConfigurationCollection GetAppSettings ( EnvDTE.ProjectItem configItem )
{
    if (configItem != null)
    {
        var configFile = new ExeConfigurationFileMap() { ExeConfigFilename = GetProjectItemFileName(configItem) };

        var config = ConfigurationManager.OpenMappedExeConfiguration(configFile, ConfigurationUserLevel.None);

        return (config.AppSettings != null) ? config.AppSettings.Settings : null;
    };

    return null;
}

Notice that we changed the return type to match what the configuration subsystem will return.  We'll have to update the foreach block as well but the rest of the code is fine.

<#
        foreach (KeyValueConfigurationElement setting in GetAppSettings()) { 
            var type = GetSettingType(setting.Value);
#>

Save the template and the same properties we had hard coded earlier should still be available.  This time though they are being pulled from the project's config file.  Feel free to add more settings to confirm everything is working correctly.  At this point we have a fully working and useful template that we can use in projects to clean up settings code.  Not bad for 300 lines of code.  We have met 3 of the 6 initial requirements.  In the next article we're going to modify the template to meet the remaining requirements making it more useful and flexible to meet the needs of complex projects.

Attachment: T4AppSettings.zip
Posted Sun, Mar 17 2013 by Michael Taylor | 6 comment(s)
Filed under:
Using T4 to Create an AppSettings Wrapper, Part 2

In the first article in this series we created a basic, static T4 template.  The template allowed us to replace standard boilerplate code for reading an app setting

string setting = ConfigurationManager.AppSettings["someSetting"];

with strongly typed property references like this

var myIntSetting = AppSettings.Default.IntValue;
var myDoubleSettig = AppSettings.Default.DoubleValue;

Here's a summary of the requirements from the first article (slightly reordered).

  1. All settings defined in the project's config file should be exposed, by default, as a public property that can be read. I'm not interested in writing to them.
  2. Based upon the default value (in the config file) each setting should be strongly typed (int, double, bool, string).
  3. The configuration file cannot be cluttered with setting-generation stuff. This was the whole issue with the Settings designer in .NET.
  4. Sometimes the project containing the config file is different than the project where the settings are needed (ex. WCF service hosts) so it should be possible to reference a config file in another project.
  5. Some settings are used by the infrastructure (such as ASP.NET) so they should be excluded.
  6. Some settings may need to be of a specific type that would be difficult to specify in the value (ex. a long instead of an int).

In this article we're going to convert the static template to a dynamic template and begin working on requirements 1-3.  From last time here are the areas of the template that need to be made more dynamic.

  1. The namespace of the type
  2. The name of the type, including pretty formatting
  3. Each public property name and type

Template Structure

Pretty much every T4 template will include a type within a namespace.  A template should follow the standard practices that are currently in use in terms of names and formatting.  Every (code) project has a default namespace.  It is expected that all types added to the project be placed into the default namespace.  However, at least in C#, if a file is placed into a subfolder then the subfolder becomes part of the namespace.  The template should use the default namespace combined with the folder(s) containing the file. Unfortunately this information is not directly accessible to the template.  We'll have to write some code to get this information but a quick overview of underpinnings of the template is in order.

A template is compiled into a derived type of TextTransformation.  This type exposes the core functionality needed for a template include error reporting, writing to the generated file and context information.  Unfortunately the base type does not expose the information we need so we'll have to call out to Visual Studio (the host).  In order to do that we first need to set the hostspecific attribute on the template directive to true.  You will probably want to set the debug attribute to true as well to make debugging easier.  Once the host attribute is set we can use the Host property to access information about the template.  This opens up most of the dynamic data we need.

It is important to distinguish between code used for template generation vs code that will end up in the generated file.  Template generation code will reside inside statement blocks (<# #>), expression blocks (<#= #>) or class blocks (<#+ #>).  A statement block is used to execute code during template generation.  Any variables defined in block are accessible to later blocks.  An expression block is used to inject a value into the generated code.  It is normally a: variable defined in a statement block, a function defined in a class block or a property of the template class.  Expression blocks are how we'll get the dynamic data into the generated code. A class block is used to add members to the generated template class.  Class blocks are useful for creating reusable functions that can be used during template generation.  One limitation of class blocks is that they cannot be followed by statement blocks.  In general this isn't an issue because a statement block is used to start the code generation and therefore it will be first.

Standard Variant Replacement

Now that we have access to Host we can get the name of the template file (as set in the Add New Item dialog).  The file name (without the extension) will be the name of the settings class so we should go ahead and store off the name in a variable for later use.

<#
    var ClassName = Path.GetFileNameWithoutExtension(Host.TemplateFile);
#>

We're using Path here so we should also include an import for System.IO since we'll be needing this namespace later.  Now that we have a variable representing the class name we should go ahead and replace the static AppSettings typename with the type name as determined by the filename using an expression block.

internal partial class <#= ClassName #>

Each time you make a change to the template you should go ahead and save the template.  Check the Error List window to ensure the template compiled correctly and verify the generated file.  Finding template generation errors can be difficult so verifying each change as you go along makes things much easier.  Note that we are currently ignoring the case where the name might contain invalid identifier characters.

Getting the namespace requires a little more work.  There are quite a few ways to get the information but I prefer to get the namespace straight from the project.  Each project item has a custom namespace property but I generally don't bother using it.  Once you start needing information from VS then you're going to run into DTE which is the main VS object.  A good understanding of the VS object model will really be beneficial but is beyond the scope of this post so I'm just going to provide the necessary code to get the project associated with the template.  Because this is useful code and we'll need it later I'm going to place it in a class block at the bottom of the template.  Remember these methods will be part of the type that is generated to back the template generation.

  1. Add assembly directives to include the required Visual Studio assemblies (EnvDTE in this case).
  2. Add import directives to include the DTE namespaces.  Note that I prefer to not import the EnvDTE namespace where the core model is at because it makes it much cleaner to read, in my opinion.
  3. Add the code to a class block at the bottom of the template.  Note there are several helper methods that will be beneficial later.
<#+
public
 EnvDTE.Project ActiveProject
{
    get
    {   if (m_activeProject == null)
        {
            if (DteInstance != null)
            {
                var projects = (Array)DteInstance.ActiveSolutionProjects;

                m_activeProject = (projects != null && projects.Length > 0) ? (EnvDTE.Project)projects.GetValue(0) : null;
            };
        };

        return m_activeProject;
    }
}

public EnvDTE.DTE DteInstance
{
    get
    {
        if (m_dte == null)
            m_dte = (EnvDTE.DTE)((IServiceProvider)Host).GetService(typeof(EnvDTE.DTE));

        return m_dte;
    }
}

public static EnvDTE.ProjectItem FindItem ( EnvDTE.Project source, string itemName, bool recurse )
{
    var items = source.ProjectItems;

    //ProjectItems.Item() will throw if the item does not exist so do it the hard way        
    foreach (EnvDTE.ProjectItem child in items)
    {
        if (String.Compare(child.Name, itemName, true) == 0)
            return child;
    };

    if (recurse)
    {
        foreach (EnvDTE.ProjectItem child in items)
        {
            var item = FindItem(child, itemName, true);
            if (item != null)
                return item;
        };
    };

    return null;
}

public static EnvDTE.ProjectItem FindItem ( EnvDTE.ProjectItem source, string itemName, bool recurse )
{
    var items = source.ProjectItems;

    //ProjectItems.Item() will throw if the item does not exist so do it the hard way        
    foreach (EnvDTE.ProjectItem child in items)
    {
        if (String.Compare(child.Name, itemName, true) == 0)
            return child;
    };

    if (recurse)
    {
        foreach (EnvDTE.ProjectItem child in items)
        {
            var item = FindItem(child, itemName, true);
            if (item != null)
                return item;
        };
    };

    return null;
}

public string GetNamespaceForTemplate ()
{
    //Get the project's root namespace
    var rootNamespace = ActiveProject.Properties.Item("RootNamespace").Value as string;

    //Get the project item for the template file
    var templateItem = FindItem(ActiveProject, Path.GetFileName(Host.TemplateFile), true);

    //Walk backwards until we get to the project root, concatenate each folder to the namespace
    var parentNamespaces = new List<string>();
    var parent = templateItem.Collection.Parent as EnvDTE.ProjectItem;
    while (parent != null)
    {
        parentNamespaces.Add(GetSafePascalName(parent.Name));
        parent = parent.Collection.Parent as EnvDTE.ProjectItem;
    };

    var orderedNamespaces = parentNamespaces as IEnumerable<string>;
    var folderPath = String.Join(".", orderedNamespaces.Reverse());

    return (folderPath.Length > 0) ? rootNamespace + "." + folderPath : rootNamespace;
}

public string GetProjectItemFileName ( EnvDTE.ProjectItem item )
{
    return (item != null && item.FileCount > 0) ? item.FileNames[0] : "";
}

public string GetSafePascalName ( string baseName )
{
    var builder = new System.Text.StringBuilder();

    //Starts with capital letter or underscore
    bool isFirstLetter = true;
    foreach (var ch in baseName)
    {
        if (isFirstLetter)
        {
            if (Char.IsLetter(ch))
                builder.Append(Char.ToUpper(ch));
            else if (Char.IsDigit(ch) || ch == '_')
                builder.Append(ch);

            isFirstLetter = false;
        } else
        {
            if (Char.IsLetterOrDigit(ch) || ch == '_')
                builder.Append(ch);
        };
    };

    if (builder.Length == 0)
        builder.Append('_');

    return builder.ToString();
}

private EnvDTE.DTE m_dte;
private EnvDTE.Project m_activeProject;
#>

The GetSafePascalName method is used to generate a safe Pascal name.  It is probably a good idea to go back and add a call to this method when setting the ClassName variable from earlier.

<#
   var ClassName = GetSafePascalName(Path.GetFileNameWithoutExtension(Host.TemplateFile));
#>

Finally we can use the new functionality to replace the static namespace.

namespace <#= GetNamespaceForTemplate() #>
{  ...

At this point we should be able to move the template file to a different folder and the generated code should update the namespace accordingly.  If we rename the template file then the generated type name should reflect this as well.

We've written a lot of code and we're not done yet.  But the functionality that has been added will be reusable in all the other templates we create.  In the next article in this series we'll replace the static properties with the real settings from the configuration file.

Attachment: T4AppSettings.zip
Posted Sun, Mar 10 2013 by Michael Taylor | 35 comment(s)
Filed under:
Using T4 to Create an AppSettings Wrapper, Part 1

AppSettings are settings stored in your configuration file under the <appSettings> element. Almost every application has them. Each setting consists of a name and value. To access such a setting in code you need only do this.

string setting = ConfigurationManager.AppSettings["someSetting"];

There are a couple of problems with this approach.

  • Quite a bit of boilerplate code to access a setting given what it is actually doing
  • The setting name is hard coded and must match the config file
  • The returned value is a string so if you need a different type then you'll need to convert it

In .NET v2.0 Microsoft added the Settings class to work around these issues. It allows you to create a setting with a type and value and the designer will generate a type to back it where each property matches the setting. This seems great but never really took off. Not even Microsoft uses it in their own framework. Part of the problem is that the config entries it generates are overly complex storing things like type information, default values and other things. Needless to say appSettings continue to be popular anyway. Fortunately we can get the simplicity of appSettings with the power of the newer Settings class all via T4.

In this series of posts I'm going to walk through the process of generating such a template including the ability to add some more advanced functionality.  A full discussion of T4 is beyond the scope of a blog so refer to the following links for more information.

Requirements

At my company I was tired of the limitations of working with appSettings so I combined some work I've done in the past with the T4 engine to produce a template that all projects can use to simplify working with app settings. Because of the dynamics of the code I work in I had some additional requirements.

  • All settings defined in the project's config file should be exposed, by default, as a public property that can be read. I'm not interested in writing to them.
  • Based upon the default value (in the config file) each setting should be strongly typed (int, double, bool, string).
  • Sometimes the project containing the config file is different than the project where the settings are needed (ex. WCF service hosts) so it should be possible to reference a config file in another project.
  • Some settings are used by the infrastructure (such as ASP.NET) so they should be excluded.
  • Some settings may need to be of a specific type that would be difficult to specify in the value (ex. a long instead of an int).
  • The configuration file cannot be cluttered with setting-generation stuff. This was the whole issue with the Settings designer in .NET.

Defining the Generated Code

Before you can write a T4 template you need to know what you are going to generate.  The easiest way to do that is to write the actual code you want, given specific inputs.  Here is the basic code to be generated given the specified inputs.

<appSettings>
        <add key="DoubleValuevalue="45.678" />
        <add key="IntValuevalue="123" />
        <add key="StringValuevalue="Text" />
    </appSettings>
using System;
using System.Configuration;

namespace P3Net
{        
    internal partial class AppSettings
    {    
        /// <summary>Gets the default instance.</summary>
        public static AppSettings Default
        {
            get { return s_defaultInstance; }
            protected set { s_defaultInstance = value ?? new AppSettings(); }
        }

        #region Setting Properties

        public double DoubleValue
        {
            get { return Convert.ToDouble(GetConfigSetting("DoubleValue")); }
        }

        public int IntValue
        {
            get { return Convert.ToInt32(GetConfigSetting("IntValue")); }
        }

        public string StringValue
        {
            get { return Convert.ToString(GetConfigSetting("StringValue")); }
        }
        #endregion

        protected virtual string GetConfigSetting(string settingName)
        {
            var setting = ConfigurationManager.AppSettings[settingName];
            return setting ?? "";
        }                

        private static AppSettings s_defaultInstance = new AppSettings();
    }
}

To make the generated type easier to use it is marked as partial and the main method (GetConfigSetting) that all settings are read through is virtual.  But to keep calling code simple the type is implemented as a singleton (although it isn't enforced).  Each property is backed by the corresponding entry from the config file.  For our purposes type conversion will be handled by the Convert class but you should really use a more resilient type conversion system.   

Identifying the Variant Parts

Once you've defined the format of the final code you need to identify the parts that will vary based upon the inputs.  For the earlier code the properties will vary based upon the config file.  There will be a property for each setting in the config.  The property name will follow the setting name and the property type will be determined by the value stored in the config file.  Because settings are returned as strings by the subsystem there will be a type conversion call as well.

But there is more variant components than just the inputs and these must be taken into account as well.  When you add a type to a project (such as the generated code) it is expected to place the type in the default namespace for the project with any child folder taken into account.  Therefore the namespace name is a variant in the template.  The actual name of the type should generally follow the name of the file.  The filename is set by the user when they add the template to the project.  Therefore the type name needs to be a variant and any references to it must be variants as well. 

In the above code all the variant parts are highlighted in yellow.  Each of the highlight parts will need to be converted to an expression that T4 can replace when it generates the template.

Creating the Basic T4 Template

Now that we know what code we want to generate it is time to create the template.

  1. Create a new project or opening an existing project where the settings class will be used.
  2. Ensure the project has a configuration file with some app settings defined.  (I assume you are using the entries mentioned above).
  3. Add a new Text Template item (under Visual C#\General) to the project called AppSettings.tt.  This will be the template name. 
  4. Inside the .tt file paste the actual code you want generated at the bottom of the file.
  5. Save the file.  Whenever the .tt file is saved it will rerun the template.
  6. Assuming nothing went wrong you can expand the AppSettings.txt file under the template and you should see the correct generated code.

If the template failed for any reason then the errors will appear in the Error List.  Otherwise you should be able to set up a simple test call to verify the code is working. 

var intValue = AppSettings.Default.IntValue;
var doubleValue = AppSettings.Default.DoubleValue;
var stringValue = AppSettings.Default.StringValue;

But if you try to do so you'll find that the type doesn't show up.  That is because, by default, T4 templates generate text files and not code files.  The output directive in the T4 file determines what type of file to generate.  Change it from ".txt" to ".generated.cs" and save the template.  The type should now show up, provided the project name matches the namespace used in the template.  If not then add a using/import statement.

Parts of a T4 Template

Let's take a look at the template file.  T4 directives appear between <#@ #> delimiters.  Directives generally appear at the top of the file. 

  • template specifies that this is a T4 template.  This directive can appear only once.

    The debug attribute should be set to true to enable debugging until the template is working.  The hostspecific attribute specifies whether the template needs access to the host.  We'll talk about this later.  The language attribute specifies the language of the template code.
  • assembly is equivalent to adding a reference to a project in that it indicates an assembly that is needed by the template.  Any number of these directives can appear.  Note that this is for the generation of the template itself and not the final generated code.  Only assemblies needed to generate the template code should be listed.  The assembly must be accessible by T4.
  • import is equivalent to a using/import statement.  Any number of these can appear.  As with the assembly directive this only impacts the template generation.  Any types needed to generate the template need to have their namespace(s) imported.  The final generated code will explicitly include the using/import statements that it will need.
  • output controls the output of the template.  As mentioned earlier the extension attribute specifies the extension added to the generated file.  It is appended to the template name to generate the final file name.  For source files you should use the standard practice of using ".generated.cs".  Designers use the default ".designer.cs".  When the file is generated, if it is not yet part of the project, then it gets added automatically.  VS will use the file extension to determine how to properly add the file to the project (ex. content, compile).

After the directives is the code that is generated.  An important note about the T4 generator, blank lines matter.  Some directives are stripped out of the generated code, such as the <#@ @> directives but other directives are replaced with a blank line.  Additionally any blank lines that appear in the template are also written verbatim.  Normally when writing code we try to format the code neatly.  In the case of T4 put your formatting rules on the bench.  Place the text to be generated right after the directives if you don't want blank lines in the generated file.  It is also generally a good idea to place a file header in the generated code that mentions the file is auto generated and any changes will be lost.

At this point we have a working T4 template but it doesn't do anything fancy.  It is nothing more than a glorified code snippet but we have set everything up to easily now convert it to a more dynamic template.  We'll cover that in the next part of this series.

Attachment: T4_Part1.zip
Posted Mon, Feb 18 2013 by Michael Taylor | 3 comment(s)
Filed under:
MVP of the Year and Connect

I recently found out that I was one of the MVPs of the year for our group.  Wow what an honor!!  Given the caliber of MVPs one cannot help but be humbled by this.  Unfortunately other obligations prevent me from attending the ceremony to receive the award.  I found out later that one of the reasons was that I was supposedly the top bug reporter on VS 2012 for US dev MVPs.  I reported 27 issues and 13 were actually resolved.  This got me to thinking about Connect and how Microsoft has historically used it.

Historically when an issue was reported someone at Microsoft would try to replicate the issue.  If they could then they would escalate it to the team and you'd receive some feedback.  At that point the issue would either be fixed or, more likely, closed without reason.  More recently Microsoft has started to close items with short descriptions like 'by design' or 'won't fix'.  The one that drives me mad though is 'we have to evaluate the priority of each item reported against our schedule and this issue is not sufficiently important.  We will review it in the future'.  Closed.  The problem is that I'm not convinced they every do "review it in the future".  Even worse is that once an item is closed you cannot do anything with it anymore. 

If, as happened recently to me, the folks at MS failed to grasp the issue you were reporting and closed the item then there is no way to tell them they messed up.  Recently I reported an issue to Microsoft about the behavior of inline tasks for MSBuild (https://connect.microsoft.com/VisualStudio/feedback/details/768289/msbuild-path-used-for-inline-task-reference-is-not-honored).  The issue is that an inline task can reference an assembly using a path.  At compile time this is fine but at runtime the assembly path is not honored so unless you copy the assembly to the same location as the task then it will fail to the find the assembly.  Now I understand the reasoning behind why it would fail.  I also know that I'm not the only one who has seen this issue.  It has been reported on multiple blogs over the years.

Somehow the folks looking at the issue got caught up with what the sample code was trying to do rather than what the actual bug was and reported that I should be using some built in task instead.  How does that in any way relate to my problem?  I don't know but the item was closed anyway.  Without starting a new item I cannot recover this closed item.  Sure I left a follow up comment about it but the item is still closed, the bug still exists and I doubt it will ever get resolved.  And I'd like to think that as a contributor to the community that my items get a least a little more attention than the general user but it wouldn't appear so in Connect.  If MS really wants us to report bugs and Connect is the tool to do so then the entire system needs to evolve into a more full featured issue tracking system where we can alert MS to issues that may have been closed incorrectly.  Even more important issues that "may be resolved in a future release" shouldn't be closed but deferred so we know that at least they are under consideration.  Right now Closed means it's fixed, it's by design, it cannot be repro or it ain't going to be fixed.

Historically MS has taken some flax from the community about the uselessness of Connect.  With VS2012 they seem to have upped their game and started taking feedback more seriously but there is still much work to be done.  There has to be more insight into where an item is in the process, policies for getting items reevaluated and a better system for identifying items that are closed or deferred.  Perhaps the User Voice site will take over bugs as well.  Right now it is more for suggestions.  Time will tell.  Having 50% of my reported items resolved indicates that Connect is starting to work, at least for me, but it has to work for everybody or else it isn't going to be used.

Our industry is plagued by large egos.  I try to keep mine in check (except around a few people who I know will take me for who I am, not what I've done).  Where I work we have a motto "If you're truly good you don't have to say anything".  What that means is that bragging about being an MVP, writing a book, publishing a popular framework or whatever else gets you nowhere.  If you're truly good your works will speak for themselves.  As such I will quietly place my plaque next to my MVP awards and move on.  But recently one of our team members won both the Chili Cookoff contest and the Employee of the Year award in the span of two weeks.  They proudly carried their awards the next few days to all their meetings.  Maybe, just maybe, I'll carry mine to a couple of meetings.  If you're truly good you don't have to say anything but awards don't talk do they :}

Hosting a Private Visual Studio Gallery

Visual Studio extensions are really popular these days.  Not surprisingly Visual Studio has a very active extensions gallery.  But unfortunately the gallery is public so if you develop some extensions for your personal use, or for your company, then you cannot use the public gallery.  Fortunately though VS supports the concept of a private gallery.  A private gallery is nothing more than an extension gallery that is not accessible to the world.  It is discussed on MSDN so refer to there for more information about the what's and why's.  Unfortunately the documentation is a little vague and setting one up is not trivial.  I'm going to discuss the steps needed to get a private gallery up and running.

Caveat

First a word of warning.  VS supports extensions and the public extension gallery.  VS also supports having multiple galleries configured (and usable) at once.  Actual support for a private gallery is "Beta" at best.  Therefore don't expect a lot of support from Microsoft in setting up and running a private gallery.  You've been warned.

Gallery Basics

Before setting up a gallery it is useful to understand what it actually is.  Ultimately a gallery is nothing more than a list of extensions (not necessarily a VS extension) with any corresponding metadata that can be installed into VS.  All a gallery is really responsible for is providing a list of available extensions and sending the extensions to VS when needed.  Therefore a private gallery really doesn't do a lot.  A gallery uses an Atom feed to send the list of available extensions to VS.  VS then displays the list in the UI.  When the user selects an extension VS calls back to the gallery and request the extension.  The gallery sends the extension to VS and VS takes it from there.  Pretty simple stuff.

Atom Feed

The feed (atom.html) provides quite a bit of information to VS - the extension name, version, unique ID, author, description, create date and update date.  From this VS can generate everything it needs.  Each extension only has one entry in the feed.  When VS pulls the feed it compares the current version information to what is installed on the user's machine.  If there is a difference then VS adds the extension to the list of updates that are available. 

Unfortunately generating the Atom feed is the hard part of hosting a private gallery.  Not only is it ugly but it gets bigger as you add new extensions.  During the betas and RC Microsoft had a tool to generate this for you but it was pulled before the release. You'll still see it documented but it is gone. Thus you'll have to either manage the file by hand or use some other tool to build it.

Here's what a basic feed might look like.

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title type="text">My Extension Gallery</title>
  <id>uuid:6df6542f-d22f-4138-90dd-a386fa370e64;id=1</id>
  <updated>2013-01-11T23:36:04Z</updated>
  <entry>
    <id>uuid:6df6542f-d22f-4138-90dd-a386fa370e64;id=2</id>
    <title type="text">My Extension</title>
    <summary type="text">Provides something useful.</summary>
    <published>2013-01-11T11:51:56-06:00</published>
    <updated>2013-01-11T11:08:09-06:00</updated>
    <author>
      <name>Me</name>
    </author>
    <category term="Templates" />
    <content type="octet/stream" src="galleryurl/Templates/MyExtension.vsix" />
    <Vsix xmlns="http://schemas.microsoft.com/developer/vsx-syndication-schema/2010">
      <Id>MyExtension.VSIX.ID</Id>
      <Version>1.0</Version>
    </Vsix>
  </entry>
</feed>

Let's break down the elements.

  • Title - The title of your gallery. 
  • Id - A unique ID for the feed. 
  • Updated - This should be updated each time the feed changes so VS will refresh its view.  I have not confirmed this but it is reasonable to assume that VS will look at the value and not bother walking through the extensions if it hasn't changed since the last time the feed was read.  Note the date/time values are in UTC format.
Each extension has an <entry> element within the feed. 
  • Id - The unique ID of the entry.  This should generally just be the VSIX unique ID.
  • Title - What the user's sees when seeing the extension. 
  • Summary - The nice description that shows up below the extension. 
  • Published - The date/time when the extension was added. 
  • Updated - The date/time when the extension was last updated.
  • Author - The author of the extension.
  • Category - An optional element that specifies a subfolder the extension should appear under in the gallery UI.  This is useful for grouping extensions.
  • Content - Specifies the type of extension. 
    • type - octet/stream indicates a binary file.
    • src - This is the full URL to the extension.  If an extension appears in the gallery but cannot be downloaded then this attribute is most likely wrong.
  • Vsix - This starts the metadata for the VSIX itself.
    • id - The unique ID of the VSIX package.  This should generally match the ID of the entry.
    • version - The version of the extension.  This should match the version of the VSIX extension.

It is generally assumed that an extension is shipped as a VSIX but it could also be an MSI. Beyond the source URL not much changes in the gallery.

Hosting a Gallery in IIS

To host a gallery all that needs to happen is that a listener on a URL needs to return back the Atom feed when requested.  It also needs to return the files when they are requested by VS.  This can be done rather easily using IIS.  I use the existing TFS web server to help keep things consolidated but you can use any web server that is available.

  1. Create a new web site (or application) in IIS (ex. MyGallery).
  2. Optionally configure it to use a custom app pool.
  3. The site does not .NET or anything so the only additional configuration is to set the atom.html file as the default document.  This should cause a basic web.config file to be generated.
  4. Add the atom.html file to the root of the site following the format given earlier.
  5. Browse to the site and you should see the RSS feed. 

    Note: In IE 10 I was unable to view the feed even though it was working.  I used FireFox to confirm it was working properly.

If the feed does not appear then the site is not configured properly.  Verify the security on the folder is correct and the site and app pool are running.  On newer versions of IIS the app pool will run under ApplicationPoolIdentity which makes it more difficult to configure.  Consider using a fixed account.  Use the Basic Settings in IIS to test the connection for the path.

Now you can add new extensions to the site.  Ideally extensions should be placed in a subfolder under the site.  Remember to account for the folder in the content src attribute.  When adding new extensions or updating an extension one be sure to update the <updated> element of the feed and the same element for the extension (if it is an update). 

Configuring Visual Studio

Configuring VS to use the new gallery is really simple.

  1. Go to the Options dialog (Tools\Options).
  2. Go to the Extensions options (Environment\Extensions and Updates)
  3. Click Add
  4. Specify a name for the gallery (ex. Company).
  5. Specify the URL to the gallery (the site root where the Atom feed is).
  6. Click Apply and then OK.
  7. Go to the extension gallery (Tools\Extensions and Updates).
  8. Expand the Online node and the new gallery should be listed.
  9. Click the gallery and the extensions (and categories) should be shown.
  10. To test the feed select one of the extensions and click Download. 

    Note: A 404 error indicates either the source URL is wrong or there is a permissions problem.

Hosting a Gallery in IIS (Part 2)

The above process works but you'll likely run into several problems.

  • IIS is picky about serving up files that end with VSIX or MSI.
  • Maintaining the Atom feed is a nightmare.

Fortunately there is a better option right now.  Jakob Ehn has written a simple MVC application for hosting an extensions gallery.  He blogged about it here.  The source is available on CodePlex.  This application is still a beta but, other than a few issues mentioned below, it seems to work pretty good.  To use it you'll do the following.

  1. Download the setup program.
  2. Run the installer on the IIS machine.

    Note: The installer must run with admin privileges so you'll need to start it from an elevated Explorer window or command prompt.
    1. On the first installer options page specify the web site and app pool to use (ex. MyGallery).
    2. On the second installer options page specify the path to the files on the disk.  The default is probably fine.
    3. Click Install.
  3. Unfortunately it is still not completely configured so open the web.config and make the following changes.
    1. Change FeedTitle to the display name of the gallery (ex. My Gallery).
    2. Change BaseURI to the base URI of the site (ex. http://myserver/MyGallery/Gallery/Extension/).  The "/Gallery/Extension/" is required.
    3. Change VSIXAbsolutePath to the file system path where the extensions will be stored.  Personally I don't like to store data files in the website directory so I use a separate path but be sure that the app pool identity has permissions to read the directory.
    4. Save the file.
  4. The application was written using MVC 4 but the installer does not install the required binaries.  Either ensure that MVC 4 is installed and configured for your web server or manually copy the MVC 4 binaries to the site.
  5. Browse to the site to confirm the feed is working.

To add or update extensions simply drop the files into the configured VSIXAbsolutePath directory.  If you want to use a category then create a subfolder.  From what I can tell VS only allows a single level of categories so do not try to nest the folders beyond a single level.  Behind the scenes the application will generate the Atom feed whenever it is requested via the URL.  To generate the feed it enumerates the files in the directory.  For each VSIX it generates an entry.  The feed is regenerated each time it is requested so the update date will always change.  For each entry the published and update date come from the file's create and last updated values.  The remainder of the information is determined by the VSIX file itself.  If the extension is in a subfolder of the root then a category entry is added for it.  Now you can use VS to browse the gallery.

As of 13 January 2013 there is a bug in the installed code that prevents download from working.  When a request is sent to the application to download the file it attempts to open the file for read/write access.  Unless the app pool has permissions (generally not a good thing) then it will fail and a "Not Accepted" error will be generated.  The fix has already been made in the code but the installer has not been updated.  Therefore you can download the source and rebuild it.  Then just swap out the binaries.

The application is still in beta so be prepared for some issues.  Here's a list of things I would like to see changed.

  1. The installer needs to be updated to contain the file download bug.
  2. The MVC 4 binaries should be installed to the application directory if they do not yet exist.
  3. The installer should prompt for the settings so we don't have to manually edit the config file.
  4. It is probably not a good idea to regenerate the feed each time VS requests it.  As more files are added this will get really slow.  Ideally a file system watcher should be used to detect changes in the files and the feed regenerated only when changes are detected.
  5. Currently only VSIX files are supported.  MSI-based extensions are not.
  6. The ID of the entries are based upon the ID of the gallery rather than the ID of the VSIX.  Ideally they should match the VSIX of the entry.
  7. The publish/update date of the files are not good indicators of the extension lifetime.  Ideally the publish date should be set the first time the entry is added to the feed and should never be changed.  This would require that the feed be updated and never recreated (as it is now).
  8. Perhaps as an alternative extension metadata could be managed in a database (like NuGet) while the files are stored on disk.  This eliminates the need for watching the file system and updating the feed.

None of these are showstoppers unless you are dealing with a lot of extensions.  Overall the Inmeta application makes hosting a private gallery a lot easier.

Summary

As VS extensions become more popular it is inevitable that you will want to write and publish your own.  But some of them will be for personal or company use and should not be published on the public gallery.  Hosting a private gallery is relatively painless to do and allows for private extensions.  VS fully supports this concept.  MSDN provides more information on how to use a private gallery.  Hopefully MS will eventually release some tools to make creating and managing private galleries easier.  Until then Jakob's project will fit the bill.

Setting Up a Private NuGet Repository

These days most developers use NuGet to reference third-party dependencies like EntityFramework, Enterprise Library and NInject.  A full discussion of NuGet is beyond this post.  There are lots of articles on what it is, how to use it and how to package and publish to it so I'll ignore all these things.  But what about internal dependencies like shared libraries used by company applications?  It turns out you can host your own private NuGet repository as well.  Visual Studio is already set up to support this.  The hard part is getting the repository set up.  At my company we've been running a local NuGet repository for a year and it works great, for the most part.  We have a company policy that all dependencies must be stored in NuGet.  This pretty much eliminates versioning issues, sharing code across projects and storing binaries in source control.  In this post I'm going to go over the steps to set up a private NuGet repository.  Others have posted on this topic and there is some documentation available but unfortunately it is out of date.

Note: These instructions are based upon the NuGet Gallery source as of 6 Jan 2013.

Prerequisites

Before you can get NuGet up and running you'll need some prerequisite software.

For compiling NuGet:

For hosting NuGet (using the preferred approach):

  • IIS to host the web site
  • SQL Server to store the site metadata
  • File system to store packages
  • URL Rewrite 2.0

Compiling the NuGet Gallery

Here's the steps for compiling the NuGet source code.

  1. Download the source from here
  2. Extract the files to your local drive.
  3. At this point you can build the source via PowerShell and the .ps1 file but I've never had any luck with it.  So I follow the next set of steps instead.
  4. Open the solution in Visual Studio.
  5. Build the project.

At this point you have the NuGet website (the Website project) compiled and ready to go.  Note that NuGet is configured to use HTTPS for authentication.  For your internal repository you might not want this.  Unfortunately to disable it you have to modify the code.  Find all references to the attribute 'RequireRemoteHttps' and comment them out.

Deploy NuGet to IIS

  1. Copy the website to the target directory on IIS (for our purposes we assume \wwwroot\NuGet).  For the current source this includes the following files and directories:
    1. \Api
    2. \App_Code
    3. \App_Data
    4. \Bin
    5. \Content
    6. \DataServices\*.svc
    7. \DynamicData
    8. \Errors
    9. \Infrastructure\*.sql
    10. \Scripts
    11. \Views
    12. All files in root other than: *.cs, *.ruleset, packages.config, *.resx, *.tt, *.csproj
  2. If not already done create a new IIS website backing the target directory.
    1. It is strongly recommended that you create a new app pool for NuGet and use a custom account.  Note that the account will need access to the SQL database so a network account is best.  Where I work we use the same account as our TFS services.
    2. The app pool will use .NET 4.0 with an Integrated pipeline
  3. Ensure that the app pool has read/write access to the App_Data/Lucene folder

Configure NuGet

NuGet is now deployed but it is not configured yet.  Open the web.config for the site and change the following settings.

  1. <appSettings>
    1. Gallery:SiteRoot - Specify the full URL to the NuGet gallery site.  This will be used by the site when creating links.
  2. <connectionStrings>
    1. NuGetGallery - Specify the connection string to connect to the SQL database that will house the NuGet metadata.  If the NuGet app pool is configured to use a custom account then integrated security can be used otherwise specify the account to use when connecting to SQL.

Configure SQL Server

NuGet will automatically initialize the database when it first connects.  However the database must exist so if you do not yet have a database for NuGet then create one now. 

  1. Open SQL Server Management Studio.
  2. Connect to the SQL server that will host the database.  Where I work we use the existing TFS server's database because we are not using the instance of SQL provided by TFS.  If you are using the SQL instance that ships as part of TFS then refer to the licensing restrictions of TFS before doing this.
  3. Create a new database for NuGet.  The default is NuGetGallery.
  4. Ensure that the account that NuGet will run under has full permissions to the database.

Start NuGet

If all is well you should now be able to browse to the NuGet site.  Note that the first time it starts it will be slow as it updates the database.  If any errors occur or the site refuses to start then disable custom errors in the web.config and resolve the errors.  When I initially tried to get the site running under Windows 8 it complained the web.config was invalid.  I narrowed down the error to the <rewrite> section.  Removing that section allowed the site to load.  A quick check revealed that the URL Rewrite 2.0 module for IIS (which NuGet needs) is not part of IIS that ships with Windows 8.  Using the Web Platform Installer I installed it and everything worked correctly.

Set Up E-mail

NuGet uses Forms authentication for authenticating users.  When creating a new user account NuGet will, by default, send an e-mail to the user to confirm their e-mail address.  You will want to set up NuGet to use your company e-mail server so that you can get this e-mail, and any other e-mails that are sent over time.  Otherwise creating a new user requires that you manually get and forward the generated e-mail. 

NuGet does not expose any UI to update the e-mail information and it is stored in the database.  Open the GallerySettings table in SSMS and set the following columns.

  • SmtpPort - The SMTP port of your e-mail server.  Normally this is 25.
  • SmtpHost - The SMTP host address.
  • SmtpUsername - The user name for authentication, if any.
  • SmtpPassword - The password for authentication, if any.
  • UseSmtp - Set to 1 to enable e-mail
  • ConfirmEmailAddresses - Set this to 0 to disable confirmation e-mail addresses.

By default the e-mail connection will use SSL. If you do not want SSL for your e-mail then you'll have to modify the source code again.  Ultimately the setting should be stored in the database with the rest of the SMTP settings.  But to keep it simple just do the following.

  1. Edit the ContainerBindings file.
  2. Find the initialization code for IMailSender.
  3. Set EnableSsl to false.
  4. Rebuild the code.

Package Storage

NuGet uses the database to store package metadata but, by default, it stores the packages themselves on disk.  The appSetting Gallery:PackageStoreType specifies the provider to use.  For performance reasons you will likely want to continue using the file system to store packages but the default directory is not the best choice when you need to upgrade the site.  Instead you should consider storing the files in a more reasonable path.  Be sure that the site has read/write permissions to the new package directory.  To set the file path where packages are stored add/update the appSetting Gallery:FileStorageDirectory.  Specify the full path to where packages will be stored.

If you do not want to use the file system to store packages then you can use a different package store by setting the appSetting Gallery:PackageStoreType.  Currently only Windows Azure is the alternative.  You can technically add your own and update the corresponding enum and config setting to use a different store - such as a database.

Registering

At this point you should be able to access the site.  You will need to register an account so that you can add packages.  Technically all users will need an account.  You will also want to assign yourself as an administrator for the site.

  1. Open the Users table in SSMS.
  2. Get the Key for the user you want to make an administrator.
  3. Add a new row to the UserRoles table where the user matches the key you have and the role key is for administrators, 1 is currently the value.

When creating an account NuGet will generate an e-mail that you need to click on unless you've disabled confirmation e-mails.  Unless you've set up an e-mail server you'll need to click on a link inside this e-mail from the account it was sent to.  If you're using a corporate e-mail client like Outlook it shouldn't be too hard to get the e-mail contents such that you can click on it.  For everybody else it is just easier to open the Users table in the database and copy the UnconfirmedEmailAddress value to EmailAddress. Then you can log in successfully.

Currently NuGet uses Forms authentication.  For a private gallery it might be more useful to use Windows authentication.  Unfortunately that requires adding code to the site.  NuGet relies on user accounts for managing packages so some updates will need to be made to handle Windows authentication.  That is left as an exercise for the user.

Connecting to Visual Studio

From this point on the private repository works just like the public repository.  You can upload and browse packages.  A word of warning - once a package is uploaded it is forever part of NuGet.  Nevertheless you should upload a package to make sure everything is in working order.  Once you've uploaded a package you can configure Visual Studio to use it.

  1. Open the Package Manager settings (Tools\Library Package Manager\Package Manager Settings or Tools\Options -> Package Manager).
  2. Go to the Package Sources section.
  3. Click the plus button to add a new source.
  4. Specify a name (ex. Company)
  5. Specify the source (http://<server>/<site>/api/v2).
  6. Click Update.
  7. Click OK to close the dialog.
  8. Open the Manage NuGet Packages dialog (I generally do so from a solution).
  9. Under Online select the new repository.
  10. The uploaded package should appear.
  11. Click Install to confirm the package can be downloaded and installed correctly.

Ownership

NuGet associates one or more owners with every package.  The owner of a package can upload new versions.  For a public repository this is fine but for a private repository it is likely that everyone should be able to update packages.  To add or remove owner(s) from a package do the following.

  1. Open the package on the NuGet site.
  2. Select Manage Owners.  This is only available for owners and adminstrators.
  3. Add or remove owners from the package.

Local Repository

One issue with NuGet is that once it is uploaded it will be forever available.  NuGet does not support deletion.  It is generally a good idea to verify that a NuGet package is working correctly before uploading it but this becomes a catch-22.  You cannot test the package unless you can download it from the repository.  You don't want to upload the package until it works.  Fortunately there is an easy solution, a local repository.

A local repository is useful for testing packages.  Ultimately it is nothing more than a directory where NuGet can retrieve packages from.  Depending upon your need you might have a local repository on each developer's machine or you might set up a file share where all devs can point to.  Either way you point the Package Manager to the directory and it'll work just like a regular repository.  To get packages into the repository you just drop the .nupkg file into the folder.  You can then install/upgrade the packages through Package Manager in Visual Studio.  Note that packages always have a full version in the filename so it is easy to tell whether there are new versions available.  Once the testing is complete you can delete the packages.

To configure Visual Studio to use a local repository do the following.

  1. Open the Package Manager settings (Tools\Library Package Manager\Package Manager Settings or Tools\Options -> Package Manager).
  2. Go to the Package Sources section.
  3. Click the plus button to add a new source.
  4. Specify a name (ex. Local)
  5. Specify the source path (ex C:\MyTestNuGetGallery).
  6. Click Update.
  7. Click OK to close the dialog.
  8. Open the Manage NuGet Packages dialog (I generally do so from a solution).
  9. Under Online select the new repository.
  10. The uploaded package should appear.
  11. Click Install to confirm the package can be downloaded and installed correctly.
MVC Property Validation Gotcha

Recently I was working on an MVC 4 application and I ran into a couple of issues with how property validation works with the default binder.  I thought it was useful enough to share.

Conditionally Required Fields

The first issue I ran into was conditionally requiring a field based upon some other field.  In my case it was for collection payment information.  On my site I can accept either credit cards or checks.  For credit cards I need to collect one set of data.  For checks a different set.  The user selects the payment method and the appropriate fields are shown.  The model to support this exposes a simple boolean property called IsCreditCard.  Now comes the fun part, marking the fields as required.  Because of how we render our required fields on the client I needed to apply the RequiredAttribute on the fields to get the client-side behavior correct.  But then all the fields would be required irrelevant of payment method.  Therefore I created a new validation attribute deriving from RequiredAttribute called RequiredIfAttribute.  This attribute allows me to specify a property to check to see if the property is actually required.  If the property is true then the standard require validation is performed. 

I dropped this attribute onto the appropriate fields and then ran the code.  To my surprise none of the fields were required.  Setting up a breakpoint I refreshed the page and suddenly all the fields were required.  What in the world?  I stopped the debugger and restarted it and then ran my test again.  This time when the breakpoint was hit I noticed my IsCreditCard property was false which wasn't right.  Then it hit me.  MVC runs the property's validators right after it sets the property's value.  In this case the IsCreditCard property happens to be lower than the field I was validating so the conditional property hadn't been set yet.  Hence the first time I ran it the value was false.  But after the first refresh the property was set to true, at least until its value gets updated by the binder. 

The solution to this was simple.  I moved the conditional property, IsCreditCard, to the top of the model so it would be set before any properties that rely on its value.  The moral is that validators run when the property is set.  Note that I could have implemented IValidatableObject but then I couldn't use the RequiredAttribute which I needed for the client-side behavior.

Cross-Property Validation

Another issue I had was with cross-property validation.  In this case the user must enter one of two fields.  They cannot be marked as required because only one technically is.  I thought that the new RequiredIfAttribute I created might help but since property's are set and then validated it would only work for one.  A couple of solutions came to mind.

The first solution would be to ignore the first property (first being in model order) and use RequiredIfAttribute on the second property to validate them both.  But this would only allow me to set an error on the second property which wasn't good enough.  For this view if both fields are empty then both must be flagged as in error.

The second solution was to implement IValidatableObject.  This interface is designed to allow a model to validate itself once all properties are set.  It has a single Validate method that returns the list of validation errors as ValidationResult. For cross-property validation this is the way to go but there is a problem.  With the data annotations an error will get associated with the corresponding property.  For our views this causes the field to be marked in a special way to the user.  With the interface that becomes harder. 

Fortunately ValidationResult allows you to specify the member(s) to apply the error to.  Unfortunately it isn't obvious how to get it to work right.  The constructor for type allows you to pass a list of member names.  Each member is then associated with whatever error you want.  For cross-property validation it would make sense to pass both properties to the constructor.  This results in both fields being marked as errors but also causes two error messages in any validation summary you are using. 

The correct solution to this is to generate multiple errors.  The first error contains the error message and (optionally) the field to associate it with.  Any related fields are added as additional ValidationResult entries but with no error message.  The result is that you get one error in the validation summary and multiple fields marked as errors.

Summary

The property validation that MVC does is a lot nicer than the ASP.NET validators we use to have but there are some gaps in its behavior.  It is truly designed for per-property validation.  For model validation you have to add the IValidatableObject interface.  This allows for more flexibility at the cost of complexity.  It seems like most errors would be associated with a single field so I don't understand why the ValidationResult constructor doesn't expose an override that allows specifying an error message and a single field name.  Beyond this quirk, and if you understand the validation the model binder does, then validating your models is pretty easy. 

I haven't tried it but it would be nice if we could apply validation attributes to an entire model rather than implementing IValidatableObject.  This would gives us the declarativeness of attributes with the ability to validate the entire model.  I could foresee a lot of power annotations being created if we could do model-level validation via attributes.  Maybe in a future version of MVC...

Posted Sun, Oct 21 2012 by Michael Taylor | no comments
Filed under:
Orchestrator Pattern

I'm finishing up my company's second rewrite of an ASP.NET application to MVC and Razor.  One of the things we learned along the way is that controllers and views are no more testable than forms and code behinds in ASP.NET.  So we have been using the orchestrator pattern for newer code.  I'd like to share our thought process on this approach.

Orchestrator Pattern

First a definition of the orchestrator pattern.  The pattern has been discussed by several different people and mentioned in at least one book but my definition of an orchestrator is something that manages the flow of data from one point to another.  Here's some links on the topic:

For MVC an orchestrator is responsible for getting data from the back end and sending it to a view for display.  The orchestrator is also responsible for getting the data from the view and passing it to the back end for validation and saving.  A typical flow in this type of application would look like this:

  1. ASP.NET calls an action on a controller
  2. Controller collects any needed data (generally the parameters to the action) and calls a method on the orchestrator
  3. Orchestrator validates the data and passes it along to the service layer for processing
  4. Service layer performs whatever action is necessary to meet the request and returns a result
  5. Orchestrator does any post-request processing on the result and returns a (potentially updated) result
  6. Controller uses result to return appropriate action result to ASP.NET runtime 

Testability

A controller in MVC is really an orchestrator but unfortunately it is too tightly coupled with the runtime to be able to focus solely on data flow.  One could argue whether a controller is testable or not.  Yes there are folks who test it and yes there are ways to make it testable but is the work involved really worth it?  I don't believe so.  Let the controller do what it is designed to do best - handle routing requests from the runtime and return the results back to the runtime.  This allows the controller to make full use of the MVC infrastructure to get data from the route and views, cache data where needed and respond to requests without having to introduce any unneeded infrastructure such as an IoC. 

One of my fundamental rules in regard to unit testing is that I should be able to architect an application with the best design possible without regard for hooks to make unit testing possible.  What this means is that I'm not making every method of every type virtual so it can be mocked.  I'm not exposing everything as an abstraction so I can replace it in a unit test.  This type of mentality produces very loosely coupled code that is very easy to test, hard to understand, slower than necessary and ultimately harder to maintain.  Of course if the design mandates a loosely coupled situation then by all means use it but never just to make unit testing easier. 

I've already mentioned before that unit testing as it stands today is not where we need it to be.  I am confident that testing frameworks will step up to the plate and allow us to design applications without regard for unit testing and frameworks will just work.  I think we're getting closer to this goal.  With the addition of Fakes to Visual Studio 2012 we can now stop worrying about things that aren't testable (static methods of static classes) because the testing framework can still circumvent it.  This is just one example of designing without testing in mind.

Going back to the orchestrator pattern, when I test the data flow from the view to the back end I don't want to concern myself with how the data got there.  It just did.  Whether that was a controller and ASP.NET or the test framework shouldn't matter to the orchestrator.  It just works.  By eliminating the need to test a controller I can keep focused on the important aspects, the orchestrator.

Controllers

The controller's job is to collect the data from the ASP.NET runtime and pass it on to the orchestrator for processing.  When the orchestrator is done it returns back data that the controller then passes on to the view for rendering.  The controller is managing the data exchange to and from ASP.NET while the orchestrator is doing the actual work.  If we need to pull in extra data that is not accessible from the view (ie. user context or perhaps configuration settings) then the controller is responsible for doing that as well. 

Currently we have found that having a single orchestrator for each controller works well.  Not every action on the controller needs a corresponding orchestrator method but we are seeing a pattern where this is generally true.   Each action basically becomes boilerplate code where data from the view (generally the action methods) is passed to a method on the orchestrator.  The results of the orchestrator are then passed on to the appropriate view.  The controller only needs additional code when the action requires extra work.  For example if an action can only be called by certain users then an attribute or some code in the action will enforce that.  The orchestrator is not responsible for that aspect. 

This nicely corresponding to code that is generally unit testable and that which is not.  Unit testing that an action is getting the right data generally doesn't make sense as we rely on the framework to do that.  The process however is testable.  Correspondingly we put code that either doesn't need to be unit tested or can't easily be unit tested in the controller and the rest moves to the orchestrator.  Orchestrators are ASP.NET agnostic so all data needed by a method must be passed to it.  We have set up a base controller class such that the controller creates its orchestrator the first time an action method needs it.  When the controller goes out of scope so does the orchestrator.  Most of our action methods are 2-3 lines long and none of them require unit testing.  Any actions longer than that force us to take a closer look to see where we can refactor the code.

The controllers reside in the main MVC application.  This project is considered not testable so nothing goes in it that we want to test.

Orchestrator

The orchestrator is really where the bulk of the controller work is handled.  Since the orchestrator does not have any connections to ASP.NET we can more easily test it.  But even the orchestrator is generally pretty simple because it is mainly responsible for invoking the service layer where the real work is done.  In a traditional MVC application the controller will call the service layer directly.  But there is often some work that has to happen before or after a service call.  For example service layers generally work with domain objects while the controller works with view models.  Where does this translation take place?  In a traditional application the controller would need to do that.  With the orchestrator pattern it can be moved to the orchestration layer where we can also test the translation.  Some models are easy to translate but some require more work (ie.concatenating an address together to form an easily displayable string).

For the projects I'm working on we start with all the code in an orchestrator method and move code that cannot be easily tested (anything related to ASP.NET for example) to the controller.  Things that can't be tested generally show up easily as we are writing our tests.  The orchestrators reside in an orchestration assembly.  The orchestration assembly is used by the main MVC application and has a dependency on the service layer.  This keeps it pretty clean.

View Models

Different companies have different opinions about what role view models play in MVC.  Some companies believe they should only be used for display while others use them as domain objects.  As a result some companies put the view models in the MVC application while others put them in the business layer.   There are advantages and disadvantages to both approaches and I don't believe either are wrong.  I believe in keeping models as close to where they are needed as possible. 

For the orchestrator pattern I think it makes sense for the orchestrator to work with view models.  The translation between view and domain models is testable so the view models must reside in a place that can be tested.  Therefore our view models reside in the orchestration assembly so we can test any complex mappings we may have.  We're not sure if this is truly the best approach yet but it is working well for the projects we're using it with.  The orchestrator is responsible for getting the domain objects from the view models it receives, calling the service layer with the domain objects and then translating the returned objects back into view models. 

The view models are dumb collections of properties with no business logic.  Only the properties needed for a specific view are provided so they stay small.  We add some formatted properties to simplify our view code but that is about it.  The hardest part is translation to and from the domain object.  Domain objects reside in the service layer and the service layer has no knowledge of the orchestration assembly.  For translation from a view model to a domain object we either expose a method off the view model directly (ie. ToDomainObject) or we create an extension method off the domain object to accept a view model (ie. FromViewModel).  One could argue that this is mixing concerns but not everything is cut and dry in software.  I tend to lean toward the extension method route but then you end up with lots of one method extension methods (one for each domain object). 

For translation from a domain object to a view model generally a constructor (on the view model) is used.  Again this is mixing concerns but it is far easier than creating an extension method and keeps the translation with the code that actually relies on it.  The downside to this approach is that the MVC application will now have to have a reference to the domain objects for compilation.  If this is a concern then extension methods might be the better route.

Service Layer

The service layer is a traditional service layer where data is validated, business rules are enforced, processes are run and changes are made.  The service layer remains agnostic of the orchestration assembly and deals only with domain objects.  The service layer is shared by many orchestrators.  One of our guidelines is to add functionality to the orchestrator first.  If more than one orchestrator needs similar functionality then we "promote" it to the service layer.  This prevents our service layer from becoming a hodgepodge of one-off methods needed by only one orchestrator.

Summary

So far our use of the orchestrator pattern has been really successful.  We might not be following a true MVC or OOP architecture but our code is solid, testable and easy to maintain.  New developers can be given a quick overview of the pattern and they can easily grasp the concepts and start working.  If a bug appears in the application we generally start with the orchestrator.  If we find a problem with getting the right data we know it is a controller/ASP.NET issue.  Otherwise we head toward the service layer.  Overall we are happy with the pattern and will continue to use it going forward.  I encourage you to check out the pattern if you have not yet used it.  Especially if you are struggling to get your MVC apps tested properly.

 

Posted Sun, Oct 7 2012 by Michael Taylor | no comments
Filed under:
Framework Compatibility Going Downhill

UPDATE: My post triggered an e-mail exchange with Rowan Miller from the Entity Framework group.  He wanted to explain and clarify some of my comments around EF.  I've added them in the appropriate sections below.  Thanks for the clarifications Rowan!

I understand the need for making some breaking changes to the framework as we transition to newer versions but one of the big selling features of .NET has been writing apps without having to worry too much about having the exact same version on end machines.  Microsoft seems to be moving away from this goal with newer releases.  .NET 4.5, as a framework, has a lot of nice, new features.  But as the next version of the framework I think it fails on many levels.  I'm fearful that this is becoming the new trend at Microsoft given the recent releases of .NET and other libraries.

Framework Versioning

While we are on the 7th framework version there have been only 3 CLR versions (currently v4).  Both 4.0 and 4.5 run on the same CLR and are technically the same framework.  This means that if you install 4.5 then all your 4.0 apps are now using the same 4.5 framework even though the code started with 4.0.  4.5 will overwrite the 4.0 binaries because they do not run side-by-side.  This is in contrast to the frameworks based upon CLR 2 (2.0, 3.0 and 3.5).  CLR 2 and CLR 4 can be run side-by-side.  This is really not much different than what we're used to from native code. 

For this to work well there must be a rule that states that an application written for CLR 4 must continue to work on 4.5 without change.  That is not really the case.  For reasons unknown to anyone but the folks at Microsoft there were changes made to the core libraries that break existing 4.0 applications.  The most blatant is applications based upon Reflection (ie. JustDebug or Reflector).  These applications now throw exceptions because of breaking changes made to the CLR 4 in what should have been an update to the CLR.  Normally you would just not update your application until you've had a chance to fix the changes that were made but because 4.5 is the same CLR as 4.0 you can't.  4.0 apps are just broken and you can't do anything about it except fix the code.  This clearly violates the rule of compatibility.

What MS should have done is held off on making these changes until the next CLR.  If the changes couldn't wait then 4.5 should have been 5.0 and we would have gotten a new CLR.  Then CLR 4 apps would continue to work.  Why did MS decided to break the code now?  The answers I keep seeing always come back to WinRT compatibility.  In order to force everybody to use Win8 and Metro MS is going full bore with WinRT.  WinRT is not the full framework (it reminds me a lot of the Compact Framework) so changes to the core library were necessary.  These changes should have been made in the next CLR version so existing apps continue to work but they weren't.  The result is some apps just won't work correctly once a user installs the 4.5 framework so expect a lot of tech support calls if your app falls into this category.

Namespaces

Another area where compatibility fails is with moving types.  There are several types that were moved from one assembly and namespace to another (ie. some data annotations from EF).  This is not necessarily a bad thing since it is important to move commonly needed functionality into the framework.  The problem is that it should silently work for existing code and not require changes unless we recompile.  But guess what, you have to recompile your code.  We ran across this recently when we upgraded from EF 4 to EF 5 and some of our core libraries weren't ready.  I thought one of the attributes added back in 2.0, ForwardTypeToAssembly or something like that, was designed to solve this issue.  Guess not.  The framework should be able to forward the request for a type from one assembly and namespace to another so existing apps continue to work.  This would be a runtime feature and wouldn't fix compilation errors.  Then a type can be moved and we can recompile when we're ready to switch frameworks.  Right now you run into problems if you try to use an assembly built against 4.5 with an application using 4.0. 

UPDATE: The data annotations issue comes from us taking some annotation that we’re shipped out-of-band in EF 4.1 (in our NuGet package) and moving them into the .NET Framework. After discussing this with the team that owns the data annotations assembly we decided that having them in the root namespace wasn't the right thing to do as they were very database specific. That left us with the decision of leaving them out of the framework or taking the namespace change. Regarding compatibility, we were very careful that already compiled code will continue to run, the only time you need to address the namespace change is if you upgrade to EF5.

Assembly version X (or version Y, depending)

Yet another area where MS is making versioning worse is with Entity Framework 5.  When you add a reference to EF 5 you are actually using either EF 5 or 4.x depending upon what framework you are targeting.  Let's think through this for a second.  You install EF 5 but you look at the assembly references and it is 4.x.  You figure something is wrong so you google and find it is by design.  At some later point you upgrade your application to 4.5 and think you're using EF 5 but you're still using 4.x.  That is unless you remember to remove your references to EF and then install them again.   What in the world is going on in Redmond?  Who thought this made any sense and would solve any problems?  I heard that the reason it was done this way was to avoid having to generate 2 NuGet packages.  What??  You just made it harder to know what version of a library someone is using just to avoid creating a NuGet package once!!!  Seriously how hard is it to create 2 different NuGet packages.  I do it all the time.  If EF 5 doesn't support .NET 4.0 then don't advertise that it does and swap assemblies behind the scenes. You've just made it harder on everybody going forward.

UPDATE: Regarding the 4.x vs. 5.x version number. We have to have separate assemblies because there are different features that light up on .NET 4.5. For example, you can’t use the enum support in EF5 when using .NET 4.0. Both assemblies are EF5, but we decided to have different assembly versions so that the CLR will treat them as different – important because they have different API surface in them. The assembly informational version, file version etc. are all 5. (Response: This is different than what I've been hearing about the differences between 4.0 and 4.5 versions of EF.  It does appear they are more different than the same which pushes me further down the path that they shouldn't have been released this way.  It seems like it'll introduce compatibility issues when making unrelated EF changes - aka framework upgrades).

Future

I can't say what the future holds for the framework but if it continues down the path it is going now I foresee we'll be accusing it of versioning nightmares just like we had back in the DLL days.  The rules of versioning isn't really that hard so I fail to see why the teams at MS are struggling with it.  In case you need a refresher here are the versioning rules:

  • A minor update (ie 4.0 and 4.5) guarantees that an application behaves the same whether the update is installed or not (although bug fixes and any additions added in the update will not exist). 
  • An application can explicitly target the update version if needed.
  • A breaking change that requires rebuilding an application is always a major update (ie 2.0 to 4.0).
  • An application built against a major version will continue to work unchanged against an updated version.
  • Installing version X of a product ensures that version X is being used.  If version X is not supported on the targeted platform/version then it won't install.
  • For libraries that have breaking changes multiple versions will be shipped to support the various versions the library needs to support.

I think at this point Microsoft should just stop the minor update process, except for pure bug fixes, and stick with major version changes only.  They are guaranteed to make at least one breaking change in the minor updates so they are violating the rules.  People are already starting to grumble about it.  How loud does the complaints have to get before Microsoft introduces the "next" solution to DLL hell?  I think the solution already exists.  It is called proper versioning of the framework.

 

 

Posted Sun, Oct 7 2012 by Michael Taylor | no comments
Filed under:
EF Challenges and Ramblings

I've recently been working on a project where I used Entity Framework for the data access.  I've run into some challenges that don't seem to be discussed much in the articles on how to properly use ORM.  I'm curious if anyone else is having these challenges.  While I'm using EF as my base I'm pretty sure that these challenges exist for other ORMs as well including NHibernate so I don't believe the challenges to be caused by a specific ORM implementation but rather as a result of using ORMs in general.

General Architecture

Before discussing the challenges I think it is prudent to explain the rationale of the architecture.  For this project the database was not directly accessible to the application.  The database contained data used by multiple applications so each application had to go through a common data library (CDL) to ensure consistent validation and behavior.  The CDL consisted of 3 layers - domain, data and entity.  The domain layer contained business objects and handled the business rules.  Applications used this layer for interacting with the database.  The data layer was a generic data model that the domain layer used.  Domain types basically wrapped the data model and provided the core functionality of the business.  The entity layer was where EF resided and was nothing more than the configurations needed to get the data model working.  Code first was used but the database schema was managed by another team so it was never necessary to replicate the database schema fully.   

EF was hidden inside the CDL.  Since applications never directly talk to the database it didn't make a lot of sense to expose EF.  An application can freely use whatever ORM it chooses while the CDL can be updated over the life of applications.  There are arguments on both sides as to whether the ORM should be exposed directly to the application.  My opinion is that sometimes it makes sense to expose the ORM directly and other times it makes sense to hide it.  It really depends upon the application that is being built.  Ultimately I don't think the determination as to whether the ORM is directly accessible or not really impacts the challenges I've encountered.  For this project it made sense to hide it.  The unit of work (UoW) pattern was used to manage changes in the system.  Additionally repositories in combination with queries were used to access data.  The query objects worked with the data model directly to retrieve data and the repositories were responsible for creating the domain object wrappers.  One of the benefits of this approach was that the CDL exposed a lot of core functionality but applications could create new queries without really getting involved in how it all came together.

For purposes of this discussion let us assume a simple data model where Employee represents an employee.  Employee has some basic properties like Name, Pay and Manager.  The Manager property is a navigation property that represents the employee's manager.  It is of type Employee.  Employee also has a navigation property called Subordinates which is a collection of Employee instances that report to them.

Implicit vs Explicit Updating

Most ORMs allow you to make changes to the underlying data and have the changes saved to the database when the UoW is saved - implicit updates.  But this can introduce unexpected behavior and complicates validation.  The standard approach is to create a base type that represents the data, such as Employee.  Under the hood the ORM is called to return the data and a proxy (from the ORM) is returned.  The proxy contains the data along with housekeeping stuff such as change tracking, relationships, etc.  The application uses the base type but in reality is working with the proxy.  When a value on the type is changed the underlying proxy marks it as modified.  Eventually when the UoW is saved any modified proxies are also saved.  This introduces a problem though.  If an application makes some changes to an object but doesn't want to persist those changes then there is no way to "undo" the changes.  This is most prevalent when dealing with multiple entities (some of which you may want to save) and when dealing with cancellations.  If the proxy data changes then the data is saved when the UoW is.  This can result in unexpected behavior if an application does a "what if" scenario on an object and then later saves the UoW (perhaps in another component). 

Another problem with implicit updating is validation.  While some validation can be done at property set time, most validation requires evaluation of the entire entity and in many cases this has to be postponed until all properties are set.  In general the order of setting properties shouldn't matter.  So it becomes the responsibility of the application to ensure that an object is valid before the UoW is saved.  For example in the case of Employee an employee cannot have a manager who is a subordinate of the employee.  While Employee could potentially detect this it would require that the properties be set in the "correct" order.  One solution that is often used is to hook the UoW's save method and do validation there but it isn't that elegant.  In the simplest case the save method has validation code in it.  Even with some interface to move the validation to the object itself the UoW still has to validate the object.  A bigger challenge is dealing with the validation errors.  In most cases the UoW is saved in a higher level type after all the work has been done.  The application has to assume validation may fail and provide some sort of generic error handling rather than letting the code that made the changes deal with the validation errors.

An alternative approach is to require an explicit update command from the application - explicit updates.  An application requests the data, makes the modifications and then updates the data.  The update process doesn't actually save the changes but rather does the validation to ensure the save should work.  While it seems a little odd it works better in my opinion.  Even better is that the object is no longer responsible for validation outside its own properties.  In most cases an object is part of a larger collection of entities so the collection is responsible for validation.  Assume for a moment that an employee's name must be unique.  The name (assume first, middle and last name properties) could be set in any order so it clearly cannot be validated by the Employee object.  The validation would need to occur by the type that manages the list of employees.  Without an explicit update method the employee could be invalid.  The validation error would not be detected until the UoW is saved.  Even worse is that there is no way to "undo" the name change to allow the save to complete anyway, such as if the user cancelled the request.  Additionally an application has a better idea of where validation errors will occur so they can code accordingly.  Of course errors can still occur when the UoW is saved but in most cases the explicit update step will catch the errors.

Explicit updates introduce a big challenge for ORMs.  To get explicit updating to work properly the proxy returned from the ORM cannot be modified until the data is ready to be persisted.  Therefore either the proxy data has to be copied or the proxy has to be held while temporary fields are used to store the changes.  When the update method is called by the application the modified data is then flushed to the proxy.  This is more work than needed by implicit updating.  It also introduces maintenance issues. 

Another challenge of explicit updating is lazy loading of data.  Most ORMs allow for lazy loading of navigation properties, such as Manager on the Employee type.  While the arguments for and against are valid assume that lazy loading is valid in some cases.  For these cases the proxy must remain around to get lazy loading to work.  It is imperative that the proxy's navigation property isn't referenced until needed otherwise extra queries occur defeating the purpose of lazy loading.  Explicitly updated objects need special code to deal with lazy loaded properties.  Often this involves checking a private field to see if the data has been loaded and, if not, referencing the proxy property to trigger the lazy loading.  For Employee the domain instance can be lazy loaded when the Manager property is referenced the first time.  The domain instance would be a private field that is checked to determine if it has been loaded yet.

Even more challenging is changing a navigation property.  In most ORMs simply assigning a new value to the property would trigger an add to the database.  To update a navigation property requires the original value being pulled from the database and assigned to the property.  One solution to this problem is using slim objects.

Full vs Slim Objects

There are a couple of different contexts in which data is needed - view and edit.  When viewing data you generally want a full domain object where you are working with objects and can access all their information, such as a manager's name.  When editing data generally only the root object is needed.  The UI generally deals with getting related data through other means, such as drop down lists.  ORMs are great for the view scenario but overkill for edit scenarios.  A slim object is a simple domain object that doesn't have any navigation properties.  Instead it exposes the keys for the related objects.  This drastically reduces the amount of data being pulled from the database just to update an object. 

For example if a web application wanted to view the details of an employee then the manager's name would be convenient to have through a navigation property.  But when editing the employee more likely the managers would be displayed in a dropdown list. The dropdown would be populated by retrieving a list of all employees who could be managers, storing the name as the text and their key as the value.  When editing the employee the manager's key would be needed but the name is useless.  A slim object would contain only the primitive properties that needed to be set and the manager's key.  To update the database the data model's manager key column would be set while the navigation property would be ignored.  In EF this works well because it takes the key column over the navigation property.

There are some issues with slim objects to be aware of.  The first issue is that they would need to do the same sort of validation the full object would need to do, albeit with less information.  Otherwise the slim object, which is where you want validation to be, could be used to circumvent validation.  A simple solution to this is to have the full objects simply wrap the slim objects and the slim objects do the core validation.  But at this point when does a slim object really just become the data model?  This leads to the bigger issue with slim objects, exposing more of the underlying data model than desired.  In my opinion domain objects are really just convenience objects that provide common functionality over the existing data model.  If performance outweighs functionality then it is a reasonable tradeoff.

Surrogate Keys vs Persistent Keys

Persistence ignorance is a common term thrown out when talking about ORMs.  The goal is to have data models be PI but where does the key, as defined by the database, fit?  Is the database key really database specific?  I don't think so.  Entities need some sort of unique identifier.  If we were working with memory objects then the address would probably work but with databases something more permanent is needed.  I don't really see the benefit in trying to hide the database key other than to complicate things, with one exception.  The generation of the key should be PI but its usage should be encouraged in order to simplify working with objects.  As far as the application goes the key is some unique value that is associated with an object.  Whether that came from the database, some key generator or whatever is not relevant and, hence, PI.

So how do we support PI keys while still allowing the data model to work?  A surrogate key is generally used.  A surrogate key is some key generated for an object and somehow mapped to it for its life.  A simple sequentially incrementing number may do or something more elaborate may be needed.  Since the database already has a key why reinvent the wheel?  I think it makes a lot of sense to use the database key when it is available (it also makes detecting new objects easier).  But the problem arises on how to deal with new objects.  Assume that a new employee is added to the system but not yet saved.  The employee needs some sort of surrogate key to identify itself.  Without the key the new employee could not be properly validated to ensure they are assigned to a manager that is not a subordinate (assuming the key is used for detection).  One solution would be to assign sequentially decrementing values to the key until the object is formally added to the database and then changed to the database key.  But doing that means that an object's key can change, at least when it transitions from new to existing in the database.  That may be ok for some applications but it might cause confusing behavior.  Even more dangerous is applications that use the knowledge of how keys are generated to implement some behavior. 

Given the problem with using database keys as unique identifiers it is no surprise that most people use surrogate keys that are completely independent of the database.  But this has its own challenges.  The first challenge is that if the code is being debugged to trace down a problem it can be confusing trying to map the domain object back to the real database object.  Another challenge is the mapping that has to happen between domain objects.  Going back to the Employee slim object, for example, the surrogate key would be used to associate the manager with the employee.  But at some point the surrogate key has to be mapped back to the actual database key, if any.  Some ORMs handle this automatically but it can be difficult to do the mapping while you are debugging your code if you don't know how the ORM does its magic.

Yet another challenge with the surrogate mapping is cross context objects.  It is common in applications to cache commonly used data, such as countries and departments.  The surrogate/database key mappings have to persist for the life of the objects so where is it stored?  Certainly you wouldn't want the ORM to store all the mappings forever as, over time, this would waste memory as objects are loaded and unloaded.  It would also be a bad idea for the ORM to generate new surrogate keys for shared data across context calls.  If this were to happen then if the same department was retrieved in two different contexts then it would have different keys.  This may or may not be a problem depending upon how they are used.  If the surrogate key is strictly to map back to the database key then everything is probably already.  But imagine if an application wanted to determine if the user had changed a value by simply comparing the surrogate keys, since they do uniquely identify the objects.  It wouldn't work.  So the application has to be aware of how surrogate keys work to ensure they are not misused?  How is this any different than understanding how database keys work?  I'm not convinced surrogate keys solve the general problem any better than using the database keys.  They just introduce a different set of problems.

Repository/Query Pattern

The repository pattern has taken a beating of recent by the ORMs folks since the ORM technically already implements the pattern.  But what most ORM folks won't clarify is that ORMs take it too far.  In a general ORM every table can be added, updated, deleted and queried.  But in most systems there are at least a core set of tables that are read only to the application, lookup tables for example.  The ORM won't distinguish.  Therefore I think the repository pattern is useful to at least control the functionality available.  This doesn't mean that you need to implement a concrete version of the repository, an interface is sufficient.  I don't even agree that a generic repository interface is good as it is really no different than the ORM.  In my case I generally create an interface that exposes a core query method and, depending upon the table, a set of methods to add, update and remove entities.   The query method is the interesting part.  It generally accepts an interface that is specific to the repository and returns an enumerable list of domain objects.  The repository is responsible for mapping data models to domain objects and vice versa.  This is about the most generic I think a repository can get.

The query interface just contains a single query method so it can actually be a generic type if desired.  But I find that generic types are harder to read in code so I generally create a domain-specific query interface that inherits from the generic query interface just to keep it clean.  This also makes mocking the interface much easier.  Because the repository understands the query interface it becomes very easy to create any number of queries without touching the repository.  I generally use an extension class to add standard queries to the repository to simplify its usage.  For example I would probably create extension methods for IEmployeeRepsository that: got an employee given their key, got the subordinates of an employee or got the employees who could be managers.  Even better is that an application could create its own queries without having to touch the repository.  Normally extension methods aren't easy to mock but since all the queries ultimately call the same method it is easy to mock the repository to return a fixed set of data for any query.

One of the challenges with this approach is efficiently querying the database.  It is unwise to pull all the rows from a table just to filter them in a query.  Therefore the ORM's entity set has to be queried directly by query objects.    Since queries are data-level objects this makes sense.  A query is strictly for getting the right data so it should be able to query the underlying database and retrieve the exact data it needs.  The data model helps here because the query can use the data models without regard for the actual ORM implementation.  All that needs to happen is that the ORM entity set needs to be exposed.  This is where the challenge comes in.  For EF that is DbSet<T>.   Unfortunately using this type puts a hard dependency on EF.  Even worse is that this type isn't testable making the queries harder to test.  EF does have an IDbSet<T> interface that can be used instead, and is mockable, but again there is a hard dependency on EF.  Ideally this should have been brought into the core framework but it hasn't.  But it is straightforward to create an equivalent interface that does the same thing, is ORM agnostic and can wrap an ORM implementation so all is good.

But there are additional challenges with the entity set.  Often we want to pull a set of related data from the database rather than lazy loading it.  At least for EF this is a problem.  EF exposes this functionality off the entity set via the Load method.  But the method is implemented as an extension method meaning it won't work for a custom entity set interface.  So you'll have to reinvent the wheel.  Even worse is that the existing implementation relies on the entity set deriving from a specific type so you have to create a wrapper that uses the right object otherwise it won't work.

One final challenge with using entity sets is LINQ itself.  Most queries will likely be implemented using it but the actual provider used is based upon the entity set (which is determined by the ORM).  As a result you can create queries, write unit tests to ensure they are valid and yet at runtime they will still fail because of the underlying LINQ provider differences.  There really isn't any good solution to this problem short of using the ORM directly and providing all the infrastructure needed to unit test it.

Final Thoughts

This post has really been more about my experience with ORMs (at least EF) than technical content.  I just felt that the challenges I encountered had to be felt by more people using ORMs.  I still feel that ORMs are viable for applications that warrant their usage.  I don't think they are useful for every application and I don't believe they solve all the problems that traditional data access have.  I believe that a traditional data access layer would be more consistent, more easily testable without extra interfaces and simpler to understand.  But at the same time ORMs make it really easy to add additional entities later on without having to write a bunch of new code.  This reduces the long term maintenance and effort needed to support an evolving system.  Lazy loading, query generation and no SQL are certainly nice features to have.  Unfortunately ORMs don't work with every database design.  Databases that are predominantly stored procedure based or use views a lot will end up challenging to implement in ORMs.  If nothing else the concepts that ORM use: repositories, UoW, queries, PI, etc are good features to use even when you cannot use an ORM.  I'll continuing to use EF, and maybe NHibernate, for complex applications until the next generation of data access layers come along.

Strongly Typed Application Settings Using T4

We all know that the Settings infrastructure added to .NET a while back is the "correct" way to create strongly typed application settings in configuration files.  The problem with this approach though is that the entries generated in the config file aren't pretty.  You have only limited control over the naming, there are lots of extra information to wade through and non-devs can easily get confused.  The Appsettings that has existed since v1 is a simpler approach that most people are comfortable with.  But you lose a lot of functionality by using AppSettings.  The Settings infrastructure brings into play a Settings class with strongly typed properties for the settings and default values for settings that are missing.  But sometimes that is overkill or simplicity is just too important. 

Fortunately the biggest benefit, strongly typed properties, can be easily implemented over the existing AppSettings infrastructure.  With a little bit of extra code you can even deduce the type of the settings.  This gives developers the benefits of type checking and compiler validation with the ease of configuration.  To get all this to work all you need to do is create a T4 template.  A full discussion of writing T4 templates is beyond the scope of this post so I'm assuming you already have basic knowledge.  For more information on T4 templates I recommend MSDN and some of the blogs on the topic.

  • MSDN
  • Scott Hanselman's blog on T4 with links to lots of information

The goal is to create a Settings class that wraps each of the settings defined in appSettings in the config file.  Rather than calling ConfigurationManager with some string property that might or might not exist we want to reference a strongly typed property that is enforced at compile time.  If someone removes a setting from the config file we want a compiler error to let us know.  It also makes refactoring and determining who is using the setting a lot easier. Here's some reasonable goals:

  1. Create a static class to expose each setting defined in appSettings as a strongly typed, readonly property.
  2. Use the most restrictive type for each property rather than string.
  3. Update the generated class whenever the configuration file changes during development.

There are many articles out there on how to write T4 templates so I'm not going to go into the details other than to point out a few interesting tidbits.  Also note that some of the code I'm using was inspired from the T4 work that others have done but unfortunately it has been so long that I cannot remember who all I should give credit due. 

Defining the template, finding the namespace to use and locating the configuration file are pretty standard stuff.  Here's the start of the template.

namespace <#= GetNamespace() #>
{
    internal class AppSettings
    {
    <#
        ExeConfigurationFileMap configFile = new ExeConfigurationFileMap();
        configFile.ExeConfigFilename = GetConfigPath();

        if (String.IsNullOrEmpty(configFile.ExeConfigFilename))
            throw new ArgumentNullException("The project does not contain App.config or Web.config file.");        
        
        var config = System.Configuration.ConfigurationManager.OpenMappedExeConfiguration(configFile, ConfigurationUserLevel.None);
    #>
    }
}

GetNamespace and GetConfigPath are contained in a separate ttinclude file that can be reused across templates.  They do what you might expect.  Code in the <# #> block runs when the template is evaluated and can contain any C# code.  For this template the code creates a Configuration object for the config file used by the project.  This will be used to enumerate the application settings later.  The key functionality for this template is to enumerate the app settings which we can do almost like we'd do in regular code. 

    var appSettings = config.AppSettings.Settings;      
    <#   
         foreach(KeyValueConfigurationElement setting in appSettings) {            
    #>
    public static <#= GetSettingType(setting.Value) #> <#= GetSafeName(setting.Key) #>
        {
            get
            {
                return <#= GetSettingConverter(setting.Value) #>(GetConfigSetting("<#= setting.Key #>"));
            }
        } 
        
    <#
        }
    #>    
    
        private static string GetConfigSetting(string settingName)
        {
            var setting = ConfigurationManager.AppSettings[settingName];
            return setting ?? "";
        }                
    }

For each setting the template generates a new, readonly property that returns the value from the configuration subsystem at runtime.  This is done through a helper method on the class.  <#= %> tells the generator to evaluate the expression at generation time store the results in the template.  So the setting key is baked into the code at generation time but the retrieval of the setting is deferred until runtime.

The trickier part is determining what type to use for the setting.  That is what the GetSettingConverter is for.  It is evaluated at generation time to determine the best type for the setting.  In my development the config file generally contains the default values to use for settings.  Either at deployment or via transforms the settings can be changed to match the needs of the environment.  Because the config file contains default values it makes sense that we can try to deduce the type based upon the default value.  To keep things simple we'll just stick with simple types like numbers, booleans and strings.  We could certainly get more complex but I don't think that is necessary.  If no type can be determined then string is used.

<#@ include file="StandardTemplateInclude.ttinclude" #>
<#+    
        private static string GetSettingConverter ( string value )
        {
            var type = GetSettingType(value);
            
            if (type == "int")
                return "Convert.ToInt32";
            else if (type == "double")
                return "Convert.ToDouble";
            else if (type == "bool")
                return "Convert.ToBoolean";
            
            return "";
        }
        
        private static string GetSettingType ( string value )
        {
            //Try to convert to int first
            int iValue;
            if (Int32.TryParse(value, out iValue))
                return "int";
                        
            //Double is next
            double dValue;
            if (Double.TryParse(value, out dValue))
                return "double";
                
            //Boolean
            bool bValue;
            if (Boolean.TryParse(value, out bValue))
                return "bool";
                
            return "string";
        }
#>

First the standard include file is added.  The include file contains the helper methods mentioned earlier.  In T4 these are known as class functions (callable at generation time) so they must appear after all the other blocks.  The rest of the template consists of the class functions used to determine the setting type.  GetSettingConverter takes the value of the setting as was stored in the config file and attempts to parse it to determine its type.  The GetSettingType method takes a simple approach of just attempting to convert the setting to various types until it succeeds.  If you want to support more types or add better logic then this is the method to change.  If the type cannot be determined then string is used.  It is important to note that the return value is the primitive type name because this will end up being the value that is used in the property signature.  The converter method then generates a call to the appropriate converter function to convert the string value, stored in the config file, to the appropriate type, at runtime.  For this sample the standard conversion routines are used but more likely you'll want to call specialized conversion routines that can handle invalid values.

Having implemented all this we can run the template (by right clicking and selecting Run Custom Tool) and a generated file should appear with the strongly typed properties.  The last thing that needs to happen is that the template needs to be updated whenever the config file changes.  While templates can be configured to refresh when certain files change I've had bad luck getting it to work correctly.  Therefore I just know that if I add a new setting to the config file (that I want programmatic access to) then I need to manually refresh the generated file.  While not ideal it is at least simple.

Attached to this post is the source code for the template I'm using.  It could probably be improved in many ways but for me it solves the problem at hand.  Feel free to use it if you need to.

Attachment: T4AppSettings.zip
Posted Sun, Jun 17 2012 by Michael Taylor | no comments
Filed under:
Diskeeper 2012

Condusiv has recently released the next version of Diskeeper, 12.  I've been running it for a while now and I'm still satisfied with the way it optimizes my drives without eating up the resources.  Traditionally I wipe my machine on a yearly basis because of all the extra stuff that gets installed and the slow down on the hard drives but since I've been using Diskeeper I'm averaging closer to 18 months now.  The hard drive just isn't running slow.

For those of you using SSDs then the new version may be very useful as it has optimizations for SSDs.  As you probably know SSDs shouldn't be defragged like regular drives because it'll reduce the life expectancy.  Diskeeper somehow is suppose to keep them running at top speed without reducing the lifetime.  I have no idea how it works or how well because I don't use SSD myself but as SSD becomes more common this kind of technology is going to be really useful.

Another area that has been improved is with the initial writes to the drives.  Prevention is better than curing fragmentation.  Diskeeper is suppose to prevent common fragmentation problems from occurring to begin with.  This means you don't have to waste resources on the defrag process.  Unfortunately this isn't easy to measure so I just have to take their word for it.

HyperBoost is also new to this release.  It is documented as speeding up boot times without impacting the system.  For an initial install Windows generally boots really fast but as more components are added over time it slows down considerably.  This is one of my key indicators to know when to clean the machine.  But for my current machine I haven't really noticed Windows slow down at all.  Therefore I can't really say whether HyperBoost is helping me at all or not because it certainly hasn't slowed things down.  Still, slow boots are common on Windows machines so anything that can help alleviate that is desired.

One area that was heavily modified is the UI.  I can't really say that I like the new UI.  It seems "prettier" but wasteful to me.  The old interface wasn't pretty but I could quickly see everything I needed.  True techno-geek stuff.  The new UI seems like it is designed so a non-techy person could use it.  I have to scroll to see information now which makes it harder to navigate and not everything is visible to me at once.  Defrag is not something non-techies would do so I don't understand the need for dumbing up the UI.  Maybe I'll get use to it over time but one of the things I've always liked about Diskeeper is the "set and forget" approach so I don't expect to have to use the UI very often.  This isn't really a show stopper though.

So if you haven't been defragging your drives, are using Window's poor man's defrag or just need to get a new version then Diskeeper 12 is a good product to check out.  It isn't that expensive, does its job without having to be tweaked with and will keep your system running well. 

VS2012 RC

Now that VS 2012 RC is available I think it is time to reevaluate VS 2012 and see where it stands.

The biggest complaint most people threw at the Beta was the dullness of the colors.  MS heard and backtracked on the coloring a little.  While not fully completed by the RC, VS2012 now has color again.  It is still a gray or black IDE but common icons have colors, the icons can be more easily understand and the overall coloring is starting to make sense.  Of course you'll still be able to change your theme but I think out of the box the light and dark themes are going to be OK to use.  MS also changed the status bar color for certain operations.  I'm not sure yet if this is good or bad but it is intriguing.  The only remaining issue in the UI, in my opinion, is the horrible all caps menus.  Still cannot understand this one but fortunately MS understands and has provided a way to get rid of it.  Unfortunately there is no option in the Options dialog to turn off all caps yet (hopefully there will be) but for now you can change a registry entry.  We'll see if they add the option for the final release.  Here's a link to the fix: http://www.richard-banks.org/2012/06/how-to-prevent-visual-studio-2012-all.html.  To make it easier on myself I'm also posting the registry entry that needs to be set.


HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\11.0\General -> SuppressUppercaseConversion = 1 (DWORD)

Another big area of complaint in the Beta was Team Explorer.  TE broke everything up into different views (work items, pending changes, etc).  While this sounds good the issue comes in that in order to check something in you often need to associate the changes with a work item.  The problem is that to get to the list of work items you have to either memorize the item ID or switch to the Work Item view, run the query and then switch back to Pending Changes view.  This was slow.  Fortunately MS has modified the Pending Changes view to allow you to access your favorite queries.  While not perfect, this at least gives us a more efficient method for associating work items with changes.  We'll see if any more changes make it into the final release.

Customizing the installation of VS has also been added in for the RC.  Unfortunately the options you're given are pretty lame.  Evidently MS believes everybody wants to develop managed and native code, use all the tools in VS and some tools that aren't even produced by them.  The list of options you can turn off are limited to things like SharePoint and some C++ libraries.  In my experience developers are either going to write all native code or their going to write all managed code.  Certainly there are folks who do it all but the majority do one or the other.  Therefore not being able to toggle managed/native languages during installation seems bad.  Additionally I, personally, find a lot of the tools that ship with VS to be useless so I'd like to eliminate them as well (especially since they have their own entries in Programs & Features) but alas I can't.  I can understand some components (like SqlLocalDB) that VS requires to run but most of the components don't fit this category.  Installation size of VS is becoming more of an issue every release.  I was really hoping they would pear it down for this release.  Oh well.

Overall I am satisfied with the performance and memory usage of VS.  I have 2 or 3 instances of it open at a time and memory isn't out of hand nor is performance slow.  The RC also seems to be really stable provided you stick with the installed components.  Extensions continue to have problems with VS2012 (even MS'es own like NuGet) and some newly release tools (like EntityFramework) really don't play well with VS2012 at all.  I expect all these issues to be resolved by release time so overall I'm looking forward to VS2012 and what it will have to offer.

(Free) WinForms/WPF is Dead in VS2012 Express

UPDATE: Great news!!! Microsoft has seen the error of their ways.  They have formally announced that they will in fact release VS2012 Express for desktop apps (instead of just Metro) - http://blogs.msdn.com/b/visualstudio/archive/2012/06/08/visual-studio-express-2012-for-windows-desktop.aspx.  This is a big win for students and hobbyists.

Well VS2012 is finally in RC mode.  I have to say the UI is looking better but please, for the love of all that use VS every day, GET RID OF THE STUPID ALL CAPS MENUS!!!!!  Who in the world is designing this stuff in Redmond?  It's like they never took UI design 101 classes.  Everybody who has used a computer for more than 5 minutes knows where the menus are so why do they have to be highlighted?  It couuld just be me but it seems like the management at MS are making more and more missteps with this release as we get closer.  Now on to the really, really disasterous news for VS2012...

You will no longer be able to create WinForms/WPF apps for any OS nor any app for pre-Win8 OSes in VS 20102, for free.  That's right the only type of desktop app you can create in VS20102 Express is Metro (which only runs on Win8).  We all know that Metro is not going to catch on any time soon because Win8 is not going to be that popular.  Yet MS is going out of their way to force Win8 and Metro down everyone's throat.  Give it up MS.  Win8 is not a mandatory upgrade so unless somebody buys a new PC (and doesn't get the downgrade DVD) Win7 will continue to be the most heavily used OS for the foreseeable future.  Yet students and hobbyist won't be able to write desktop apps with VS2012 Express. 

I can't possibly imagine what group at MS thinks this is a good idea.  As a part-time professor we use MSDNAA (or DreamSpark as it is called now) to teach classes.  But it takes a couple of weeks to get the students into the system and access to the software.  During this time students need to be able to write programs.  Currently they can download the Express edition and upgrade (if they desire) to the full version when it is available.  But now, thanks to the absolutely moronic decision by MS, schools are going to have to make a tough decision - stick with VS 2010 or switch to a different IDE.  The third option (that MS is likely hoping for) - upgrading all school computers to Win8 and teaching Metro is just completely unrealistic.  Firstly first year students cannot possibly be expected to write Metro-style apps after just a couple of lectures.  That's sort of what console apps are for.  Secondly upgrading school computers is something that takes time, at least a semester, and never done lightly.  Thirdly most schools skip a version because they can rarely justify the resources and (I believe) most schools already switched to Win7.  Finally, unless you're teaching .NET, any compiler will do and their are a lot of free compilers already available.  VS is great but if the cost of using it is high then nobody will.

At this point I think MS has just put the nail in the coffin for student and hobbyist programmers.  VS2012 Express is not a viable product for them anymore.  Therefore they will either stick with VS2010 or switch to another product entirely.  Now normally I'd say that a free product cannot be knocked but in this case one of the very big benefits of VS Express is early accessibility to new programmers which leads to a more likely use of the (paid) product professionally.  But with VS2012 Express, as the current RC stands at any rate, this will not be an option.

So, fellow developers, band together and let MS know how stupid this plan to not allow non-Metro development on VS2012 Express is.  Voice your opinion and force MS to change their plans before they doom the students and hobbyists to VS2010 or subpar IDEs.  It can only hurt us long term as we have to deal with junior developers who have no experience with VS in school because of this draconian practice.

More Posts Next page »