January 2013 - Posts

Hosting a Private Visual Studio Gallery

Visual Studio extensions are really popular these days.  Not surprisingly Visual Studio has a very active extensions gallery.  But unfortunately the gallery is public so if you develop some extensions for your personal use, or for your company, then you cannot use the public gallery.  Fortunately though VS supports the concept of a private gallery.  A private gallery is nothing more than an extension gallery that is not accessible to the world.  It is discussed on MSDN so refer to there for more information about the what's and why's.  Unfortunately the documentation is a little vague and setting one up is not trivial.  I'm going to discuss the steps needed to get a private gallery up and running.

Caveat

First a word of warning.  VS supports extensions and the public extension gallery.  VS also supports having multiple galleries configured (and usable) at once.  Actual support for a private gallery is "Beta" at best.  Therefore don't expect a lot of support from Microsoft in setting up and running a private gallery.  You've been warned.

Gallery Basics

Before setting up a gallery it is useful to understand what it actually is.  Ultimately a gallery is nothing more than a list of extensions (not necessarily a VS extension) with any corresponding metadata that can be installed into VS.  All a gallery is really responsible for is providing a list of available extensions and sending the extensions to VS when needed.  Therefore a private gallery really doesn't do a lot.  A gallery uses an Atom feed to send the list of available extensions to VS.  VS then displays the list in the UI.  When the user selects an extension VS calls back to the gallery and request the extension.  The gallery sends the extension to VS and VS takes it from there.  Pretty simple stuff.

Atom Feed

The feed (atom.html) provides quite a bit of information to VS - the extension name, version, unique ID, author, description, create date and update date.  From this VS can generate everything it needs.  Each extension only has one entry in the feed.  When VS pulls the feed it compares the current version information to what is installed on the user's machine.  If there is a difference then VS adds the extension to the list of updates that are available. 

Unfortunately generating the Atom feed is the hard part of hosting a private gallery.  Not only is it ugly but it gets bigger as you add new extensions.  During the betas and RC Microsoft had a tool to generate this for you but it was pulled before the release. You'll still see it documented but it is gone. Thus you'll have to either manage the file by hand or use some other tool to build it.

Here's what a basic feed might look like.

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title type="text">My Extension Gallery</title>
  <id>uuid:6df6542f-d22f-4138-90dd-a386fa370e64;id=1</id>
  <updated>2013-01-11T23:36:04Z</updated>
  <entry>
    <id>uuid:6df6542f-d22f-4138-90dd-a386fa370e64;id=2</id>
    <title type="text">My Extension</title>
    <summary type="text">Provides something useful.</summary>
    <published>2013-01-11T11:51:56-06:00</published>
    <updated>2013-01-11T11:08:09-06:00</updated>
    <author>
      <name>Me</name>
    </author>
    <category term="Templates" />
    <content type="octet/stream" src="galleryurl/Templates/MyExtension.vsix" />
    <Vsix xmlns="http://schemas.microsoft.com/developer/vsx-syndication-schema/2010">
      <Id>MyExtension.VSIX.ID</Id>
      <Version>1.0</Version>
    </Vsix>
  </entry>
</feed>

Let's break down the elements.

  • Title - The title of your gallery. 
  • Id - A unique ID for the feed. 
  • Updated - This should be updated each time the feed changes so VS will refresh its view.  I have not confirmed this but it is reasonable to assume that VS will look at the value and not bother walking through the extensions if it hasn't changed since the last time the feed was read.  Note the date/time values are in UTC format.
Each extension has an <entry> element within the feed. 
  • Id - The unique ID of the entry.  This should generally just be the VSIX unique ID.
  • Title - What the user's sees when seeing the extension. 
  • Summary - The nice description that shows up below the extension. 
  • Published - The date/time when the extension was added. 
  • Updated - The date/time when the extension was last updated.
  • Author - The author of the extension.
  • Category - An optional element that specifies a subfolder the extension should appear under in the gallery UI.  This is useful for grouping extensions.
  • Content - Specifies the type of extension. 
    • type - octet/stream indicates a binary file.
    • src - This is the full URL to the extension.  If an extension appears in the gallery but cannot be downloaded then this attribute is most likely wrong.
  • Vsix - This starts the metadata for the VSIX itself.
    • id - The unique ID of the VSIX package.  This should generally match the ID of the entry.
    • version - The version of the extension.  This should match the version of the VSIX extension.

It is generally assumed that an extension is shipped as a VSIX but it could also be an MSI. Beyond the source URL not much changes in the gallery.

Hosting a Gallery in IIS

To host a gallery all that needs to happen is that a listener on a URL needs to return back the Atom feed when requested.  It also needs to return the files when they are requested by VS.  This can be done rather easily using IIS.  I use the existing TFS web server to help keep things consolidated but you can use any web server that is available.

  1. Create a new web site (or application) in IIS (ex. MyGallery).
  2. Optionally configure it to use a custom app pool.
  3. The site does not .NET or anything so the only additional configuration is to set the atom.html file as the default document.  This should cause a basic web.config file to be generated.
  4. Add the atom.html file to the root of the site following the format given earlier.
  5. Browse to the site and you should see the RSS feed. 

    Note: In IE 10 I was unable to view the feed even though it was working.  I used FireFox to confirm it was working properly.

If the feed does not appear then the site is not configured properly.  Verify the security on the folder is correct and the site and app pool are running.  On newer versions of IIS the app pool will run under ApplicationPoolIdentity which makes it more difficult to configure.  Consider using a fixed account.  Use the Basic Settings in IIS to test the connection for the path.

Now you can add new extensions to the site.  Ideally extensions should be placed in a subfolder under the site.  Remember to account for the folder in the content src attribute.  When adding new extensions or updating an extension one be sure to update the <updated> element of the feed and the same element for the extension (if it is an update). 

Configuring Visual Studio

Configuring VS to use the new gallery is really simple.

  1. Go to the Options dialog (Tools\Options).
  2. Go to the Extensions options (Environment\Extensions and Updates)
  3. Click Add
  4. Specify a name for the gallery (ex. Company).
  5. Specify the URL to the gallery (the site root where the Atom feed is).
  6. Click Apply and then OK.
  7. Go to the extension gallery (Tools\Extensions and Updates).
  8. Expand the Online node and the new gallery should be listed.
  9. Click the gallery and the extensions (and categories) should be shown.
  10. To test the feed select one of the extensions and click Download. 

    Note: A 404 error indicates either the source URL is wrong or there is a permissions problem.

Hosting a Gallery in IIS (Part 2)

The above process works but you'll likely run into several problems.

  • IIS is picky about serving up files that end with VSIX or MSI.
  • Maintaining the Atom feed is a nightmare.

Fortunately there is a better option right now.  Jakob Ehn has written a simple MVC application for hosting an extensions gallery.  He blogged about it here.  The source is available on CodePlex.  This application is still a beta but, other than a few issues mentioned below, it seems to work pretty good.  To use it you'll do the following.

  1. Download the setup program.
  2. Run the installer on the IIS machine.

    Note: The installer must run with admin privileges so you'll need to start it from an elevated Explorer window or command prompt.
    1. On the first installer options page specify the web site and app pool to use (ex. MyGallery).
    2. On the second installer options page specify the path to the files on the disk.  The default is probably fine.
    3. Click Install.
  3. Unfortunately it is still not completely configured so open the web.config and make the following changes.
    1. Change FeedTitle to the display name of the gallery (ex. My Gallery).
    2. Change BaseURI to the base URI of the site (ex. http://myserver/MyGallery/Gallery/Extension/).  The "/Gallery/Extension/" is required.
    3. Change VSIXAbsolutePath to the file system path where the extensions will be stored.  Personally I don't like to store data files in the website directory so I use a separate path but be sure that the app pool identity has permissions to read the directory.
    4. Save the file.
  4. The application was written using MVC 4 but the installer does not install the required binaries.  Either ensure that MVC 4 is installed and configured for your web server or manually copy the MVC 4 binaries to the site.
  5. Browse to the site to confirm the feed is working.

To add or update extensions simply drop the files into the configured VSIXAbsolutePath directory.  If you want to use a category then create a subfolder.  From what I can tell VS only allows a single level of categories so do not try to nest the folders beyond a single level.  Behind the scenes the application will generate the Atom feed whenever it is requested via the URL.  To generate the feed it enumerates the files in the directory.  For each VSIX it generates an entry.  The feed is regenerated each time it is requested so the update date will always change.  For each entry the published and update date come from the file's create and last updated values.  The remainder of the information is determined by the VSIX file itself.  If the extension is in a subfolder of the root then a category entry is added for it.  Now you can use VS to browse the gallery.

As of 13 January 2013 there is a bug in the installed code that prevents download from working.  When a request is sent to the application to download the file it attempts to open the file for read/write access.  Unless the app pool has permissions (generally not a good thing) then it will fail and a "Not Accepted" error will be generated.  The fix has already been made in the code but the installer has not been updated.  Therefore you can download the source and rebuild it.  Then just swap out the binaries.

The application is still in beta so be prepared for some issues.  Here's a list of things I would like to see changed.

  1. The installer needs to be updated to contain the file download bug.
  2. The MVC 4 binaries should be installed to the application directory if they do not yet exist.
  3. The installer should prompt for the settings so we don't have to manually edit the config file.
  4. It is probably not a good idea to regenerate the feed each time VS requests it.  As more files are added this will get really slow.  Ideally a file system watcher should be used to detect changes in the files and the feed regenerated only when changes are detected.
  5. Currently only VSIX files are supported.  MSI-based extensions are not.
  6. The ID of the entries are based upon the ID of the gallery rather than the ID of the VSIX.  Ideally they should match the VSIX of the entry.
  7. The publish/update date of the files are not good indicators of the extension lifetime.  Ideally the publish date should be set the first time the entry is added to the feed and should never be changed.  This would require that the feed be updated and never recreated (as it is now).
  8. Perhaps as an alternative extension metadata could be managed in a database (like NuGet) while the files are stored on disk.  This eliminates the need for watching the file system and updating the feed.

None of these are showstoppers unless you are dealing with a lot of extensions.  Overall the Inmeta application makes hosting a private gallery a lot easier.

Summary

As VS extensions become more popular it is inevitable that you will want to write and publish your own.  But some of them will be for personal or company use and should not be published on the public gallery.  Hosting a private gallery is relatively painless to do and allows for private extensions.  VS fully supports this concept.  MSDN provides more information on how to use a private gallery.  Hopefully MS will eventually release some tools to make creating and managing private galleries easier.  Until then Jakob's project will fit the bill.

Setting Up a Private NuGet Repository

These days most developers use NuGet to reference third-party dependencies like EntityFramework, Enterprise Library and NInject.  A full discussion of NuGet is beyond this post.  There are lots of articles on what it is, how to use it and how to package and publish to it so I'll ignore all these things.  But what about internal dependencies like shared libraries used by company applications?  It turns out you can host your own private NuGet repository as well.  Visual Studio is already set up to support this.  The hard part is getting the repository set up.  At my company we've been running a local NuGet repository for a year and it works great, for the most part.  We have a company policy that all dependencies must be stored in NuGet.  This pretty much eliminates versioning issues, sharing code across projects and storing binaries in source control.  In this post I'm going to go over the steps to set up a private NuGet repository.  Others have posted on this topic and there is some documentation available but unfortunately it is out of date.

Note: These instructions are based upon the NuGet Gallery source as of 6 Jan 2013.

Prerequisites

Before you can get NuGet up and running you'll need some prerequisite software.

For compiling NuGet:

For hosting NuGet (using the preferred approach):

  • IIS to host the web site
  • SQL Server to store the site metadata
  • File system to store packages
  • URL Rewrite 2.0

Compiling the NuGet Gallery

Here's the steps for compiling the NuGet source code.

  1. Download the source from here
  2. Extract the files to your local drive.
  3. At this point you can build the source via PowerShell and the .ps1 file but I've never had any luck with it.  So I follow the next set of steps instead.
  4. Open the solution in Visual Studio.
  5. Build the project.

At this point you have the NuGet website (the Website project) compiled and ready to go.  Note that NuGet is configured to use HTTPS for authentication.  For your internal repository you might not want this.  Unfortunately to disable it you have to modify the code.  Find all references to the attribute 'RequireRemoteHttps' and comment them out.

Deploy NuGet to IIS

  1. Copy the website to the target directory on IIS (for our purposes we assume \wwwroot\NuGet).  For the current source this includes the following files and directories:
    1. \Api
    2. \App_Code
    3. \App_Data
    4. \Bin
    5. \Content
    6. \DataServices\*.svc
    7. \DynamicData
    8. \Errors
    9. \Infrastructure\*.sql
    10. \Scripts
    11. \Views
    12. All files in root other than: *.cs, *.ruleset, packages.config, *.resx, *.tt, *.csproj
  2. If not already done create a new IIS website backing the target directory.
    1. It is strongly recommended that you create a new app pool for NuGet and use a custom account.  Note that the account will need access to the SQL database so a network account is best.  Where I work we use the same account as our TFS services.
    2. The app pool will use .NET 4.0 with an Integrated pipeline
  3. Ensure that the app pool has read/write access to the App_Data/Lucene folder

Configure NuGet

NuGet is now deployed but it is not configured yet.  Open the web.config for the site and change the following settings.

  1. <appSettings>
    1. Gallery:SiteRoot - Specify the full URL to the NuGet gallery site.  This will be used by the site when creating links.
  2. <connectionStrings>
    1. NuGetGallery - Specify the connection string to connect to the SQL database that will house the NuGet metadata.  If the NuGet app pool is configured to use a custom account then integrated security can be used otherwise specify the account to use when connecting to SQL.

Configure SQL Server

NuGet will automatically initialize the database when it first connects.  However the database must exist so if you do not yet have a database for NuGet then create one now. 

  1. Open SQL Server Management Studio.
  2. Connect to the SQL server that will host the database.  Where I work we use the existing TFS server's database because we are not using the instance of SQL provided by TFS.  If you are using the SQL instance that ships as part of TFS then refer to the licensing restrictions of TFS before doing this.
  3. Create a new database for NuGet.  The default is NuGetGallery.
  4. Ensure that the account that NuGet will run under has full permissions to the database.

Start NuGet

If all is well you should now be able to browse to the NuGet site.  Note that the first time it starts it will be slow as it updates the database.  If any errors occur or the site refuses to start then disable custom errors in the web.config and resolve the errors.  When I initially tried to get the site running under Windows 8 it complained the web.config was invalid.  I narrowed down the error to the <rewrite> section.  Removing that section allowed the site to load.  A quick check revealed that the URL Rewrite 2.0 module for IIS (which NuGet needs) is not part of IIS that ships with Windows 8.  Using the Web Platform Installer I installed it and everything worked correctly.

Set Up E-mail

NuGet uses Forms authentication for authenticating users.  When creating a new user account NuGet will, by default, send an e-mail to the user to confirm their e-mail address.  You will want to set up NuGet to use your company e-mail server so that you can get this e-mail, and any other e-mails that are sent over time.  Otherwise creating a new user requires that you manually get and forward the generated e-mail. 

NuGet does not expose any UI to update the e-mail information and it is stored in the database.  Open the GallerySettings table in SSMS and set the following columns.

  • SmtpPort - The SMTP port of your e-mail server.  Normally this is 25.
  • SmtpHost - The SMTP host address.
  • SmtpUsername - The user name for authentication, if any.
  • SmtpPassword - The password for authentication, if any.
  • UseSmtp - Set to 1 to enable e-mail
  • ConfirmEmailAddresses - Set this to 0 to disable confirmation e-mail addresses.

By default the e-mail connection will use SSL. If you do not want SSL for your e-mail then you'll have to modify the source code again.  Ultimately the setting should be stored in the database with the rest of the SMTP settings.  But to keep it simple just do the following.

  1. Edit the ContainerBindings file.
  2. Find the initialization code for IMailSender.
  3. Set EnableSsl to false.
  4. Rebuild the code.

Package Storage

NuGet uses the database to store package metadata but, by default, it stores the packages themselves on disk.  The appSetting Gallery:PackageStoreType specifies the provider to use.  For performance reasons you will likely want to continue using the file system to store packages but the default directory is not the best choice when you need to upgrade the site.  Instead you should consider storing the files in a more reasonable path.  Be sure that the site has read/write permissions to the new package directory.  To set the file path where packages are stored add/update the appSetting Gallery:FileStorageDirectory.  Specify the full path to where packages will be stored.

If you do not want to use the file system to store packages then you can use a different package store by setting the appSetting Gallery:PackageStoreType.  Currently only Windows Azure is the alternative.  You can technically add your own and update the corresponding enum and config setting to use a different store - such as a database.

Registering

At this point you should be able to access the site.  You will need to register an account so that you can add packages.  Technically all users will need an account.  You will also want to assign yourself as an administrator for the site.

  1. Open the Users table in SSMS.
  2. Get the Key for the user you want to make an administrator.
  3. Add a new row to the UserRoles table where the user matches the key you have and the role key is for administrators, 1 is currently the value.

When creating an account NuGet will generate an e-mail that you need to click on unless you've disabled confirmation e-mails.  Unless you've set up an e-mail server you'll need to click on a link inside this e-mail from the account it was sent to.  If you're using a corporate e-mail client like Outlook it shouldn't be too hard to get the e-mail contents such that you can click on it.  For everybody else it is just easier to open the Users table in the database and copy the UnconfirmedEmailAddress value to EmailAddress. Then you can log in successfully.

Currently NuGet uses Forms authentication.  For a private gallery it might be more useful to use Windows authentication.  Unfortunately that requires adding code to the site.  NuGet relies on user accounts for managing packages so some updates will need to be made to handle Windows authentication.  That is left as an exercise for the user.

Connecting to Visual Studio

From this point on the private repository works just like the public repository.  You can upload and browse packages.  A word of warning - once a package is uploaded it is forever part of NuGet.  Nevertheless you should upload a package to make sure everything is in working order.  Once you've uploaded a package you can configure Visual Studio to use it.

  1. Open the Package Manager settings (Tools\Library Package Manager\Package Manager Settings or Tools\Options -> Package Manager).
  2. Go to the Package Sources section.
  3. Click the plus button to add a new source.
  4. Specify a name (ex. Company)
  5. Specify the source (http://<server>/<site>/api/v2).
  6. Click Update.
  7. Click OK to close the dialog.
  8. Open the Manage NuGet Packages dialog (I generally do so from a solution).
  9. Under Online select the new repository.
  10. The uploaded package should appear.
  11. Click Install to confirm the package can be downloaded and installed correctly.

Ownership

NuGet associates one or more owners with every package.  The owner of a package can upload new versions.  For a public repository this is fine but for a private repository it is likely that everyone should be able to update packages.  To add or remove owner(s) from a package do the following.

  1. Open the package on the NuGet site.
  2. Select Manage Owners.  This is only available for owners and adminstrators.
  3. Add or remove owners from the package.

Local Repository

One issue with NuGet is that once it is uploaded it will be forever available.  NuGet does not support deletion.  It is generally a good idea to verify that a NuGet package is working correctly before uploading it but this becomes a catch-22.  You cannot test the package unless you can download it from the repository.  You don't want to upload the package until it works.  Fortunately there is an easy solution, a local repository.

A local repository is useful for testing packages.  Ultimately it is nothing more than a directory where NuGet can retrieve packages from.  Depending upon your need you might have a local repository on each developer's machine or you might set up a file share where all devs can point to.  Either way you point the Package Manager to the directory and it'll work just like a regular repository.  To get packages into the repository you just drop the .nupkg file into the folder.  You can then install/upgrade the packages through Package Manager in Visual Studio.  Note that packages always have a full version in the filename so it is easy to tell whether there are new versions available.  Once the testing is complete you can delete the packages.

To configure Visual Studio to use a local repository do the following.

  1. Open the Package Manager settings (Tools\Library Package Manager\Package Manager Settings or Tools\Options -> Package Manager).
  2. Go to the Package Sources section.
  3. Click the plus button to add a new source.
  4. Specify a name (ex. Local)
  5. Specify the source path (ex C:\MyTestNuGetGallery).
  6. Click Update.
  7. Click OK to close the dialog.
  8. Open the Manage NuGet Packages dialog (I generally do so from a solution).
  9. Under Online select the new repository.
  10. The uploaded package should appear.
  11. Click Install to confirm the package can be downloaded and installed correctly.