vcsjones

The AJAX Update Panel and Recursive Common Table Expressions

So my blog has become quiet again, but don’t worry! I’ve simply started pushing more of my posts toward our company blog. Check out my latest posts there:

The AJAX UpdatePanel: Is it your worst enemy?

Recursive Common Table Expressions

And from here on out are going to be a series on .NET 4.0.

Vitamin D, Sunshine, and Rainbows

Today is the start of me coming off of a client project that I have been on and off for years now. I really like the project and I tend to jump in when the client is planning some large items over the coming months. Sadly that reign has come to an end, and it’s off to different things. Regardless, a somewhat unique thing that we do as often as possible is work at the client’s location, their building, their office. While this provides enormous benefits when working with the client, it does mean that we take what we can get when it comes to office space. Recently, they gave us a new room when they expanded to the floor below them. It’s a nice room with great tables and layout for pair programming, comfy chairs, and good development machines. Though the joke around the office is we’re going to suffer from a Vitamin D deficiency. Why? It’s because there is no window or place that natural light gets in.

Today I had the good fortune of working next to a window in our office.

I never really considered how much of a difference it makes, I always wanted to think “lamps are good enough”. Honestly though, I found myself having more energy and being much perkier. I’m not usually one known to be “perky” either.

So I want opinions, how important is a window in the office to you?

Retargeting Assemblies for SQLite

If you read my previous post about dealing with SQLite and SQL CE, then you know that I am on a mission to get unit tests working correctly with a SQLite database. I decided to see if I could get “assembly retargeting” working. First, what is “retargeting”?

.NET allows assemblies to be “redirected” in a sense when the platform is different. This is how the .NET Compact Framework actually works. All .NET CF assemblies are retargetable. In your application, you are referencing assemblies such as System.dll. However, ever notice that .NET Compact Framework application will run on a desktop? That’s because the runtime is silently retargeting them to the full assemblies, like System. This is key to getting the application working on different platforms. Under all of the covers, .NET relies heavily on platform invoke to do the heavy lifting in the WinForms world. However, the P/Invoke calls between a Windows Mobile application and a workstation are wildly different. This is why redirection is needed to work on multiple assemblies.

How can we get this to work for SQLite?

There are a few things you need to get retargeting to work. You need two assemblies, one for the Compact Framework and one for the full framework. I am using the managed wrapper provided by PHX software, which happens to be open source as well. They have ZIPs of both full and compact assemblies.

The second catch is they need to have the same strong name. Oddly, these two managed wrappers use a different strong name key depending on the platform, but since it’s open source it’s very easy to download, synchronize the keys, and recompile.

The final bit is putting the assemblies in a place the application will look for them without causing problems. In your project, add a reference to the Compact assembly. Leave this for now.

Open a command prompt with administrative permissions and use gacutil to install the Full one.

Finally, you need to make sure the unmanaged libraries are ending up in your project output directory for both compact and full. There are 5 files:

• System.Data.SQLite.exp
• System.Data.SQLite.lib
• SQLite.Interop.065.lib
• SQLite.Interop.065.exp
• SQLite.Interop.065.dll

All of these are needed. The best bet is to include them in your project with compile action to “None” and "”Copy if newer” for copying.

All things done right…Tada! It works, but what is really going on? What we are really doing is tricking the .NET Framework into loading the full framework assembly instead of the compact assembly. This has to do with how assemblies are resolved. Without going over a lot of details, assemblies are resolved in this order:

1. The GAC
2. The probe path of the current application as specified by the AppDomain (like the bin)

That’s the very brief description, and a bit simplified. For the real down-and-dirty, check out this MSDN article. When the runtime looks in the GAC, it finds the assembly we installed. The assembly hash isn’t the same, but since it was retargetable and the strong name keys matched, it loaded it from the GAC anyway and didn’t even bother looking at the one we actually referenced. When it is deployed to a mobile device, it is not in the GAC, so it loads the one we referenced which should have been copied to the output. For the platform invoke, since we copied both the Win32 and the ARM libraries, platform invoke resolved their location from the deployed directory.

Neat stuff, but seemed harder than it could have been.

For what it’s worth, this is also how SQL CE is working. However it does it’s magic by putting the assembly in the GAC for the desktop version and the compact version.

C# and VB.NET

I don’t like to think of myself as a “C# Developer”. I prefer to think of myself as a “.NET Developer”, mainly because C# is not the only tool I know how to use. However, I primarily use C# as a language of choice because that’s the majority of what most .NET developers use, and I never had a reason to know or understand VB.NET. It’s not to say I was clueless about VB.NET, I’ve glanced at the syntax and I felt that it was similar enough to C# that I didn’t really need to get into the dirty details. Nor do I want to spend the time on comparing the two. There have been countless blog posts on that already.

Now, I chose to write a Compact Framework Application in VB.NET. Some of my co-workers looked at me like I was nuts. That’s one thing I really never got: Why does the general C# crowd hate VB.NET? When I say hate, I mean hate. That’s at least the sense I get from them. I’ve asked, and one such answer was “It’s too wordy and verbose,” but does that really qualify hating it?

Now I ask those C# developers, can you, with a pen and a paper and no references, write a property in VB.NET? How about class with a generic? Maybe a generic with multiple type constraints? Subscribing to an event? You can see where I am going with this. I had an experience where I did a one day bit of consulting to someone helping them port a .NET 1.1 application to the latest stuff. It went smoother than they expected once I explained the difference between a “Web Site” and a “Web Application”. They had some time to ask me, “What’s new in VB.NET?”

Ulp.

Well, there is generics of course, but I struggled to remember syntaxes. Simple things, like a property. Also, the editor for VB.NET is slightly different with how it handles intellisense completion, so I botched that a few times. It was rather embarrassing. I decided at that time there was zero excuse for me, as a consultant, to not know VB.NET as well as I know C#. To do this the most effective way possible, I needed to write a real application in it.

I’m comfortable with VB.NET, in fact I have some preferences over C# with it. I am really looking forward to the VB.NET “catch-up” for VB 10. One thing I noticed that C# had a lot of syntactic sugar that VB.NET lacked, like auto properties. The killer for me right now is that a lambda expression must be a Function instead of a Sub.

I have a new appreciation for VB.NET and VB.NET developers. Maybe I’ll do a code camp or user group presentation in it once :-).

Nastiness with Data in the .NET Compact Framework and the spiral downward (With Updates!)

The First Try

As of late, if you for whatever reason on earth follow me on twitter, you might’ve picked up on the fact that I am working on a Windows Mobile project. Written in VB.NET. Here is my experience with working with a SQL database in the .NET Compact framework, and hopefully save someone some headaches.

I naturally wanted to use SQL CE for a database. It comes out of the box, right? It’s made by the SQL team, which I usually have no problems with. SQL Server has been a solid Microsoft product for years, so let’s use that, I mean Visual Studio was “nice” enough to install it for me.

Well, on a previous application we used SQL CE. And it immediately crashed the application when we ran a query. Let’s look at some numbers. The database contained 2 tables, and one foreign keyed to the other. TableA contained roughly 45,000 records, and TableB contained roughly 4 million records. TableB contained a foreign key to tableA. The database would be read only. No inserts, updates, deletes – nothing of the sort. TableA contained 2 columns, an identity column and a nvarchar lookup that was 250 characters wide (500 bytes for those counting). TableB contained an identity, a reference to TableA, and 5 nvarchar columns that were also 250 characters.

Not the most complex of datasets. Also note that I am generalizing names of tables and columns to avoid being specific about a project.

Now, the device was admittedly underpowered. We’re talking a 250 MHz ARM processor with 16 MB of RAM. 8 MB of the RAM was being used by the OS and a Bluetooth loader. We have 8 MB of memory. Not exactly a ton, right?

SQL CE 3.5 flat out out ran out of memory. Poof, gone. Well, we tweaked some settings in the connection string like pool size, max working set, etc. No dice. The database was already given to me the way it was, was TableA really needed? Let’s merge the two tables – the thought being that the JOIN was killing it. Nope, didn’t make things better. What was the query? A lookup by an ID:

   1: SELECT Id, FirstName, LastName, Location FROM TableB WHERE LookupId = @LookupId

Yeah, pretty simple. It should only return one record, too. It was accessed using a SqlDataReader to avoid the overhead of datasets, LINQ, etc. Here’s where things got interesting. LookupId was an nvarchar, but was really just a set of numbers, like 3487982429803758. Too big to be an Int, but small enough to be a BigInt. The data had leading zeros in them, but regardless the number would be unique. So let’s change it to a BigInt. Though after the conversion, it just really didn’t look right. Some numbers were flat out converted wrong. Here is how I converted it:

   1: ALTER TABLE [TableB]

   2:     ADD [LookupIdTemp] BIGINT NULL

   3: GO

   4: UPDATE [TableB] SET [LookupIdTemp] = CONVERT(BIGINT, [LookupId])

   5: GO

   6: ALTER TABLE [TableB]

   7:     DROP COLUMN [LookupId]

   8: GO

   9: ALTER TABLE [TableB]

  10:     ADD [LookupId] BIGINT NULL

  11: GO

  12: UPDATE [TableB] SET [LookupId] = [LookupIdTemp]

  13: GO

  14: ALTER TABLE [TableB]

  15:     DROP COLUMN [LookupIdTemp]

  16: GO

  17: ALTER TABLE [TableA]

  18:     ALTER COLUMN [LookupId] BIGINT NOT NULL

  19: GO

Not very fast, but I was able to execute it from a desktop machine and let it run for a few minutes (sp_rename doesn’t exist for SQL CE for those unaware). I tried it on some sample data, and it seemed to work fine. No so. Here is a sample input, and sample output of where things didn’t work and did work. See if you notice a pattern.

See a pattern emerging? It’s the leading zeros. SQL CE does not convert a string to a BIGINT correctly if it has leading zeros. Simple as that. For every insignificant zero on the left, it chomps that many digits from the right.

See now? That’s pretty pathetic. As expected, if you do this with a full SQL engine, it works as expected. Also, it only seems to do this for a BIGINT. The conversion works just fine for INT, SMALLINT, TINYINT, you name it. I got in touch with the SQL CE team, and acknowledged the bug. They were also kind enough to provide a recent work around: put a sign in front of the digit. i.e. “+050” is properly converted to 50.

Even after all of that the query flat out didn’t run on the device. We started to worry that the device just wasn’t powerful enough. So we tested it on a slightly better device, a 550 MHz XScale with 64 MB of RAM. It ran, but was slow. Painfully slow. Now we really started getting worried.  Was our goal just too much for a CE device for a .NET Application? Were we going to have to move to something native like C++ eMbedded?

Well, what about indexes? Did we have our LookupId column indexed? You bet. That’s the first thing you do on a lookup column with a ton of rows. It didn’t seem to help much either. Usually indexes perform miracles. So what were we doing wrong?

We went through all sorts of suggestions, like using the Seek operation,  downgrading to SQL CE 2.0 (you never know) any nothing worked.

Well, let’s try SQLite first at the suggestion of a hardware vendor. We found a nice app on CodePlex that even converted it for us. SQLite is a native implementation, and we had a tricky time finding a managed wrapper for it, but we did here. the conversion took ages, but the managed wrapper was very clean and written in ADO.NET style. Let’s give it a try.

It came back instantly. I thought something went wrong or the conversion didn’t convert all of the rows. A count check confirmed that all of the millions of rows were there and came back. Though I tried a LookupId of “3”. What happens when I try to use 10000000000000?

Came back 10 seconds later. Ah but wait – we didn’t get the index pulled over. Let’s add the index in try again. The size of the database grew significantly, from 400 MB to 750 MB. The database was on a compact flash card, so we weren’t too concerned about that. Aside from the size, the query was always returning instantly (or at least sub-second).

I don’t know exactly why SQLite is infinitely better, but I think indexes are implemented in a strange way – the size of the database didn’t seem to change much in SQL CE without the index. If anyone wants to tell me how indexes work in SQL CE, I’d be happy to know. Or possibly tell me if I was doing something wrong. Some back research seemed to affirm my point with SQL CE. It’s very bloated for the Compact Framework.

One last little annoyance – the SQL CE Framework is not as simple as including an assembly in your project. It requires a full CAB installation. That isn’t exactly correct. A SQL CE MVP was kind enough to tell me that it can be distributed with the application. However, I was not able to find anywhere on MSDN, or much other places on the internet, on how to do that. What files do I need? Where do I get them? Microsoft wants you to use the MSI so they can update it with Windows Updates.

That left me with a pretty bad taste in my mouth for SQL CE. Given that the only difference between SQL CE and SQLite were a few key class names, I didn’t think I ever wanted to use SQL CE again.

The Second Try

I later was working on a project that needed to persist cached data somewhere. This is for a WPF-based Desktop application. SQLite came to mind, but it had one flaw that annoyed me. Different assemblies for x86 and x86-64. I could either have two different builds, or latebind, or do something clever, but I thought it was time to try SQL CE again, I don’t plan on having more than 100 rows in any given table anyway. SQL CE isn’t much better either. Again, it isn’t redistributable in any form other than an MSI. At least the documentation doesn’t recommend it. But at least I had the freedom to just distribute the SQL CE MSI for the correct platform and the app would just work.

You might be thinking, “Why not just force your application to run in x86 all the time and avoid the x86-64 problem?” Well, I am a 64-bit junky. If it’s a managed app, it better well damn work on the OS’s preferred architecture. It’s a pet peeve of mine I suppose. Unless there is a good reason to force x86 (like performance problems with 64-bit – remember the x86-64 Jitter is slower) or moving a legacy application to at least work properly on x86-64 as a starting point.

SQL CE is actually an OK database platform for a desktop. There are some quirky bugs with it, like the issue with BIGINT, but not bad. I suspect it will improve as well. However I will never use it for a Compact Application ever again, regardless of how big I think the data is going to be.

Unit Testing Insanity

Now, I do unit testing whenever I can. Sometimes the client prefers I don’t due to time constraints or don’t see the benefit. I push for it, but there is only so much I can push. The previous client was one of those, so I didn’t notice this issue until recently.

If you’ve never written a Compact Framework application, there are a few things to know. First, an application targeted for the Compact Framework is fully capable of being referenced by Full .NET Framework assemblies, which makes sense. the stranger one is that in a Compact Framework project you can reference a Full .NET Framework assembly, it just may not run properly. That in mind, here is a sample solution setup.

• ProjectFoo.Data
• ProjectFoo.Business
• UnitTests.ProjectFoo.Data
• UnitTests.ProjectFoo.Business

All of them are Compact Framework Class Libraries in VB.NET. The UnitTests projects need to be Compact Framework projects because for whatever reason, a Full VB.NET Project cannot reference a Compact Framework VB.NET Project (though C# can do it). My reasons for using VB.NET is worth another blog post later.

Regardless, things were going smoothly with this approach. NUnit has no problem working with the Compact Framework, my CI server runs the tests fine, Resharper runs them, seems like a good setup. Time to add some data! I went for SQLite since this is a Compact Framework application. I referenced it, added all of the native dependencies, at this point it’s like clockwork for me. My first test is to test a class that creates the database for the first time. Here was my result:

System.IO.FileLoadException: Could not load file or assembly 'System.Data.SQLite, Version=1.0.65.0, Culture=neutral, PublicKeyToken=1fdb50b1b62b4c84, Retargetable=Yes' or one of its dependencies. The given assembly name or codebase was invalid.

Huh? Lets check the Bin directory and make sure it’s there. Yep.

I knew this wasn’t going to be an easy one. I’ve dealt with some bad problems with loading assemblies in the past when I was writing an multi AppDomain project in the past, and knew the best place to start was the Fusion log. It happened to already be enabled from previous problems like this, which is lucky, I guess. Here’s what was in the log:

*** Assembly Binder Log Entry  (9/28/2009 @ 12:33:27 AM) ***

The operation failed.
Bind result: hr = 0x80131047. No description available.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
Running under executable  C:\Program Files (x86)\JetBrains\ReSharper\v4.5\Bin\JetBrains.ReSharper.TaskRunner.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: User = VCSJONESDC\Kevin Jones
LOG: DisplayName = System.Data.SQLite, Version=1.0.65.0, Culture=neutral, PublicKeyToken=1fdb50b1b62b4c84, Retargetable=Yes
 (Fully-specified)
LOG: Appbase = file:///C:/development/Thycotic/SecretServer.WindowsMobile/trunk/UnitTests.SecretServer.WindowsMobile.Data/bin/Debug
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = UnitTests.SecretServer.WindowsMobile.Data
Calling assembly : SecretServer.WindowsMobile.Data, Version=1.0.3557.41832, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: No application configuration file found.
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v2.0.50727\config\machine.config.
ERR: Failed to apply policy (hr = 0x80131047).
ERR: Unrecoverable error occurred during pre-download check (hr = 0x80131047).

Not too much help. The HR is the same as the except which I doubled checked like so:

   1: [System.Runtime.InteropServices.Marshal]::ThrowExceptionForHR(0x80131047)

Yes, I used a PowerShell script to test this out. Once you get use to it, I find it very handy for such uses.

So, let’s start digging into the IL of SQLite using our good friend ILDASM. Reflector doesn’t quite show one of the inner workings we want. If you look at the manifest of SQLite, it’s got this oddball in there:

   1: .module extern 'SQLite.Interop.065.DLL'

Reflector simply shows it as a comment at the top of the manifest disassembly. So what on earth is SQL.Interop.065.DLL?

Other assemblies contain the same .module reference, and ironically they can point to unmanaged libraries, like kernel32.dll. This is not the same as the unpopular .NET Modules. A .NET Module is, in short words, an assembly without a manifest. So what is SQL.Interop.065.DLL? It’s not a valid Win32 PE or PE+ file either. A look at it with the Dependency Finder tells us it’s compiled for the ARM architecture and it’s Subsystem is Windows CE 2.0+.

Here’s a screen grab.

For those curious, coredll.dll is the equivalent of kernel32.dll on a Windows CE system, but since I am using Windows 7 it obviously won’t be found on my desktop.

What does this mean? Why is a managed assmebly referencing an unmanaged assembly? Well, it isn’t exactly a “reference”. What happens is, whenever you do platform invoke on a DLL, the DLL is added as a “module” in the manifest. Let’s try it. Make a new C# project and throw in this code:

   1: [DllImport("user32.dll")]

   2: [return: MarshalAs(UnmanagedType.LPTStr)]

   3: internal static extern string CharUpper

   4:     (

   5:         [In, MarshalAs(UnmanagedType.LPTStr)] string lpsz

   6:     );

Now look at our manifest in ILDASM, we have this:

   1: .module extern user32.dll

I’m not entirely sure why it was quoted in the first example and not in the second, or if it has any significance.

The reason at this point is pretty obvious: We’re trying to do Platform Invoke on a library that my processor is incapable of handling.

Makes sense, but I am a bit bummed. Is SQL CE a better solution? Since SQLite is retargetable, can I use different versions of SQLite depending on the platform? I’m not sure right now, but I’ll be sure to follow up. If you have a working solution for this I would love to hear it, too.

Thoughts on Office 2010 Technical Preview

While it was leaked out via a Bit Torrent, as most things do these days (erm, Windows 7 “RTM” as people are calling it) I decided to wait until it was available to me under Microsoft Connect. Yesterday I got the chance to download and install it. I wanted to share my thoughts on it as well.

Native x64

Yes, Office 2010 is now available to run x64. This surprised me actually, that’s a huge undertaking and I didn’t immediately see the benefit. However, people that spend their lives doing analysis and equations in Excel that have a x64 operating system will appreciate this. One thing that does interest me is how this affects 3rd party tools. 64-bit program cannot load a 32-bit library, and Office is inherently  COM oriented, so I wonder how this will play with plug ins. Most Virus Scanners offer some plug in for scanning a document as it is loaded. Macro enabled documents are a long time distributor of viruses. Office 2007 took a small bite out of that by changing the document’s extension for macro enabled documents as well as tightening the security.

Initially I was also worried about some of the downsides of x64 – mainly pointer bloat. Pointers in x64 are 8 bytes wide, as opposed to 4 for x86. This can lead to additional memory use by the application. I didn’t see any obvious memory issues in any of the office programs though.

Outlook 2010

Right off the bat I noticed they fixed something that was really annoying, and that is if you use Outlook Anywhere (Exchange over HTTP) it would not remember your password. Every time Outlook started, you had to enter your password, no choice. My Windows password is pretty strong, so someone would have to get on my Windows account first before they could read my email. However since Outlook forced me to type my password every time, it also forced me to create a somewhat weak password – at least one that I could remember. Since we also have Outlook Web Access, someone could have gotten into my email through the web client if they guessed my password. Now I am able to keep my Exchange password pretty complex and only have specific devices – which are always secured with their own password – remember it.

Aside from that addition they also did a little interface revamping, specifically the addition of the Ribbon. It looks more polished, and allows you to access more features. It was something that also irritated me in Office 2007 was the inconsistent UI in the Office Suite. Notably, Outlook and Visio were missing the Ribbon – both of which have been fixed.

Outlook 2010 does a better job of organizing your email now as well, as opposed to a linear timeline of when it was received. It takes a bit of a GMail approach to it by grouping related emails into a conversation-like format. It was a bit confusing at first, so here is how I think it works. To the left of all grouped messages are an expanding arrow to allow you to view the conversation. Initially, it only shows ones that are in the inbox. If you click it again then it will include the entire conversation – ones from Sent Items and the Inbox. Once you get the swing of it it makes finding emails a lot easier. If you get a lot of emails in one day, you can catch up on what was already said much quicker. Of course, if the idea of grouping emails disgusts you, you can go back to the linear timeline – or any other sorting mechanism.

Something else that is new is the idea of Quick Steps – and I really like this. It allows you to create predefined actions on a message, say “Move to Folder Y and mark it as read”, or “Forward this to Tom and send it”. Here’s the kicker – you can assign keyboard shortcuts to it as well. My boss wanted this feature so badly he even took to writing a VBA Script that does it for him in Outlook 2007.

Maybe it’s me, but it appears that some basic artwork for icons are missing in the Technical Preview and are substituted with orange dots.

Small feature, but cool. Maybe it did this in 2007 – but I couldn’t find it – it now tells you how much of your space quota you are using. My quota is 500 MB and I am using 80. This probably only works for Exchange, possibly anything IMAP. I’ll have to test it.

Composing emails hasn’t changed that much. Aside from the ribbon, some other small features like “insert a screenshot” are now there as well.

Stability overall seems good. My mailbox isn’t huge so I can’t really vouch for how good it is with performance, but it is still a COM based application, and thus at the root a single threaded application. It didn’t hang on me though – I’ll update if it does.

One final thing, which I hope they fix, is that Outlook Today screen. That screen hasn’t changed since Outlook 97 or 2000, and it definitely feels worn and old with the new slick Ribbon UI. There’s so much potential with that screen but it isn’t being used.

Perhaps there are some additional things I missed. I can’t test it against Exchange 2010 so I don’t know if using Exchange 2010 will unlock some additional goodies.

Word 2010, Excel 2010, PowerPoint 2010

Word and Excel are important applications to many, and it is unfortunate for me to say, at least so far – I can’t see anything different. The UI is updated to the new Office Tab rather than the Button – but the look and feel are the same. Startup is lightning fast though – I don’t even have time to see the splash screen. It seems to work with large documents better and not get sluggish like it use to. Maybe that is a new found power of x64, but I like it!

I’ll keep playing around and update if I find anything new.

Updates

One of my buddies pointed out to me that Office 2010 has better support for the Open Office formats. Though 2007 also supported it, many people called it incomplete and often saved it in a way that only Office could open (thus defeating the point of being “Open”). I would be a little surprised, but not completely, if this is a 2010 feature and not also an update to 2007 by means of a Service Pack.

I’m not sure if it is because it is coming with Windows 7 or Office 2010, but there are a few new fonts as well.

ASP.NET AJAX 4.0 Client Template Rendering – The Observer Pattern - Part 2

We last left of here, http://msmvps.com/blogs/vcsjones/archive/2009/06/07/asp-net-ajax-4-0-client-template-rendering.aspx about a simple introduction to the AJAX 4.0 Client Template Rendering. I’d catch up on that one if you haven’t read it before you read this one.

Our last example was pretty simple. We took a chunk of JSON, and displayed it in an unordered list with simple JavaScript and the AJAX Framework. Let’s take this a step further with actually manipulating the data. The framework allows manipulation of data, and it does this with the observer pattern. The definition of the observer pattern is a bit weak, but Microsoft is advertising this as “a true implementation of the observer pattern” rather than a notification pattern that uses constructs like events. This leads a new chunk of the AJAX Framework, Sys.Observer. We need to understand how this pattern works before we can manipulate data.

This allows the AJAX Framework to watch, or “observe” for changes on certain things. This can be a DOM element or any JavaScript object. In our case, we are working with a collection. The simplest means of working with our collection is using the Sys.Observer.add() method. This allows you to add an item to a collection, and also notifies the AJAX Framework that the collection was changed. If the AJAX Framework has something bound to our collection, like our unordered list, it knows it needs to update the list.

Take a look at our observer in action:

   1: function add() {

   2:     Sys.Observer.add(descriptions, { Name: "Cat", Description: "Cats make a meowing sound" });

   3: }

This code sample is build on the code in the previous blog entry.

When we call “add()” we are adding a new item to the descriptions collection. Not only does this add it to the collection, it also notifies anything that is using the descriptions collection that it was notified, and the AJAX Framework correctly “rebinds” the UI, in our case the unordered list.

The Observer class has a lot of additional functionality that you would expect for manipulating a collection, remove, removeAt, etc. The full details are outlined in MSDN.

If we add an input element that calls our add function, we can see it in action:

   1: <input type="button" value="Do Add" onclick="add()" />

Neat huh? It took little effort to get us this far. The Observer pattern is pretty powerful as well. Using the observer pattern, we can also create our own listeners and react appropriately when the observer makes a change to an object, and it goes far beyond working with collections.

We’ll dig further into the observer pattern in the next post.

ASP.NET AJAX 4.0 Client Template Rendering

Recently I have been dabbling with the ASP.NET 4.0 Framework. There were two things that immediately got my interest:

1. An actual new version of the CLR. More of come on that later…
2. AJAX 4.0

Now, I didn’t mean to say I was pumped an excited about it, but a certain element caught my attention, and that is Client Rendering. This basically, allows you to bind simple HTML to JSON or a simple JavaScript array. The idea is interesting, but I wasn’t sure how well it’d play.

To get started, I looked here: http://www.asp.net/learn/whitepapers/aspnet40/. This is a brief overview of the application. Not entirely informative, but it got me started. I fired up Visual Studio 2010 and began to tinker.

To start off, I wanted to build something simple. No JSON or Web Service calls, basics! I’m a slow learner. So, what exactly am I doing with this? Given the scenario where I have a JSON object with some information on it, I want to display it on the page. Now, there have been a few ways of doing this in the past. jQuery makes UI component for adapting JSON objects to tables, ordered elements, etc. The Microsoft implementation isn’t too different. It defines a binding for some sort of repeatable HTML element. Tables have repeating rows, unordered lists have repeating list items, etc. Unordered lists are pretty simple, so lets start there. Let’s say we are given a JSON object similar to this:

   1: var descriptions = [{ Name: "Foo", Description: "Foo is a great product." }, { Name: "Bar", Description: "Bar is OK, but not great."}];

Pretty simple. We have 2 objects in a collection with a Name and Description property. Now we want to show them in in an unordered list.

First, to get started, we need a ScriptManager in our page. This wires in all of Microsoft’s AJAX scripts for us.

   1: <asp:ScriptManager ID="MyScriptManager" runat="server">

   2:     <Scripts>

   3:         <asp:ScriptReference  Name="MicrosoftAjaxTemplates.js" />

   4:     </Scripts>

   5: </asp:ScriptManager>

I just plopped this right after the <body> tag. Notice the script reference is simply working on name. One of the new features of AJAX 4.0 is Microsoft broke apart the AJAX Framework JavaScript into different libraries. This allows you to selectively use the libraries you need and avoid large chunks of script you don’t need. In this case, we need the AJAX Templates script.

The AJAX functionality we will be using is an attached DataView. The scripting library works by attaching itself it an HTML element. In our case, it’s a unordered list. The attachment is done with a simple attribute on the element, sys:attach and then specify what you want to attach. Before we can go there, we need to tell our browser certain things, mainly namespaces. The body tag will contain certain namespace imports, similar to how WPF XAML imports namespaces. We want the DataView and the Sys namespace, so we’ll throw those in there:

   1: <body xmlns:sys="BLOCKED SCRIPTSys"

   2:     xmlns:dataview="BLOCKED SCRIPTSys.UI.DataView"

   3:     sys:activate="*">

There is one additional attribute, sys:activate that is in there. We’ll touch on that in a bit. So now we want to build our unordered list and attach a dataview to it. Let’s put together our unordered list and go through it.

   1: <ul id="descriptionsList" sys:attach="dataview" class="sys-template" dataview:data="{{ descriptions }}"> 

   2:     <li>    

   3:         <h3>{{ Name }}</h3>

   4:         <div>{{ Description }}</div>

   5:         <hr />

   6:     </li>

   7: </ul>

Notice the sys:attach attribute, and we are attaching a dataview to our unordered list. The other important attribute is the dataview:data. This is basically what data you want to bind to. Finally, we have a list item that is our “template” for what AJAX should use for all of the items. We denote a binding using double curly braces {{ }}. So {{ Name }} binds to the Name attribute in our JSON, etc. The dataview:data attribute is bound to our descriptions JSON object we declared.

There is one little piece left, and that is the class of sys-template. This is just a minor styling that sets display: none. This hides the entire element and the AJAX Framework will set the display properly once it has done fully loading all of the data and rendered it.

All together, it renders something like this:

All together, this was all of my source:

   1: <%@ Page Language="C#" AutoEventWireup="true" CodeBehind="ClientTemplates.aspx.cs" Inherits="DataControls.ClientTemplates" %>

   2: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

   3: <html xmlns="http://www.w3.org/1999/xhtml" >

   4: <head runat="server">

   5:     <style type="text/css">

   6:     .sys-template

   7:     {

   8:         display: none;

   9:     }

  10:     </style>

  11:     <script type="text/javascript">
   1:  
   2:         var descriptions = [{ Name: "Foo", Description: "Foo is a great product." }, { Name: "Bar", Description: "Bar is OK, but not great."}];
   3:     
</script>

  12: </head>

  13: <body xmlns:sys="BLOCKED SCRIPTSys"

  14:     xmlns:dataview="BLOCKED SCRIPTSys.UI.DataView"

  15:     sys:activate="*">

  16:     <form id="form1" runat="server">

  17:         <div>

  18:             <asp:ScriptManager ID="MyScriptManager" runat="server">

  19:                 <Scripts>

  20:                     <asp:ScriptReference  Name="MicrosoftAjaxTemplates.js" />

  21:                 </Scripts>

  22:             </asp:ScriptManager>

  23:             <ul id="descriptionsList" sys:attach="dataview" class="sys-template" dataview:data="{{ descriptions }}"> 

  24:                 <li>    

  25:                     <h3>{{ Name }}</h3>

  26:                     <div>{{ Description }}</div>

  27:                     <hr />

  28:                 </li>

  29:             </ul>

  30:         </div>

  31:     </form>

  32: </body>

  33: </html>

And I didn’t write a single line in the code-behind, either.

This demonstrates the basic functionality of the binding. Next we’ll look at how to actually bind to dynamic data from a web service call, and manipulating the data.

Update: If you want to download a Visual Studio 2010 project, I’ve uploaded it here: http://bit.ly/VCWFd. It looks like Community Server is blocking some of the code snippets thinking it is XSS.

Centering Content Properly

As an ASP.NET Developer, not a designer, it’s pretty easy to write crappy HTML. As I have come to learn, content designers will hate you for that. One that has always fooled me is how to properly center content. Centered content is nice sometimes. It put information right in front of the users face, exactly where they expect it. HTML has come a long way since 2000, and it’s worth looking at some of the right and wrong ways to do it. Your designers will love you for it.

The oldest, and simplest way to center content is to use a <center> tag. Designers hate this. It works differently between IE 6 & 7, and 7 will display it differently depending on the DOCTYPE. In quirks mode, it has an interesting effect. If you center a DIV tag in quirks in IE, the DIV will be centered, but the DIV’s contents will be left aligned. That’s actually the behavior we want, but unfortunately not a good option. It only works that way in IE, and your forced to use quirks. If you use standards mode, the center tag behaves correctly: everything is centered, even the content of the DIV, which is not what we are looking for.

To that extent though, you could use a center tag, and just left align the contents of the DIV. Something like this:

   1:  <html>

   2:      <body>

   3:          <p>Some other content here, maybe a header.</p>

   4:          <center>

   5:              <div style="width: 500px; border: solid 1px black; text-align:left;">I want this DIV centered, but the text still left aligned.</div>

   6:          </center>

   7:          <p>Some other content here, maybe a footer.</p>

   8:      </body>

   9:  </html>

Of course I would put my CSS in an external file, otherwise you’ll really catch hell from a designer. Needless to say, it gives the right effect:

But this really isn’t the best choice. Depending on what your designers have to work with, for say, a product, they may only have access to the style sheets. A center tag is not very flexible, and cannot be put into an external file like a CSS stylesheet.

So what other options are there? Tables would work, but that’s even worse than the center tag, and has the same problem as the center tag: It’s not CSS. So how do you do this using CSS? The trick is auto margins. It’s a fantastic part of the CSS layout that I find surprisingly, not enough developers know about. The HTML looks like this:

   1:  <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

   2:  <html>

   3:      <body>

   4:          <p>Some other content here, maybe a header.</p>

   5:          <div style="width:500px; margin: 0px auto; border: solid black 1px;">

   6:              test

   7:          </div>

   8:          <p>Some other content here, maybe a footer.</p>

   9:      </body>

  10:  </html>

Note that IE, even the famed standards compliant IE 8, will NOT render this correctly without the correct DOCTYPE. IE will leave it left aligned, but Gecko based browsers (Firefox) will still render it correctly.

So why is this the best solution (IMHO)? It’s completely controlled by CSS, and content designers love that.

What’s essentially happening here is the browser is automatically computing the margins for both the left and right side, giving it the appearance of being in the center.

Enum Friendly

Have you ever seen a huge switch on an enum just to return a string? Enumerations can be tricky to deal with, mostly from the aspect of displaying them back to a user. So I whipped up a little code to make it easier to display enumerations, a little extension method. Consider this enumeration:

public enum StatusEnumeration
{
    Normal,
    NotRunning
}

If I want to display this back to a user in English, I can switch on the value, and return “Not Running” when the value is NotRunning. This can be a tad tedious, so I created an attribute to decorate an enumeration. I call it the RenderAsAttribute:

[AttributeUsage(AttributeTargets.Field, AllowMultiple = false, Inherited = false)]
public sealed class RenderAsAttribute : Attribute
{
    private readonly string _renderAs;

    public RenderAsAttribute(string renderAs)
    {
        _renderAs = renderAs;
    }

    public string RenderAs
    {
        get { return _renderAs; }
    }
}

All you do is put this on the enumeration’s values, like so:

public enum StatusEnumeration
{
    Normal,
    [RenderAs("Not Running")]
    NotRunning
}

Notice I put the “RenderAs” on NotRunning. Now for the cool part, a static extension method to make this attribute useful. It would be used something like this:

string notRunningAsString = StatusEnumeration.NotRunning.Render();

Notice how I just call “Render()” on a the value of the enumeration.

public static class EnumRenderer
{
    public static string Render(this Enum value)
    {
        FieldInfo[] enumerationValues = value.GetType().GetFields(BindingFlags.Static | BindingFlags.Public);
        foreach (var fieldInfo in enumerationValues)
        {
            if (value.CompareTo(fieldInfo.GetValue(null)) == 0)
            {
                var attributes = fieldInfo.GetCustomAttributes(typeof(RenderAsAttribute), false).Cast<RenderAsAttribute>();
                if (attributes.Count() == 1)
                    return attributes.Single().RenderAs;
            }
        }
        return value.ToString();
    }
}

This uses refection to look up the value on the enumeration. It’s a bit slow as far as code performance is concerned, but you can easily add caching to it by Type and value. Or, instead of using hard coded strings, you can use the string as a resource key and look up the string in a satellite assembly.

MVP 2009 and other stuff

Recently for April 1st I was re-awarded as an MVP for 2009, and I’m happy to be able to keep doing what I do for the community.

So where has my blog been lately? Well I’ve been focusing on other things at the moment, but I am still blogging. I recently wrote a blog post for my company's blog about refactoring, and getting your feet wet with it. We’ll see where the series goes too. You can read my blog post about refactoring here:

http://www.thycotic.com/refactoring-code-a-programmers-challenge

TDD Training Course

Ever want to learn more about Test Driven Development? This coming weekend on Saturday, January 10th Thycotic Software will be hosting a free one day TDD Training Course. This course is ideal for novice TDD developers or people that are just interested in getting into it. Some areas that will be covered are an introduction to TDD development, the philosophy of TDD development, and a brief introduction to Mock Testing. Later in the near future a full training session will be available, and with any luck if this one turns out well we'll keep doing it!

It's a bit of a plug, but the offer is so great who wouldn't want to hear about it?

Managing IIS Pipeline Mode for Backward Compatibility

Ever since the induction of IIS 7, there came a cool new feature for developers to leverage called Integrated Pipeline. This feature in a short explanation closely couples ASP.NET and IIS more closely. It allows allows writing of IIS Modules in managed code, how neat is that? This also slightly changes the behavior of ASP.NET, such as introducing new events on the HttpApplication (Global.asax) and the event cycle. A classic example is where in Integrated mode you cannot access the current request using HttpContext.Current.Request during theApplication_Start event, which makes sense.

Now for obvious reasons this can break existing applications, and cannot be used for applications that will be installed in mixed environments, such as Server 2003 and Server 2008. Fortunately, IIS 7 make it easy to switch it back back to its old behavior using the Classic Pipeline by configuring the IIS AppPool. Most commonly, if you make a web-based product then your application will usually require Classic Pipeline to stay backwards compatible.

Even if you have a detailed instruction manual, this can be an easy step to miss - and results in some icky error messages. Wouldn't it be nice if you could display an nice error message to your customers that their pipeline is configured improperly?

Well fortunately there is a way to do that. Ironically, we will be using part of the Integrated Pipeline to get that accomplished with the help of an HttpModule.

What we will be doing is writing an HttpModule that will basically stop the current request, write out a friendly message that links to a Knowledge Base article, and only do this if our application is running in Integrated Pipeline. The module to get this done is pretty simple, we would write it like any other HttpModule. Here is the code for mine:

using System;
using System.Web;

namespace MyWebApplication
{
    public sealed class IisPiplineCheckModule : IHttpModule
    {
        public void Init(HttpApplication context)
        {
            context.BeginRequest += context_BeginRequest;
        }

        private void context_BeginRequest(object sender, EventArgs e)
        {
            HttpContext.Current.Response.ClearContent();
            HttpContext.Current.Response.WriteFile(VirtualPathUtility.ToAbsolute("~/pipeline.htm"));
            HttpContext.Current.Response.End();
        }

        public void Dispose()
        {
        }
    }
}

As you can see, there is nothing particularly difficult. It just hooks into the BeginRequest event of the Application, and writes a file called pipeline.htm to the response stream and closes the response. Pretty simple, right?

Well the final trick is getting this to work only in Integrated Pipeline. Unusually, there is no easy way to do this with the .NET Framework (that I was able to find). However we will leverage the web.config to accomplish this.

For IIS 7, there is a new section in the web.config file called <system.webServer>. This is one of the sections that IIS 7 pays attention to. What we will do is introduce our module into IIS 7's Integrated Pipeline, but not into the old <httpModules> that has been existing for a long time. Let's look at our new section and examine it.

<system.webServer>
    <validation validateIntegratedModeConfiguration="false" />
    <modules>
        <add preCondition="integratedMode" name="IisPipelineCheckModule" type="MyWebApplication.IisPiplineCheckModule,MyWebApplication" />
    </modules>
</system.webServer>

There are two interesting points here. The first is actually adding out module. Notice the preCondition. This is telling IIS 7 that our module should only be running in Integrated Pipeline by specifying the integratedMode flag. For the Type we just set the full namespace and class to our HttpModule and the assembly it is located in.

The second interesting point is that if you have standard HttpModules registered in the configuration/system.web/httpModules section IIS 7 will point out that there are modules in there that are not in the system.webServer configuration. We can, if we want, disable that through the validation element.

And ta-da, we now have a module that writes out a file, but only for Integrated Pipeline. You can customize the pipeline.htm file as much as you'd like. As far as I can tell, this is the only easy way to determine if your application is in Integrated or Classic Pipeline. The alternative is to WMI query the IIS metabase, which is a little more high risk and more complex.

A Step Too Far

Occasionally, I have a bit of a compulsive behavior. When presented with a challenge, I usually won't give up until I have a working answer... and sometimes that answer get's a little crazy. Here's one of my more recent journeys down that path.

I was asked, "Hey Kevin. I am a method that accepts a type parameter, 'T'. Is there a type constraint I can add to T so that I may sum them?"

As an example, consider this code: 

public string FormatNumber<T>(T t1, T t2) where T:IFormattable
{
    T sum = t1 + t2;
    return String.Format("{0:N2}", sum);
}

This sample is not very useful by itself, and it just an example.

The problem is, t1 and t2 cannot be added because the compiler cannot guarantee that t1 and t2 can even be summed. Unfortunately, generics do not have a way to add a constraint for operators yet.

The eventual solution was that the code needed some rethinking. The need for generics and adding them was legitimate, but not worth going over. Either way, I started wondering how this could be possible without doing something like this:

public T Add<T>(T t1, T t2)
{
    if (t1 is int)
        return (T) (object)(((int) ((object) t1)) + ((int) ((object) t2)));
    if (t1 is long)
        return (T) (object)(((long) ((object) t1)) + ((long) ((object) t2)));
    if (t1 is float)
            return (T) (object)(((float) ((object) t1)) + ((float) ((object) t2)));
    throw new NotImplementedException("Can't do addition.");
}

That just seems unattractive to me. So I starting thinking... and thinking... and it spiraled down from there. The solution that I came up with is very unattractive, slow, and nuts.

My solution? Dynamically generate an assembly using Reflection.Emit. It works, but not very elegant. Here it is:

public static T Add<T>(T t1, T t2)
{
    if (!typeof(T).IsPrimitive)
    {
        throw new Exception("Type is not primitive.");
    }
    ModuleBuilder moduleBuilder = assemblyBuilder.DefineDynamicModule("MainModule");
    TypeBuilder typeProxy = moduleBuilder.DefineType("AdditionType", TypeAttributes.Class | TypeAttributes.Public | TypeAttributes.Serializable);
    MethodBuilder methodBuilder = typeProxy.DefineMethod("SumGenerics", MethodAttributes.Static, typeof(T), new[] { typeof(T), typeof(T) });
    ILGenerator generator = methodBuilder.GetILGenerator();
    generator.Emit(OpCodes.Ldarg_S, 0);
    generator.Emit(OpCodes.Ldarg_S, 1);
    generator.Emit(OpCodes.Add);
    generator.Emit(OpCodes.Ret);
    Type adder = typeProxy.CreateType();
    MethodInfo mi = adder.GetMethods(BindingFlags.Static | BindingFlags.NonPublic)[0];
    return (T)mi.Invoke(null, BindingFlags.Default, null, new object[] {t1, t2}, null);
}

Not to mention, that you have to tell your AppDomain how to resolve the type, and that involves a little magic with the OnAssemblyResolve of the AppDomain:

private static Assembly OnAssemblyResolve(object sender, ResolveEventArgs args)
{
    return args.Name == assemblyBuilder.FullName ? assemblyBuilder : null;
}

Sometimes, it just feels good to write crazy code and get it out of your system, before you really do start writing code...

XP SP3 and Internet Explorer

Perhaps it's just me, or maybe I missed something. I was a little excited about Service Pack 3, particularly the Network Level Authentication feature made it into Remote Desktop, which means that I can force the NLA requirement for Windows Server 2008 and Terminal Service Gateways. Also the added support for WPA2 is a huge security advantage for people with wireless networks. I was happy to be able to change my wireless router to use WPA2 and still be able to support my machine that is still running XP, and this is also something Network Administrators look for.

Not to mention, it was pretty painless to install.

However, I also had Internet Explorer 8 Beta 1 installed. Beta 1 was interesting, but due to some stabilities issues, I thought it was time to remove it and hope for the best in Beta 2. However, when looking at Add / Remove Programs, I saw that I could not remove it. The Uninstall button was missing. I did a little bit of digging, but I didn't find anything online that caught my attention.

At Tech Ed 2008 though, I did talk to Jane Maliouta on the IE 8 team, and she explained to me that Service Pack 3 caused IE 8 to be un-installable. The only known work around is to remove Service Pack 3, then remove Beta 1, then put Service Pack 3 on.

Please read Jane's blog entry on how else XP SP3 effects Internet Explorer. Service Pack 3 also prevents you from un-installing Internet Explorer 7 as well.

http://blogs.msdn.com/ie/archive/2008/05/05/ie-and-xpsp3.aspx

Tech·Ed 2008

Tech Ed 2008 is the first Tech Ed to be split into two separate conventions, one for IT Professionals and one for Developers. For 2008, I will be attending the 2008 IT Professional conference from June 10th - 13th in Orlando, Florida. I'll be on the convention floor with my company, Thycotic Software, showing off our flagship product, Secret Server. If you're going to be there, be sure to come check our booth out!

DocBook

Let's take a break from the text encoding idea real quick so I can talk about a new tool that I recently got into..

One of the things that every product needs, regardless of how simple it is to use, is good documentation. It's not fun, it takes time, and isn't technically intriguing. Regardless, it has to be done. The part that myself and team members have struggled with is a tool take makes it easy. We looked at a few commercial applications such as RoboHelp, but it always left me the impression we were rabbit hunting with a Barrett M107 .50 rifle. Our requirements were pretty simple:

• Easy to use
• Text based - This makes differentials and merging easy
• Reasonably priced
• Able to produce different types of documents (HTML, PDF, etc)

We finally settled on what is the best solution (not to mention, it's open source and free) called DocBook. It's based on XML, and does have a standard. XML is extremely flexible, and their output is generated by XSL transformations, so we can easily customize the output to meet our requirements. We started using the e-novative DocBook Environment, which gives you a simple command line environment for compiling your DocBook books. It too uses a GPL license, so you can customize it to your needs.

A simple book looks something like this:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE book
  PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" "file:/c:/docbook/dtd/docbookx.dtd"
  [
    <!ENTITY % global.entities SYSTEM "file:/c:/docbook/include/global.xml">
    %global.entities;

    <!ENTITY % entities SYSTEM "entities.xml">
    %entities;
]
>
<book lang="en">
    <bookinfo>
        <title>My First Book</title>
        <pubdate>May, 2008</pubdate>
        <copyright>
            <year>2008</year>
            <holder>Kevin Jones</holder>
        </copyright>
    </bookinfo>
    <chapter>
    <title>My First Chapter</title>
    <sect1>
        <title>How to crash a .NET Application</title>
        <para>Call System.AppDomain.Unload(System.AppDomain.CurrentDomain)</para>
    </sect1>
    </chapter>
</book>

Pretty simple, right? Each book can be broken down into separate chapters, which are broken down into sections, then paragraphs. It takes care of some dirty work for you, such as maintaining a Table of Contents for you. It offers a lot of other standard features as well, embedding graphics, referencing other places in the document.

Since DocBook is capable of understanding external entities, I can place chapters, sections, any part of the document that I want into another file and create an <!ENTITY ... > for it.

Compiling it is pretty easy. From the e-novative environment, just use the docbook_html.bat for docbook_pdf.bat to create your generated output, something like this:

>C:\docbook\bat\docbook_html.bat MyFirstBook

MyFirstBook is the name of the project in the projects folder, which is all automatically created for you by the docbook_create.bat script. Using the compiler, the out-of-box HTML template looks like this:

(click for full image)

There you have it, a simple documentation tool. Not very pretty at the moment, but of course it's easy enough to theme it to your company or product by changing the XSL.

Text Encoding (Part-1)

I was recently on the ASP.NET Forums and a member was asking, "How can I figure out the encoding of text?" and that got me thinking. There should be a reasonable way to do this, right? It's a useful thing to know. First, we need a little background on how text is encoded into bytes.

Long ago, back when 64K of memory was a big deal, characters took up a single byte. A byte ranges from 0 - 255, which allows us to support a total of 256 characters. Seems like plenty, no? English has 26, 52 for both cases, 62 with numbers, 92 with punctuation, and a few extra for line breaks, carriage returns, and tabs. So about 100, give or take a few. So what's the problem?

Well, this worked great and all, but other languages use different characters. The Cyrillic language by itself supports 33 letters. This is where encoding was introduced. In order to support multiple character sets, what each byte meant was determined by its encoding. This was done simply by knowing what encoding was used.

In today's world, where that average calculator has more memory than PCs did long ago, we now also use 2 byte encoding. That means that we can support 255 to the second power of characters, or 65,536. That is enough to support all languages in a single encoding, even though it takes up double the space. Problem solved, right? Not exactly.

While in this day and age we support double byte encoding, there are still other factors involved, such as the endianness (the order of the bytes. Big endian is backwards). Even then, there is still a lot of legacy data to support that is still single byte.

Say I give you a big binary chunk of data, and I tell you to convert it to text. How do you know which encoding is used? How do you even know which language it is in? I could be giving you a chunk of data using IBM-Latin. So how do we figure this out? Some smarts and process of elimination. Let's start with things we know.

All of the non single-byte encodings have what's called a Byte Order Mark, or BOM for short. This is a small amount of binary data pre-appended to the rest of the data that identifies which encoding it is. In .NET world, this is called the Preamble. Since the BOM is an ISO standard, it is always the same for the encoding used regardless if you are using .NET, Python, Ruby on Rails, etc. We can look at our data and see if the BOM can tell us.

To achieve this in .NET, we will be using most of the classes in the System.Text namespace. Specifically, the Encoding class. An instance of the encoding class has a method called GetPreamble(). Which will give us our BOM for that encoding. A BOM can be from 2 - 4 bytes, depending on the number of bytes used in the encoding. Remember when I said two bytes would be plenty? Well I fibbed, since there is an encoding called UTF-32 that supports 4 bytes (a whopping 4.2 billion character support).

We can then check our data to see if it starts with the BOM.

private static bool DataStartsWithBom(byte[] data, byte[] bom)
{
    bool success = data.Length >= bom.Length && bom.Length > 0;
    for (int j = 0; success && j < bom.Length; j++)
    {
        success = data[j] == bom[j];
    }
    return success;
}

So lets look at this method. This method takes our data, and a BOM, and determines if the data starts with the BOM. There are a few assumptions:

1. The data length is always greater than or equal to the BOM. If it is not, then there is no BOM at all, and we'll cover that in a bit.
2. The BOM's length is always greater than zero.

So let's put it to use (assume the local data is a byte[]):

foreach (EncodingInfo encodingInfo in Encoding.GetEncodings())
{
    Encoding encoding = encodingInfo.GetEncoding();
    byte[] bom = encoding.GetPreamble();
    if (DataStartsWithBom(data, bom))
        return encoding;
}

Here, we get all of the encodings that .NET knows of, and looks to see if our data byte array starts with that encodings BOM. If the encoding has no BOM, the DataStartsWithBom method will handle that with the bom.Length > 0 on the 3rd line. Once we know the encoding, we can decode it. You have to ensure that you don't actually try to decode the BOM itself:

encoding.GetString(data, bom.Length, data.Length - bom.Length);

Pretty straight forward so far, right?

Yes? OK let's move on. What about the case where we can't figure it out by the BOM? Most encodings don't have a BOM, only the UTF encodings do. ISO and OEM encodings, do not.

This is where it gets tricky, and where some pretty complex algorithms can come into play. The most important piece of information that you can have at this point, is knowing which language the text is in. With that, we can take a reasonable stab at which encoding is it.

.NET supports languages through the System.Globalization.CultureInfo class. This class will be very useful from here on forward. Let's take baby steps on attacking this problem, and while we don't know everything, we can use clues.

Each language has what's called an ANSI encoding. This a standard encoding used for that language assigned by the American National Standards Institute. The ANSI encoding is always a single byte encoding. This seems like a reasonable place to start.

We can get this Encoding by calling cultureInfoInstance.TextInfo.ANSICodePage. This only gives us the numeric code page (an identifier), but it's simple enough to create an instance of the Encoding class with the code page by calling Encoding.GetEncoding(int codePage).

How do I figure out the language? Chances are you know what language your users are using, or at least most of them. A case where you wouldn't know is screen-scraping. That can be figured out by looking at the encoding of the response. You can do that by looking at the ContentEncoding property off of the HttpResponse instance.

In most cases, this will probably work. By no means am I saying, "this will always work" in fact, there are a lot of bases that I haven't covered that I hope to in future blog posts. There are other code bits out there that do this already, and do a good job, but it's always good to know how it actually works, and fully understand the problem you are trying to solve.

So what'll be in part 2? How to decode text without knowing the language, and maybe in part two (part 3?) lossy decoding.

Too Little Too Late?

I booted up my PC today and saw this nice message from Adobe telling me that there was an update for my Flash installation.

I couldn't help but notice that one of the highlighted features was support for HD content. I can't help but feel that this is in response to Silverlight's support for High Definition content. I wonder though, is it too little too late? I've heard a lot of stories from people about switching from Flash to Silverlight just for the support for HD. Now don't misunderstand me, I think this will be a huge hit for the Flash community, and definitely merits use.

This means to me that Silverlight is doing something right, and is going to be able to hold its ground.

CMAP User Group Presentation

On Tuesday, May 6^th at the CMAP User Group Meeting will be Heroes Happen {here} Launch where I will be discussing the new features in SQL 2008 and Steve Michelotti will be discussing the new C# 3.0 Language Enhancements. If you are in the Baltimore area, I encourage you to come to the presentation to learn some cool new stuff.

Take a look at the meeting details here for more information and directions.