Avoid the volatile Modifier

[Update: 25-Jan-12 5:45 pm; fixed typo]

I was reminded recently of the misconceptions of the volatile modifier in C#, and I’d thought I’d pass along the recommendations of other’s that is tantamount to “avoid the volatile modifier”.  The volatile modifier in C# “indicates that a field might be modified by multiple threads that are executing at the same time” [1].

The first problem is that documentation.  What does that really mean to someone developing code that uses multiple threads?  Does that make the code “thread-safe”?  The answer is “maybe”; the real answer is “rarely”. Most people just stick volatile on a field because they think that’s what they need to do.

What volatile modifier does do to a field in C# is make all reads to the field use “acquire semantics” and all readswrites use “release semantics”.  Much clearer, right?  Acquire semantics means access is “guaranteed to occur prior to any references to memory that occur after it in the instruction sequence” and release semantics means access is “guaranteed to happen after any memory references prior to the write instruction in the instruction sequence” [6,2]

One of the problems with modern compilers [3] and processers and multithreaded code is optimization.  Within a block of code with no externally visible side effects the compiler is free to re-order instructions and remove instructions to result in the same visible side-effect.  “x+=1;x+=1;” can be freely optimized to “x+=2;” and “x+=1;y+=2;” can be reordered so that order of execution is effectively “y+=2;x+=1;”, for example.  The processor doesn’t optimize multiple instructions into one; but can re-order instructions and make the results of executed instructions visible to other cores/processors visible well after the instructions were executed (processor caching).  With fields, the compiler[3] has less freedom to optimize because side-effects to fields have more visibility—but the processor doesn’t discern between C# fields and any other bit of memory.

So, what does volatile really do for a field?  Realistically it prevents processor re-ordering and caching.  It tells the processor that all accesses to the value of that variable should come from or be made directly to memory, not the cache.

Not having a value cached by the processor so all other processors (and thus all other threads) can see what happens to the value of a variable seems like a really good thing though, doesn’t it?  In that respect, yes, it is a good thing.  But, let’s look at some of the drawbacks of volatile.

The syntax of volatile is that it just annotates a memory location.  In reality it really modifies the operations that occur on that memory location.  Code that operates on that memory location looks the same, regardless of whether it’s modified by volatile or not.  i.e. it’s unclear the code has different side-effects for multithreaded scenarios.  Another problem is that “volatile” is a very overridden word.  It’s used in C++ and Java; but it means something different (sometimes slightly) in each.  Moving back and forth from C++ to C# to Java could lead to using volatile incorrectly.  Volatile also assumes that all accesses to the field need to be protected from re-ordering.  Most of the time, there’s very specific times you want to force side-effects to a field to be made “visible” and, effectively, flush the processors cache to/from memory.  Flushing to the cache on every access is not very performant if you don’t need side-effects truly visible until specific times.  Plus, the volatile modifier means nothing if you pass the field by reference somewhere else. [4,5] There’s also limitations to what you can apply volatile to (e.g. you can’t have a volatile long).

What does volatile really mean from a code perspective?  Well, when you read from a volatile field effectively the following code is executed:

Thread.VolatileRead(ref field);

And when you write to a volatile field effectively the following code is executed:

Thread.VolatileWrite(ref field, newValue);

Both of these methods really just translate to a memory access and a call to Thread.MemoryBarrier (before the memory access for read, and after for write).  What MemoryBarrier does for is is ensure that all cached writes are flushed to memory at the call to MemoryBarrier (i.e after MemoryBarrier, any cached writes that occurred prior to MemoryBarrier are flushed and any cached reads are abandoned).  So, if you’ve noticed there’s nothing field-specific about Volatile[Read|Write], it makes everything flush, not just the field in question.  The performance aspect of the volatile modifier comes into play when you have multiple volatile fields within a class, it’s really only the last one that needs to use VolatileWrite when writing and the first one to use VolatileRead when reading. To steal some code from Jeffery Richter:

m_value = 5;
Thread.VolatileWrite(ref m_flag, 1);

is just as thread-safe as:

m_value = 5;
m_flag = 1;

if m_value and m_flag were volatile.

Same holds true for VolatileRead:

if(Thread.VolatileRead(ref m_flag) == 1)
	Display(m_value);

is just as thread-safe as the following with volatile fields:

if(m_flag == 1)
	Display(m_value);

I feel compelled to bring up invariants here.  When you’re dealing with modifications or accesses of multiple things that can’t operate atomically (i.e. the “validity” of m_value depends on the value of m_flag, making the two fields an “invariant” that can be accessed by a single instruction—i.e. not atomically).  While the above example ensures that changes to m_flag and m_value are made visible to other threads “at the same time” it doesn’t do anything to stop another thread from accessing the m_value before m_flag has been updated.  This may or many not be “correct”.  If it’s not correct, using lock or Monitor to model atomic access to such an “invariant” that isn’t natively atomic is a better choice.

On the topic of lock/Monitor.  It’s important to note that the end of a lock or the call to Monitor.Exit has release semantics and the start of the lock or Monitor.Enter (and variants) have acquire semantics.  So, if all access to a field is guarded with lock or Monitor, there’s no need for volatile or Thread.Volatile[Read|Write].

Using Monitor, lock, volatile, VolatileRead and VolatileWrite correctly shows that you understand what it means to be thread-safe and that you understand what it means to be externally visible and when in the context of your fields and invariants.

There have apparently been some discussions about the usefulness and applicability of volatile in C# as well as VB (and I assume C++11); but, I’m was not privy to those discussions and haven’t been able to find much in the way of reference to those discussions other than third-hand information…  Needless to say, VB doesn’t have anything similar to “volatile”, apparently for a reason.

I’m not saying there aren’t perfectly valid scenarios for volatile; but, look carefully at what you need; you probably could make better use of VolatileRead or VolatileWrite.  Just understand your needs and use what is correct—code on purpose.

[1] volatile (C# Reference)

[2] Acquire and Release Semantics

[3] CSC and JIT compilers.

[4] Sayonara volatile

[5] Aug 2011 Getting Started with Threading webcast with Jeffrey Richter (45MB)

[6] C# Language Specification (ECMA-334)

C#, Async, Limits, oh my!

One of the great sessions at Codemash was a dual-speaker session with Bill Wagner and Jon Skeet—Async from the Inside.

In that session Bill and Jon describe (in great detail) the state machine that the C# compiler generates when it compiles async code involving the await keyword.  When the Async CTP was released this state machine was one of the first things I noticed when I was reflecting through some generated code.  I too noticed the type of the state variable (int) and wondered, at the time, if that would be an issue.  All the information Bill, Jon and I portray is what we’ve learned from reflection, none of this is “insider information”.

Some background: a state machine is generated for each method that uses the await keyword.  It manages weaving back-together the asynchronous calls that you so lovingly linearly describe in an async method.  The state machine not only weaves together the various “callbacks” that it generates, but it also (now) manages the state changes of the data being modified between and by asynchronous calls.  Wonderfully complex stuff.

The state variable is basically an enum of the various states between async calls that the generated code can be in.  I won’t get into too much detail about that (you should have gone Codemash like all the other cool people :) but needless to say having a “limit” on the number of states a method can be in sounds a bit scary.

Of course Jon, at one point, brought the same thing up in the session about the “int” state variable.  This reminded me that I wanted to look into it further—not because I wanted to break the compiler (directly) but to know what the limits are, if any.

A couple of days later while I had some time waiting around at airports on my way home.  I thought I’d test my theory.  If you follow me on Twitter you probably saw some of my discoveries in real time.  For those of you who didn’t, here’s an aggregation of what I found.

First of all, the state machine represents the async states a single method can be in.  This is represented by a signed int.  the only negative value that seems [1] to mean anything is –1, leaving 2,147,483,647 states (or, roughly, await invocations) that can occur in an async method.

First glance, this seems disconcerting.  I quickly wrote some code to generate a method with 2,147,483,648 await invocations in it (that’s Int32.MaxValue + 1 for those without OCD).  Needless to say, that took a few minutes to generate on my laptop, and I have an SSD (which I’ve clocked at increasing my IO by about 7 times on average).  That generated a 45 gigabyte file (one class, one method).

Problems with the type int for the state variable started to seem ridiculous.  But, I didn’t stop there.

Now I’m doing everything outside of Visual Studio (VS) from this point on.  I’m running the app to generate the code from within VS; but everything else is from the command-line.  So, I run CSC on the 45 gig file honestly expecting and internal compiler error.  But, what happens is that I get a CS1504 error ‘file.cs’ could not be opened (‘Arithmetic result exceeded 32-bits. ‘).  Now, the state variable is 32-bits so, it sounds like that could be the culprit.  But, if you look at the error message, it can’t even open the file.  I tried opening the file in VS and it told me the file couldn’t be found…

Okay, at this point it’s seeming even more unlikely that a type of int for the state variable is even remotely going to be an issue.  But, now I’m curious about what 32-bit value has been exceeded.  My theory is now the number of lines in the file…  The compiler has to keep track of each line in the file in case it has to dump an error about it, maybe that’s the 32-bit value? I modify my code generation to limit the the number of await invocations so the number of lines in the file is 2,147,483,647 (kind of a binary search, if this works then I know it still could be the number of lines in the file). Same error.

The error isn’t from the # of lines.  Now my theory is that the overflow is from trying to allocate enough memory load the file (keep in mind, I’ve got 8 GB of RAM and I’m trying to load a 45GB file; but, I have yet to get an out of memory error).  So, I modify my code to generate a file that is approaching 2,147,483,647 bytes in size.  Things are much faster now…  I try again. Now I get the same CS1504 error but the message is ‘file.cs’ could not be opened (‘Not enough storage available to complete this operation’) (I’ve got 100 GB free space on the drive…).  Interesting.  I’ve lessened the data requirement and only now effectively getting “out of memory” errors.

Now I’m just looking for a file that the compiler will load—I’ve given up on some super-large number of await statements…  Longer story, short, I kept halving the size of the file until I reached about 270 megabytes in size then the compiler finally succeeded (meaning ~540 Megabytes failed).

At this point, I’ve successfully proven that a type of int for the state variable is not an issue.  If the compiler could load the 540 megabyte file and I somehow could use 8 bytes per await invocation (“await x;” for example) then I could never reach more than about 70,778,880 await calls in a single method.  Of course, I’m way off here; that number is even much lower; but 70,778,880 is about 3% of 2,147,483,647.  Clearly int is the smallest type that could store anything close to 70,778,880 or less…

Of course, I’m completely ignoring the fact that a 540 MB cs file is completely unmanageable in VS or a project in general; but, why get caught up in silly realities like that.

This state machine is almost identical to those generated by enumerator methods (yield return).  If we assumed that the async state machine generation code is “inherited” (by pragmatic reuse) from the enumerator method state machine generator, we can assume it has very similar limits (but even smaller)—meaning you’d never get close to overflowing its int state variable.

HTMYL.

[1] Again, this all observed; I’m surmising the compiler never uses any other negative value.

The TeamCity Database Migration Documenation Could Use Some Work

I have a client who wants all the benefits of ALM but doesn’t really want to spend the money on them.  They’re a start-up; so, I can respect that. As a result, I’ve been getting them up and running with some of the freely-available and open-source tools.  They’re also cross-platform, so getting on some of the great Microsoft programs like BizSpark and WebsiteSpark wasn’t really in the cards.

Anywho…  One of those tools was TeamCity.  It’s a great too for Continuous Integration and Deployment.  I’ve used it with a couple of clients now and have nothing but good things to say about it.  When I installed it, I probably read some yellow box talking about HSQLDB and External Database and evaluating the system and production; but, honestly, just follow the steps. :).  I created several configurations and had artifacts and build logs, so upgrading needed to include backup and restore…  I had a couple spare cycles to look into migrating to an external database (as well as run through a backup/restore cycle) so I thought I might as well bite the bullet.

Well, the title of the post explains it.  I’m not going to go too much into the existing documentation and how it jumps from page to page before you actually get a migration done; I’m just going to say there wasn’t just a single list of steps (and, I think a couple things were missing).  There’s support for multiple platforms and multiple databases, so I get it’s hard.  It’s seemed a bit too hard; so, I’m going to detail the process of migrating a TeamCity database to a SQL Server Express database here in case anyone else might find it useful.

[NOTE: I did this with TeamCity 6.5.4 build 18046, YMMV]

First off, backup.  I did this from the web interface.  I clicked on the Administration link near the top right, clicked the Backup Data link on the right-hand list, selected “Custom” scope and checked everything.  This created a zip file in the backup directory in the TeamCity data directory. For me, this was C:\TeamCity\.BuildServer\backup\TeamCity_Backup_20111130_210020.zip—but, yours will be different.

After doing a TeamCity backup, stop the TeamCity services.  (Start menu, Control Panel, … you know the drill … or net stop TCBuildAgent and net stop TeamCity from a command prompt).

Next, I did a file system backup.  I just copied the TeamCity directory to another folder.  I had installed TeamCity to C:\TeamCity.  I think the default was my user folder; but, I’m only at my client for a limited time and when I’m gone that’s going to cause a bit of grief (yeah, it happened :), so that meant copying C:\TeamCity to somewhere else on C:\.  I’ve digressed from the the Jetbrains instructions already…

Next, I set up the external SQL Server Express database.  I already had SQL Server 2005 Express installed, so, I won’t go through that installation process.  In SQL Server Management Studio I created a “TeamCity” database, and added a “TeamCity” login—you’ll need those later.

Next, I installed some JDBC drivers for SQL Server.  I just grabbed the Microsoft ones—the “Native” driver—I figured they might know a thing or two about writing a SQL Server driver.  I downloaded from here. I unzipped (or ran the EXE) into a directory then copied the sqljdbc4.jar file into the lib\jdbc TeamCity data sub-directory (which was C:\TeamCity\.BuildServer\lib\JDBC for me).

Next, I configured TeamCity to use SQL Server.  I created a database.properties file in the config data folder (C:\TeamCity\.BuildServer\config, for me).  It’s later: the file was similar to:

connectionUrl=jdbc:sqlserver://localhost\\sqlexpress:1433;databaseName=TeamCity
connectionProperties.user=TeamCity
connectionProperties.password=P@nda$

…names and passwords might have been changed to protect the pandas.  Since I’m using SQL Express I had to throw the \\sqlexpress named-instance stuff in there.  If you’re using a different named instance, the”sqlexpress” part will be different.  If you’re using a “real” install of SQL Server, you don’t need the “\\sqlexpress” part at all, e.g. “//localhost:1433”.  I also had to throw the :1433 stuff in there (long story), you may not need that.

At this point, be sure not to be running the TeamCity services and don’t even think about logging into the TeamCity web interface (well, you can’t if the server isn’t running).  As far as TeamCity is concerned at this point, you’ve got a fresh install.  If you log in now it will ask you to create an Administrator login and create all the database tables.  The next step requires an empty database and will fail if you do create that login…

Now, restore.  Okay, I lied, this is one of those parts where the documentation has misled you.  The restore requires that you have an empty config directory—i.e. you can’t have database.properties in the config directory.  You also need an empty system directory (C\TeamCity\.BuildServer\system\ for me)—a fresh install might mean those directories are empty, I don’t know.  I simply renamed config to config-old and system to system-old and created new config and system directories.  Then, I had to add the Java bin directory to the PATH.  TeamCity has all the Java binaries included with it—it doesn’t actually install Java (oh, BTW, TeamCity uses Java).  For me, I just ran path=%path%;c:\TeamCity\jre\bin to let the command prompt know where java.exe is.  If you have Java installed, you might not need to do this.  Then, I performed a restore; which is just a matter of running maintainDB with the restore command from the TeamCity bin directory (C:\TeamCity\bin directory—note no “.BuildServer”—for me):

…which tells TeamCity to restore from my backup (zip) file, where the data directory is, and what database configuration to use (note the config-old business).  Either TeamCity or Java doesn’t grok relative directories; so, note that I used “C:\TeamCity” instead of “..\”.

Once the restore is done (and it’s kind enough to remind you :), copy the database.properties file that you used to the config directory (for me, this was just copy ..\.BuildServer\config-old\database.properties ..\.BuildServer\config ).

Then, restart the TeamCity services:

And you’re done.  If you can now log into the TeamCity web interface with all your old logins and all your old configurations are still there…  I did a bit of housework by deleting config-old, system-old, and went ahead and deleted the C:\TeamCity copy that I made. (I don’t need that, now that I’ve migrated to SQL Server and verified that everything works).

If you ran into a problem along the way, you could simply copy the backed-up TeamCity directory over-top the old one (making sure the services were stopped first) and you should be back to where you were before.  I didn’t have a problem, so I can’t confirm that actually works; but the docs detail that it does.

Since you have to have an empty database and empty config/system directories, I gather the actual restore process would be identical (minus dropping all the tables in the TeamCity SQL Server database).

If You’re Using “#if DEBUG”, You’re Doing it Wrong

I was going through some legacy code the other day, refactoring it all over the place and I ran into many blocks of code wrapped in “#if DEBUG”.  Of course, after a bit of refactoring in a RELEASE configuration these blocks of code were quickly out of date (and by out of date, I mean no longer compiling).  A huge PITA.

For example, take the following code:

	public class MyCommand
	{
		public DateTime DateAndTimeOfTransaction;
	}
 
	public class Test
	{
		//...
		public void ProcessCommand(MyCommand myCommand)
		{
#if DEBUG
			if (myCommand.DateAndTimeOfTransaction > DateTime.Now)
				throw new InvalidOperationException("DateTime expected to be in the past");
#endif
			// do more stuff with myCommand...
		}
		//...
	}

If, while my active configuration is RELEASE, I rename-refactor DateAndTimeOfTransaction to TransactionDateTime then the block of code in the #if DEBUG is now invalid—DateAndTimeOfTransaction will not be renamed within this block.  I will now get compile errors if I switch to DEBUG configuration (or worse still, if I check in an my continuous integration environment does a debug build).

I would have run into the same problem had I been in a DEBUG configuration with all the “#if RELEASE” blocks.  Yes, I could create a configuration and define DEBUG and RELEASE and do my work in there.  Yeah, no, not going down that twisted path… i.e. try doing it without manually adding a conditional compilation symbol.

I got to thinking about a better way.  It dawned on me that #if blocks are effectively comments in other configurations (assuming DEBUG or RELEASE; but true with any symbol) and that comments are apologies, and it quickly became clear to me how to fix this.

Enter Extract Method refactoring and Conditional Methods.  If you’ve been living under a rock or have simply forgotten, there exists a type in the BCL: System.Diagnostics.ConditionalAttribute.  You put this attribute on methods that[whose calls] may or may not be included in the resulting binary (IL).  But, they are still compiled (and thus syntax checked).  So, if I followed the basic tenet for code comment smells and performed an Extract Method refactoring and applied the ConditionalAttribute to the resulting method, I’d end up with something like this:

		//...
		public void ProcessCommand(MyCommand myCommand)
		{
			CheckMyCommandPreconditions(myCommand);
			// do more stuff with myCommand...
		}
 
		[Conditional("DEBUG")]
		private static void CheckMyCommandPreconditions(MyCommand myCommand)
		{
			if (myCommand.DateAndTimeOfTransaction > DateTime.Now)
				throw new InvalidOperationException("DateTime expected to be in the past");
		}

Now, if I perform a rename refactoring on MyCommand.DateAndTimeOfTransaction, all usages of DateAndTimeOfTransaction get renamed and I no longer introduce compile errors.

Code Contracts

If you look closely at the name I chose for the extracted method you’ll notice that what this DEBUG code is actually doing is asserting method preconditions.  This type of thing is actually supported in Code Contracts.  Code Contracts implement a concept called Design by Contract (DbC).  One benefit to DbC is that these checks are effectively proved at compile time so there’s no reason to perform this check at runtime or even write unit tests to check them because code to violate them becomes a “compile error” with Code Contracts.  But, for my example I didn’t use DbC despite potentially being a better way of implementing this particular code.

Anyone know what we call a refactoring that introduces unwanted side-effects like compile errors?

[UPDATE: small correction to the description of conditional methods and what does/doesn't get generated to IL based on comment from Motti Shaked]

Working with Subversion Part 2 (or Subversion for the Less Subversive)

In my previous Subversion (SVN) post I detailed some basic commands for using SVN for source code control.  If you’re working alone you could get by with Part 1.  But, much of the time you’re working in a team of developers working on versioned software that will be deployed to multiple places.  This post will detail some more advanced commands for systematically using SVN to simultaneously work on multiple editions and versions of a software project. (hence the “less subversive”).

Practices

The basic concept of working on two editions or two versions of a software project at the same time is considered “branching”.  Simultaneous work is each done on it’s own “branch” (or “fork”) of the project.  Branch can be considered “copy”; but that’s generally only true at the very start of the branch.  With any project you can add and remove files from it; at which point a branch ceases to become a “copy”.  Semantics aside, SVN doesn’t really have a concept of a “branch”.  This is a practice used by SVN users.  SVN really only knows how to “copy” and “merge”.  The practice is the create a standard location to control branches, copy existing files and directories to a branch location, and merge changes from a branch to another location (like the trunk).  To support branching many repositories (repos) have a “trunk” and “branches” folder in the project root.  Trunk work is obviously done in the “trunk” folder and branches are recreated in the “branches” folder.

Branching Strategies

Before getting too much farther taking about things like “trunk”; it’s a good idea to briefly talk about branching strategies.  The overwhelming branching strategy for SVN users has been the “trunk” or “main line” branching strategy.  This strategy basically assumes there is really one “main” software product that evolves over time and that work may spawn off from this main line every so often to work independently and potentially merge back in later.  In this strategy the “current” project is in the “trunk”.

Another strategy is sometimes called the Branch per Release strategy which means there’s a branch that represents the current version of the project and when that version is “released” most work transitions to a different branch.  Work can always more from one branch to another; but there’s no consistent location that project files are located over time.  This a perfectly acceptable strategy and almost all Source Code Control (SCC) systems support it.  The lack of a consistent location of files makes discovery difficult and it really forces the concept of branching onto a team.  I’ve never really found this strategy to be very successful with the teams I’ve worked on, so I prefer the trunk strategy.

Branching

Branching is fairly easy in SVN.  The recommended practice is the perform a copy on the server then pull down a working copy of that branch to perform work on.  This can be done with the svn copy command, for example:

svn copy http://svnserver/repos/projectum/trunk http://svnserver/repos/projectum/branches/v1.0 -m "Created v1.0 branch"

…which makes a copy of the trunk into the branches/v1.0 folder.

Now, if you checkout the root of the project (e.g. svn checkout http://svnserver/repos/projectum) you’ll have all the files the trunk and all the branches.  You can now go edit a file in branches/v1.0 in your working copy and svn commit will commit that to the branch.  If you want to just work with a branch in your working copy you can checkout just the branch.  For example:

svn checkout http://svnserver/repos/projectum/branches/v1.0 .

..which makes a copy of all the files/directories in v.1.0 to the current local directory.  So, if you had …/branches/v1.0/readme.txt in the repo, you’d now have readme.txt in the current local directory.

Same holds true for the trunk, if you want to work on files in the trunk independently of any other branches checkout just the trunk, for example:

svn checkout http://svnserver/repos/projectum/trunk .

It’s useful to work just with the trunk or just with a branch because over time you may have many branches.  Pulling down the trunk and all branches will get time-consuming over time.

While SVN doesn’t really have the concept of a “branch”, it does know about copies of files and tracks changes to those copies.  So, if you shows a log of the changes to a file you’ll see the commit comments for all the branches too.  For example, edit readme.txt in the branch directory and commit the change (svn commit –m "changed readme.txt" for example) then go back to the trunk directory and show the log of readme.txt, for example:

svn log –v readme.txt

…you’ll see the commit comments for both trunk/readme.txt and branches/v1.0/readme.txt.  For example":

------------------------------------------------------------------------
r2 | PRitchie | 2011-11-17 10:23:36 -0500 (Thu, 17 Nov 2011) | 1 line
Changed paths:
   A /trunk/readme.txt
initial commit
------------------------------------------------------------------------

Merging

Okay, you’ve been working in v1.0 for a few days now, committing changes for v1.0.  One of those changes was a bug fix that someone reported in v1.0.  You know that bug is still in the trunk and it’s time to fix it in the trunk too.  Rather than go perform the same editing steps in the trunk, you can merge that change from the v1.0 branch into the trunk. For example, a typo fixed in the readme.txt needs to be merged into the trunk, from a clean trunk working copy (no local modifications):

svn merge http://svnserver/repos/projectum/branches/v1.0

This merges the changes from v1.0 into the working copy of trunk.  You can now review the merges to make sure they’re want you want, then commit them":

svn commit –m "merged changes from v1.0 into trunk"

Merge Conflicts

Of course, sometimes changes are made in the trunk and a branch that conflict with each other (same line was changed in both copies).  If that happens SVN will give you a message saying there’s a conflict:

Conflict discovered in 'C:/dev/projectum/trunk/readme.txt'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options:

You’re presented with several options.  One is to postpone, which means the conflict is recorded and you’ll resolve it later.  You can see the differences with diff-full.  Or you can accept the left or the right file as iss with mine-conflict or theirs-conflict—to accept the local file, use mine-conflict.  There’s also edit, which will allow you to edit the merged file showing the conflicts.  For example:

one
<<<<<<< .working
three=======
four >>>>>>> .merge-right.r9

This details that your working file has a changed second line “three” but the remote version has a changed second line “four “.  The lines wrapped  with <<<<<<< .working and ======= is the local change, which is followed by the remote change and >>>>>>> .merge-right.r9.  The “merge-right.r9” is diff syntax telling you which side the change came from and which revision (r9 in this case).  You can edit all that diff syntax out and get the file the way you want to merge it, then save it.  SVN will notice the change and present options again:

Select: (p) postpone, (df) diff-full, (e) edit, (r) resolved,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options:

Notice you now have a resolved option.  If your edits fixed the conflict you can choose resolved to tell SVN the conflict is gone.  You can then commit the changes and the merges will be committed into the repo.

Merging Without Branches

Of course branches aren’t the only source for merges.  You might be working on a file in the trunk that a team member is also working on.  If you want to merge any changes they’ve committed into your working copy, you can use the update command.  For example:

svn update

This will merge any changes files with your local files.  Any conflicts will appear the same way they did with svn merge.

It’s important to note that with SVN you can’t commit changes if your working copy was out of date with the repo.  (e.g. someone committed a change after you performed checkout).  If this happens you’ll be presented with a message similar to:

Transmitting file data .svn: E155011: Commit failed (details follow):
svn: E155011: File 'C:\dev\projectum\trunk\readme.txt' is out of date
svn: E170004: Item '/trunk/readme.txt' is out of date

This basically just means you need to run svn update before you commit to perform a merge and resolve any conflicts.

Mouse Junkies

If you don’t generally work with command-line applications and don’t care for the speed increase of not using the mouse, there’s some options you can use to work with SVN.

TortoiseSVN

TortoiseSVN is a Windows Explorer extension that shows directories/files controlled with SVN differently within Explorer.  It will show directories and files that have been modified or an untracked with different icons (well, icon overlays) within Explorer.  It will also let you perform almost all SVN commands from with explorer (via context menu).

VisualSVN

VisualSVN provides much of the same functionality as TortoiseSVN but does it within Visual Studio.  It’s not Source Code Control Visual Studio extension; which is interesting because you can use VisualSVN and another source code control extension (like TFS) at the same time.  VisualSVN requires TortoiseSVN to work correctly, so install that first.

Both TortoiseSVN and VisualSVN make dealing with merge conflicts easier.  I recommend using these for merging instead of the command-line.

Resources

http://svnbook.org/

“Explicit” Tests with Resharper

NUnit introduced a feature called Explicit Tests (a long time ago, I believe) that basically mean a test is considered tagged as Ignore unless the test name is explicitly given to the NUnit runner.

This is useful if you have tests that you don’t want run all the time.  Integration tests or tests highly coupled to infrastructure or circumstance come to mind…  But, it’s difficult to automate these types of tests because you always have to maintain a list of test names to give to the runner.

The ability of NUnit to run explicit tests aside, I don’t generally use the NUnit runner directly; I use other tools that run my tests.  I use tools like Resharper to run my tests within Visual Studio, ContinuousTests for continuous testing, and TeamCity to run my tests for continuous integration.

Continuous integration is one thing, I can configure that to run specific test assemblies and with specific configuration and get it to run whatever unit tests I need it to for whatever scenario.

Within Visual Studio is another story.  I sometimes want to run tests in a class or an assembly but not all the tests.  At the same time I want the ability of the runner to run tests it wouldn’t normally run without having to edit and re-compile code.

With Resharper there are several ways you can do this.  One way is to use the Ignore attribute on a test.  This is effectively the same as the NUnit Explicit attribute.  If I run the test specifically (like having the cursor within the test and pressing Ctrl+U+Ctrl+R) Resharper will still run the test.  If I run all tests in solution/project (Ctrl+U+Ctrl+L/right-click project, click Run Unit Tests) the test is ignored.  This is great; but now this test is ignored in all of my continuous integration environments.  Sad Panda.

If you’re using NUnit or MSTest (amongst others that I’m not as familiar with) as your testing framework you can tag tests with a category attribute (MS Test is “TestCategory”, NUnit is “Category”).  Once tests are categorized I can then go into Resharper and tell it what category of tests to “ignore” (Resharper/Options, under Tools, select Unit Testing and change the “Don’t run tests from categories…” section in Resharper 6.x).  Now, when I run tests, tests with that category are effectively ignored.  If I explicitly run a test (cursor somewhere in test and I press Ctrl+U+Ctrl+R) with an “ignored” category Resharper will still run it.  I now get the same ability as I did with the Ignore attribute but don’t impact my continuous integration environment.  I’ve effectively switched from an opt-in scenario to an opt-out scenario.

With the advent of ContinuousTests, you might be wondering why bother?  That’s a good question. With ContinousTests only the tests that are affected by the changes you’ve just saved—automatically, in the background.  In fact, having any of your tests run whenever you make a change that affects the test is one reason why I make some tests “explicit”.  I tend to use test runners as hosts to run experimental code, code that often will become unit tests.  But, while I’m fiddling with the code I need to make sure it’s only run when I explicitly run it—having it run in the background because something that affects the test isn’t always what I want to do.  So, I do the same thing with ContinousTests: have it ignore certain test categories (ContinuousTess/Configuration (Solution), Various tab, Tests categories to ignore).

Test Categorization Recommended Practices

Of course, there’s nothing out there that really conveys any recommendations about test categorization.  It’s more or less “here’s a gun, don’t shoot yourself”…  And for the most part, that’s fine.  But, here’s how I like to approach categorizing tests:

First principle: don’t go overboard.

Testing frameworks are typically about unit testing—that’s what people think of first with automated testing.  So,  I don’t categorize unit tests.  These are highly decoupled tests that are quick to run and I almost always want to run these tests.  If the tests can’t always run or I don’t want them run at any point in time, they’re probably not unit tests.

Next, I categorize non-unit tests by type.  There’s several other types of tests like Integration, Performance, System, UI, Infrastructure, etc.  Not all projects need all these types of tests; but these other tests have specific scenarios where you may or may not want them run.  The most common, that I’ve noticed, is Integration.  If the test has a large amount of setup, requires lots of mocks, is coupled to more than couple modules, or takes a long time to run, it’s likely not a unit test.

Do you categorize your tests?  If so, what’s your pattern?

Working with Subversion, Part 1

Working with multiple client projects and keeping abreast of the industry through browsing and committing to open source and other people's libraries means working with multiple source code control (SCC) systems.  One of the systems I use is Subversion (SVN).  It's no longer one of the SCCs I use most often so I tend to come back to it after long pauses and my SVN fu is no longer what it used to be.  I'm sure my brain is damaged from this form of "task switching", not to mention the time I spend trying to figure out the less common actions I need to perform on a repository (repo).  I usually spend more than few minutes digging up the commands I need for the once-in-a-decade actions I need to perform. 

I don't foresee getting away from SVN in the near future; so, I'd thought I'd aggregate some of these commands into one place.  My blog is the perfect place to do that (because it's just for me, right? :)

Backing Up

Outside of adding/committing, the most common action to be performed is backing up the repository.  Unfortunately for my brain this is automated and I don't see it for months at a time. To back up an SVN repository that you're not hosting or being hosted by third-party software (like VisualSVN Server) then I like dump/load:

svnadmin dump repo-local-path > repo-bkp-path

This let's you restore to the host that contains all the configuration data like permissions, users, and hooks.

If the repository is completely autonomous (i.e, just a directory on your hard drive and maybe an SVN daemon) then hotcopy is better:

svnadmin hotcopy local-path destination-path

Restoring

If you used the dump method of backing up, you need to use the load command to put the backup into an existing repository.  If you're not using a hosted repository, you'll first need to create the repository (svn create repo-local-path) to run load (in which case I'd recommend using hotcopy instead).  To load the dump into the existing repository:

svnadmin load repo-local-path < repo-bkp-path

If you’ve used hotcopy then the backup is the fully functional repository; just make it available to users (i.e. put it where you want it :).

Migrating

Migrating is basically just a backup and restore.  If you're backing up one repository and putting into an existing repository, use dump/load.  On System A

svnadmin dump repo-local-path > repo-bkp-path

On System B after copying repo-bkp-path from System A

svnadmin load repo-local-path < repo-bkp-path

Even if you weren't migrating to an existing repo, you could use this method; just add svnadmin create repo-local-path before svnadmin load.  The dump/load method has the added benefit of upgrading the data from one format to another if both systems don’t have the same version of SVN running.  The drawback of migrating with dump/load is that you’ll have to configure manually (or manually copy from the old repo) to get permissions, hooks, etc…

Now you've migrated your repo to another computer, existing working copies will be referencing the old URL.  To switch them to the new URL perform the following:

svn switch --relocate repo-remote-URL

Creating Working Copy

If you don’t already have a local copy of the repo to work with, the following command

svn checkout repo-remote-URL working-copy-path

Committing Changes

I added this section because I’ve become used to GIT.  GIT has a working directory and staging area model; so you tag files in the working directory for staging before committing.  This allows you to selectively commit modifications/additions.  SVN is different in that the working directory is the staging area so you effectively have to commit all modifications at once.  (you can stage adds because you manually tell SVN which files to start controlling).

svn status will tell you what’s modified (M) and what’s untracked (?).  To commit all modified

svn commit –m "description of changings included in the commit"

Undoing Add

Sometimes you a schedule something to add on commit by mistake or it’s easier to add by wildcard and remove the files that you don’t want to commit on your next commit.  To remove them from the next commit:

svn revert file-path

Be careful with revert because if the file is already controlled this will revert your local modifications.

Undoing Modifications

To revert the modifications you’ve made locally and restore a file to current revision in the repo:

svn revert file-path

Resources

    http://svnbook.org/

What’s your favourite SVN workflow?

Getting a Quick Feel for a New Software Team

I deal with many different teams and many different companies.  I see a lot of teams not get the benefit of known techniques and fail to evaluate and improve.  There's some check-lists out there that are used to evaluate a company/team before joining it; but I find the lists to be deeply rooted in the past.  They detail such fundamental things that knowing the criteria on the list really doesn't tell you much about how your time with this team will be or your role within it.

There's so many low-level features within software development team to aid in successfully delivering software.  But these low-level features are part of higher-level initiatives that if absent make the lower-level features almost moot.  "Do you use source code control?" for example.  Sounds like this is a fundamental part of being able to delivery quality software; and it is.  But, on its own it doesn't really do a whole lot to help the team deliver software.  Is the code put into source code control using accepted design practices?  Is the code put into source code control in a timely fashion.  Is the code put into source code control not impeding other people?  Etc. etc. etc.  "Yes" to "do you use source code control" without any of other initiatives that follow-on doesn't give me a warm and fuzzy feeling about getting on the team and focusing on software value and not spending a lot of time on, or stuck in, process.

Over the years I've been on many teams hired for a specific role that changed drastically almost as I began working with the team.  I've observed many things about many teams and have come up with some things I like to find out about a team before I start so I can better gauge the type of work that I'll be doing and how successful the team will be fulfilling their goals.

Does the team:

• have a known product owner/sponsor,
• have a cross-functional team 6-9 people in size,
• use appropriate tools,
• foster SOLID design principles,
• use continuous integration,
• use continuous deployment,
• foster communications with team members and stakeholders,
• have a known and managed process and visualize workflow,
• evaluate and improve process as a team,
• have new candidates write code in their interviews,
• have a plan that limits work in progress,
• have a plan that orders or prioritized tasks;

and, how easy is it to change to start doing any of the above items?

“Do you use source code control” is covered by “Does the team use continuous integration” as it’s not just about using source code control, it’s about a process that can’t function properly without source code control.

And, for what it's worth, this list doesn't tell you whether you should work on a team or not; it just tells you the type of work you will be doing.  It's up to you to dig deeper and decide whether you unrealistically want to limit your work to a specific or a small set of roles.  I would only use the last question as a acid test of whether I would join the team or not.  If they are not willing to improve, there's not much I'm going to be able to do for them.

What do you look for in a team/project?

DevTeach Comes to Ottawa!

DevTeach is coming to Ottawa November 2-4 2011.

DevTeach has had some of the best speakers in the industry and some of the best sessions/tracks that I’ve ever attended.

This year is no exception, the speaker line-up is full of experts and MVPs like David Starr, Mitch Garvis, James Kovacs, Laurent Duveau, Mario Cardinal, Guy Barrette, Paul Neilson, Rob Windsor, Benjamin Day, Erik Renaud, Joel Semeniuk, etc. and locals like Garth Jones, Wes MacDonald, Joel Hebert, Colin Smith, Colin Melia, Brad Bird, Rick Claus, etc.

Tracks include Agile, Web Dev, Rich Client, SQL Server Dev, SQL Server BI, and for the first time at DevTeach, an IT Pro track.

If you think you might have trouble convincing your boss you need to go, have a look at http://www.devteach.com/WhoShouldAttend.aspx.

To register: http://www.devteach.com/Register.aspx

Ottawa Speaker’s Idol Nov. 2nd at DevTeach Ottawa

Do you like to share your IT knowledge with others? Do you know a tip, trick or a feature that can save time for others and you would like to share it?

Well, Ottawa Speaker Idol is your solution and there is still time to submit your session proposal. The deadline is Wednesday Oct. 26^th.

The Ottawa IT Community is organizing a contest where you can win a free registration to DevTeach Nov. 3-4 (value of $649). On November 2nd, at the venue for DevTeach Ottawa (Ottawa Conference Center), you will be invited to present your tip, trick or a feature for 10 minutes. After all presentations have completed, the attendees will vote for the winners. Prizes will be distributed to the best presenters. This is a free event and everyone is welcome to attend.

Date: Wednesday November 2^nd , at 18:30
Location: Ottawa Conference Center (room 103)

55 Colonel By Drive

Ottawa , ON, K1N 9J2 Canada
Rules:

• You must not be a regular speaker in the IT Community. This contest is to encourage new speakers.
• It must be the first time you participate to the Ottawa IT Speaker Idol
• No need to be register to DevTeach to participate in the contest.
• You must submit the description of your presentation (no more than 200 words) by October 26^th to jrroy@DevTeach.com
  
• We will accept 10 presenters
• If more than 10 submissions, a committee will select 10 presentations
• The list of presentations will be announced on October 31st
• A laptop will be available for your presentation or you can use your own laptop
• You will need to bring your PowerPoint and code on a USB key. All presentations will be posted on our web site after the contest.

Prizes:

First place:    Registration to the main conference DevTeach Nov. 3-4 (value of $649) from DevTeach Inc.

Second place:    Win 7 Phone (Value of $549) from LIKE 10 Inc.

Third place:    TBA from Microsoft

Windows 8, What Does It Mean to Me?

Well, unless you’ve been hiding under a rock, you’ve probably heard about Microsoft Build conference that happened in Anaheim last week.  It was during this conference that Microsoft finally released details about the successor to Windows 7.  This event is solely developer-focused and, in my opinion, tablet-specific.  It went into a lot of detail about the added APIs and usability changes in Windows 8 to better support tablet and touch-based computers.

First off, the new touch-first usability changes are called “Metro” and applications written for touch are called “Metro-style apps”.

The term “touch-first” is used because it’s not touch-only.  Metro will work fine with a mouse and a keyboard; but the user experience is obviously designed for a multi-touch device and that you can “get by” adequately with a mouse and a keyboard.  This was done for various reasons, one being that touch isn’t good for entering reams of text and that you want to dock your tablet and use a keyboard and mouse when you want to do something that would be better served by a keyboard or a mouse.  e.g. tablet/touch-based UX is better for consuming information, whereas keyboard is better for producing information.

So, with all the focus was on tablet and touch, does that mean Windows 8 is all about Metro?  The focus of the conference was to show off (likely in answer to other vendors producing tablet-based hardware and systems) tablet- and touch-based user experiences.  In fact, paying attendees received a custom-made Samsung tablet computer with a version of Windows 8 (called Developer Preview) and the next version of Visual Studio (called Dev11, for now) pre-installed.  This puts the focus of Developer Preview almost solely on touch.  But, does this mean that mouse/keyboard was ignored or that we need a touch screen on our computers to be able to use Windows 8 when it is released?  No, of course not.  Discussing various things with numerous Softies, the message time and time again was that if something isn’t mentioned, or doesn’t have focus, that doesn’t mean it won’t exist in the release version or in the near future.  It was clear that this version of Windows 8 was produced strictly for this event, and since it was effectively deployed on tablet, it had a tablet edition of the OS.  I.e. if you by a desktop without a multi-touch monitor after Windows 8 comes out, the edition of Windows 8 that will be deployed will not likely have Metro on by default.  But, that’s my opinion.

We now have Metro apps, and they’re XAML-based and very similar to Silverlight.  You initialize the runtime slightly differently and do some other things slightly differently; but if you’re a Silverlight developer you’ll be right at home in Metro.  Does that mean all Windows 8 development is Metro?  Definitely not.  It’s not C++ based and it is still managed code backed by the CLR (they shows using VB and C#; but F# support should be there for RTM).  The CLR backing Metro apps is the same CLR that backs “Desktop” apps written as WinForms (yes, they’re still supporting that) traditional Silverlight, Console apps, Windows services apps, etc. etc. etc.

Does this mean Silverlight and .NET is dead?  Well, no; but you could interpret it that way.  Metro apps are driven by a new Windows API called WinRT—which is basically a object-oriented alternative for the Win32 API and removes blocking methods leaving, or replacing them with, asynchronous methods.  This and the CLR that backs it are inherent parts of Windows 8.  So, it’s technically not it’s own product and simply just another part of Windows.  So, you could say that .NET and Silverlight are just bits of Windows 8 and thus no longer exist as “.NET” and “Silverlight”.  Just like Hydra or Terminal Services really doesn’t exist any more because it’s baked right into Windows.  But, that doesn’t mean they don’t exist in this new form.

Some discussion has occurred as to whether WinRT is truly CLI-compliant because it doesn’t support all the types and methods that .NET does now.  This isn’t true.  We had these same discussions when Silverlight and Windows Phone came out.  WinRT is just another “profile”.  It has a subset .NET API.  Just like how Silverlight doesn’t include classes like System.Windows.Forms.Form and Windows Phone doesn’t include classes like System.Windows.Browser.HtmlDocument.  The depth of the .NET classes that are available to WinRT developers is restricted, just restricted differently than Silverlight and Windows Phone.

Just how restricted is the WinRT profile?  Well, that remains to be seen at this point; but, from the looks of it it is minimally CLI compliant.  This means it doesn’t include things like the reflection namespaces.  WinRT is, after all, for producing touch-based applications that may run on very minimal processors and hardware.

One thing of note, to finish, is that the theme of WinRT is asynchronous.  Anything that could take a noticeable amount of time (“blocking”) is not available in the WinRT profile; so, get used to asynchronous programming.

Enterprise Service Buses, Brokers, Message Queues, Oh My!

Continuing on my theme of message-oriented, I thought I’d get into a bit of detail about the various middleware options and some details about these options.

Regardless of what type of middleware is chosen, one thing is common to all types of message-oriented middleware—the fact that they broker messages from producers to consumers.

Impetus

Before I get too far into the options, let’s look at a couple of the common reasons behind moving to a message-oriented architecture.

One impetus towards MOM is to integrate various legacy or third-party systems into a new or exiting enterprise system.  This type of MOM has been know as Enterprise Application Integration (or EAI).  There’s all sorts of information about this architectural style.

The other major impetus towards MOM is to facilitate distribution.

Technically EAI is also distributed; but that’s more because the individual components are completely independent of one another and thus executing on different computers.  The purpose of EAI isn’t to get distribution but to integrate distributed components.

Enterprise Application Integration

Enterprise Service Bus

The technology-du-jour in the MOM world seems to be Enterprise Service Buses (ESB). Enterprise Service Buses, by definition are are horizontally scalable MOM that integrate disparate (including legacy) loosely coupled SOA components using web services. Not only do ESBs technically broker communication between producers and consumers, they also deal with routing, transformation and translation. It’s hard for ESBs to do what they’re good at without that specific loose coupling you get from web services, where interface definition is discoverable and service-oriented (web services, over http, etc. etc.).

From a .NET standpoint there’s a few options out there calling themselves ESBs like NServiceBus and MassTransit.

Brokers

One form of EAI is a Broker.  “Broker” is considered an anti-pattern by more than a few people in the industry given what we now know because of it’s inherent hub-and-spoke shape.  A broker is a single component (hub) that brokers messages from producers to consumers (spokes).  It’s frowned upon because of its inherent scalability problems.  A single broker is constricted to a single computer and to gain scalability it’s forced to scale vertically so it has finite and expensive scalability options.  It’s unfortunate that this architecture is frowned upon because it’s still more than adequate when used where appropriate and technically a subset of all the other MOM options.

There isn’t technically a whole lot of options in NET for hub-and-spoke integration.  .NET became popular just as the many of the problems with hub-and-spoke were coming to light.  But, the closest thing to hub-and-spoke seems to be BizTalk.  It’s message box would technically be the hub and components integrated into BizTalk be spokes.  But BizTalk is a bit of hybrid of sorts (it supports translation, SOA, etc) so it’s a bit of a matter of opinion.

Distribution

Message Queues

Almost all EAI middleware uses some form of queue BizTalk: MSMQ, MassTransit: MSMQ, RabbitMQ, ActiveMQ, etc, NServiceBus MSMQ, Azure Queue, etc.

Other

Technically the other options are considered message queues; but they’re not message queues in and of themselves; they just operate like a message queue.  Like databases, RPC, web services, etc.

When to use which option

So now that I’ve confused the horizon with a bunch more terminology and a fair share of technobabble, why would you choose one over another?

The reason I divided the options in to two is because the choice on which to use should really be based on the impetus behind the composition of the system.

If you need to integrate legacy or disparate 3rd party services together, enterprise service buses are definitely the way to go.  If you’re dealing with integrating many components together with much routing and translation requirements, and a high number of messages, ESBs will help you immensely.  If your scalability requirements are very low, going the broker route is probably more than adequate.  The impetus here is the integration; but you don’t want to make a system completely unusable because it can’t scale.

But, if you have control over all the components in your system and your overall impetus is the distribution of work and horizontal scalability, then your focus is on performance not integration.  You want your endpoints to communicate in the most efficient manner and not be bogged down by unnecessary translations and routing. In which case you most likely want to avoid ESBs.

ESBs are kind of a Golden Hammer at the moment. ESB implies a SOA, web services, certain discoverability, lack of control of some or all components, etc.  If you *do* have control over all the components and don’t specifically require SOA, Web Services, and discoverability; implementing specifically to work with an ESB seems overkill.  If your have control over all components in your system and every message coming from a producer can be directly accepted by the consumer, you don’t need any of the translation aspects of ESBs.  You end up really only gaining in the routing aspect of ESB—which if you control all the components in your system, should be fairly static.

Message-Oriented Architecture is a Misleading Term

I recently blogged about Message-Oriented Architecture (and Message-Oriented Middleware, MOM).  The technology leads to extremely flexible and scalable applications; but it really distracts from the system.  It’s like calling a standard single-process application as Control-Unit Oriented Architecture.  Sure, the application is divided into instructions that eventually make their way to the control unit of the CPU; but that’s not only a implementation detail of the application; it’s an essential implementation detail. It’s pretty hard to write an application that doesn’t eventually distil down to instructions given to a control unit of a CPU.

And that’s kind of the problem with Message-Oriented Architecture.  Messaging facilitates distributed systems; but you can’t implement a distributed system without some sort of messaging.  MOM comes into play when you don’t want to have to implement your own layer or code to broker communications.  MOM is middleware that you acquire the distribute messages from producers to consumers.  You don’t have to use MOM to do messaging; but, you can’t do distributed without messaging—at least in some form.

What, you say?  If that’s the case then why are things like MOM and message-oriented architectures just now coming into focus?  Well, it’s true, MOM is a fairly recent technology on the software scene.  Mostly because we’ve been simply accepting Moore’s law as a way of scaling our applications.  Now that Moore’s law is slowing down, we have to scale our applications much more differently.  Technologies like Map/Reduce, cloud, etc. are mechanisms for horizontally scaling our systems rather than upgrading a computer to make our software go faster (vertical scaling).  Technologies like cloud and map/reduce give us ways to break up the work and execute it elastically on multiple computers; but it doesn’t give us a way for our individual components of our system to communicate between each other.

I think message-orientation, although not really new, has come to the forefront simply because it’s so inherent to designing and implementing distributed systems.

A distributed system is composed of many processes—it’s a composite system.  Writing distributed systems is the art of creating composable systems.

I’m going to continue with my series of posts relating to messaging between processes in a distributed system with a focus on message-oriented middleware; but, keep implementation details in mind.

Message-oriented Architectures (Updated)

I’ve had a few people ask me about message-oriented architectures and my knowledge and experience of it has evolved over time; so, I figured it would be a good idea to collect and publish some information about it.  It’s not one of those topics on the Internet that has gotten a lot of content, so I’ll try and change that.

A Message-Oriented Architecture is sometimes called Message-Oriented Middleware or MOM because there’s a middle tier or middleware that acts as the broker to relay messages from producers to consumers.  At the most succinct definition MOM is simply an architecture that supports sending and receiving messages.  That’s an extreme simplification and when implementing MOM many more facets of it’s design come into play, so I’m going to expand out into some of the common ways of implementing a message-oriented architecture over time.  So, I may go outside of MOM from a purists point of view.

This will be the first post in a series of posts so it will attempt to be very introductory.  To have a stable base on which to discuss this topic, let me first outline some terms relating to messaging.

Terminology

Producer

Something that produces a message. A message could be a command sent to a single command handler or an event published to zero or more event handlers.

Consumer

Something that consumes a message

Handler

Synonym for consumer

Message Broker

A component, often in an ESB, that transforms/translates and/or routes messages from one subsystem to another.

Queue

A storage mechanism for messages.  Depending in the implementation, a queue may be where messages are received from. Generally a separate process that receives messages through a known protocol like HTTP or TCP.  Generally used to implement load balancing semantics.

Aggregator

Something that consumes messages and contains them in some way and directs the message to another consumers.  Used, partially, to decouple producers from consumers.  A Queue is a form of aggregator.  Often lives within the a process.  Sometimes used to to merge pipelines.

Topic

Metadata associated with one or more messages that is generally used for publish and subscribe semantics.  Something publishes to or subscribes to topics rather than having to process all messages.

Exchange

In messaging systems that implement exchanges, exchanges are where messages are sent (compared to a queue, which is technically where messages are received from).

Service Bus

A integration platform that generally combines web services, messaging, transformations, and routing, to transactionally integrate many disparagedisparate applications or systems.  Often referred to an Enterprise Service Bus or ESB.

Pipeline

Often used in the context of messaging to describe a particular stream of messages or the fact that messages arrive or are processed one-by-one in a particular order.

Event

A message that informs subscribers of something that occurred in the past (albeit possibly not in the very distant past).

Command

A message that requests a change in state or data.

Saga

A series of related long-lived transactions that are executed as a non-atomic unit.

Idempotent

A operation that when repeated gives the same result.  This concept is used in mathematics as well; but becomes an important concept in messaging when we deal with things like guaranteed delivery.

Guaranteed delivery

Reliable messaging that guarantees a message is delivered to its intended recipients.  This often involves store-and-forward techniques to ensure message are not lost before delivery.

A least once delivery

A form of delivery reliability that involves concessions which may mean a message is delivered more than once.  See Idempotent.

Pub/Sub

AKA Publish/Subscribe.  A form of messaging where a publish publishes an event that is subscribed to by zero or more subscribers.  Publishers don’t can who, if anything, subscribes to events because an event is a representation of something that occurred in the past and requires no response.

Publisher

A producer that produces messages that are considered events.  A producer could produce events and commands.

Subscriber

A consumer that consumes message that are considered events.  There nothing stopping someone from implementing a consumer that subscribes to events and handlers commands; but they are generally kept separate, ala CQRS.

There’s all sorts of examples of messaging that have been around quite a while.  Sockets, HTTP, Web Services, WCF, etc are all examples of messaging techniques or technologies.  I’m going to discuss things at a slightly higher level where we deal with some of the application-level details of a message-oriented system.  i.e. things like Sockets, HTTP, Web Services, WCF, etc are implementation details—they may facilitate our messaging, but I’m not really going to get into that level of detail.

Messages are sent by producers asynchronously.  This means the producer is decoupled from consumers in every way except the fact that they share knowledge of one or more messages.  This also means that producers don’t know whether or not consumers have or will consume a message that it sends.  This sounds a bit scary at first; but it opens up a world of possibilities through this decoupling.  You have to account for the fact that producers don’t know about what consumers are up to.

Features

Very loosely coupled

Consumers and produces are separated by messages.  These messages are simply data.  They could be a binary serialized blob of data that a producer needs to be able to understand how to deserialize (often limiting producer/consumer to the same platform), or the message could be in the form of XML; where the consumer need only know how to process XML (not limiting consumer/producer to the same platform).

Platform agnostic

Messages can be sent and received on any platform.  Much in the same way an HTML page can be downloaded and viewed on any platform.  There are degrees to which a message-oriented system can support multiplatform (see above w.r.t. binary serialization); but the architecture itself imposes no real platform specifics.

Asynchronous

Consumers give messages to a broker, aggregator, or queue of some sort and that message is processed and delivered asynchronous.  The consumer is free to go about its business before effectively before the message has even left the broker.  Of course when consumer and producer and living in the same memory space using certain types of aggregators, this may not be the case; but typical message-oriented architectures use middleware that processes messages asynchronously.

Benefits

Flexibility is one of the important benefits of message-oriented architectures.  A message-oriented architecture is limited only by the coupling you impose.  If you some sort of middleware and don’t impose any requirements on how your message needs to be processes, you messages can be processed on any platform and at any point in time.  Need to publish a message from Windows and have it consumed in a Windows 7 Phone?  No problem.  Need that processing to occur almost instantly?  It’s entirely up to the consumer how quickly they need to consume a message.  Message doesn’t need to be processed right away?  Again, no problem the consumer can take as long as it needs to process the message.

The flexibility of the system results in many indirect benefits.  The producer doesn’t know anything about the consumer of a message (or how many consumers there are, if any) which means the message is free to be routed, aggregated and delivered completely independently of the producer.

One of the most important aspect, in my opinion, of message-oriented architectures is scalability.  By separating producers and consumers at the process level from one another by way of a message you introduce the ability to horizontally scale almost infinitely.  Because a producer asynchronously produces a message and one or more consumers may process the messages of a consumer, the broker is free to load balance or scale out to multiple consumers.  This is not without purposeful, and correct, design; but offers a system-level scalability the supports vertical and horizontal scaling.  For example, multiple consumers can be running on a computer to take advantage of each processor on that computer (vertical) or multiple consumers can be running on multiple computers to take advantage of any number or type of computers (horizontal) resulting in an architecture that supports near infinite scalability.

Drawbacks

One of the biggest drawbacks of a message-oriented architecture is the topology.  You’re generally looking at least at a publishing computer, a computer to house the messaging middleware, and another computer handle the consuming.  Each of these computers needs to be configured and publishers/consumers/middleware needs to be deployed and working correctly.  This can be cost prohibitive on smaller systems and implies much management.

If a message-oriented architecture is used in a situation it is not intended, there can be drawbacks to using it.  If a producer needs to synchronously produce a message (i.e, now when the consumer handles the message before it continues on with anything else), this can be a difficult scenario to implement.  If the producer needs to know about the consumer or consumers, this can also be a difficult scenario to implement.  Both of the scenarios can cause various headaches in a particular deployment.  I don’t really consider these drawbacks because MOM is not truly being used correctly; but the technology is not widely understood and the concepts it requires are often misunderstood so I mention it here because an implementation of MOM may encounter them early in the implementation.

Standards have come a long way recently, but—depending on your choice of implementation—there may be no real messaging standard.  This could mean you end up being tied to a particular vendor or even a particular platform with no means of extending to other platforms or other components using another messaging mechanism.

References

Wikipedia’s Message-oriented Middleware

CQRS

Taking Customer Support Into Social Era

We’ve entered the Social Era with the advent and popularity of technologies like Twitter, Facebook, FriendFeed, FourSquare etc.  Supporting a product or service can take customer satisfaction to the next level by making use of these technologies.

To be clear, I’m of the opinion that customer support has the ability and responsibility for improving customer experience with a product or service.  My opinion is that customer support should be proactive with regard to supporting customers; customer support teams that don’t believe this need not read the rest :)  The best product or service can be meaningless if the customer support experience is subpar.

Most interaction with customer support teams means hunting down resources like web sites or phone numbers and either figuring out or deciphering the process involved in interacting with that particular customer support team.  For some people, this just isn’t worth their time.  They’re either too busy, or they’ve been through the process before and know how tedious it is and just don’t feel like subjecting themselves to it again, or doesn’t have enough return on investment (i.e. benefit to friction ratio).

Technologies like Twitter allow people to post their feelings, experiences, opinions, etc. on the Internet for the world to see.  Some of these postings could be viewed simply as complaining or badmouthing a product or service.  That’s one way of view it; but I think viewing these postings as an opportunity—and opportunity to reach out to the product’s community in another way to improve their experience or continue an existing great experience—can elevate a company in the minds of the community.

So, how can that be done?  Well, very easily as a matter of fact.  Technologies like Twitter make it easy to search for information.  You can add searches like “technology A” or “company B” to see what customers are saying and pick out comments that seem like customers having problems.  If they’re problems that can easily be fixed through some interaction with the customer and providing clarified or extra information then the team an simply open a conversation with the customer online and guide them through the problem.  If the issue is really a bug, then the team and open a conversation online to get details about the bug, pass information about patches to the customer, and evaluation on fixes that can quickly as possible make there way into the product.  Of course, simply “re-tweeting” allows the organization (and other users) to broadcast good PR about the company and the product.

Why is this good? This is good on multiple levels.  One of course is the good PR.  The company is viewed as doing good for customers and turning a potential negative into a positive.  The next is proactively finding and fixing problems.  If the comment is the result of a bug in the product or service then using the comments as a way of finding and fixing the bug means other customers may never experience the bug.  Finally, of course, is making customers happy.  There’s many stories of companies responding to people on Twitter to help with problems and these people blogging or commenting on the wonderful “over and above” service they’ve received.

Robustness with RabbitMQ in .NET

Recently I’ve been doing a bit of work with queues, in part with RabbitMQ.

Unfortunately, (or fortunately, depending on your point of view) network connections where I am have had a tendency to be unreliable.  That’s a story for the pub; but, needless to say we needed our clients of RabbitMQ to be robust in light of disconnections.  Fortunately, RabbitMQ has ways of dealing with this.

A bit about queues

Queues are a two way communication mechanism.  You can enqueue messages and dequeue messages.  In a distributed system you often have a local queue that you asynchronously deal with when you enqueue and dequeue messages.  MSMQ works in this way.  RabbitMQ can be used in this way too.  Eventually, though, a connection to a remote computer needs to be made for the message to be delivered or received from the queue.  RabbitMQ can support asynchronous communications; which means failure can occur outside of when you give your message to the local queue (or “client”, in RabbitMQ parlance) or receive a message from the local queue.  RabbitMQ doesn’t have the luxury of a the OS having a built-in queue client like MSMQ to deal with connection issues out-of-process—the RabbitMQ client is in-process to your application, so it can be a bit tricky dealing with disconnections.

RabbitMQ supports many “patterns” of messaging like pub/sub, push/pull, etc.  I’ve been working in a more pub/sub model; so I’m focusing on publication and subscription aspects of RabbitMQ

Robustness can be a complex beast; and I’m not going to discuss many of the aspects certain systems need to deal with.  I’m going to take the stance that the queue and it’s client API deals with correct delivery, receipt, acknowledgement, and resending of unacknowledged messages.

With that said, I’ve alluded to two points of failure.  During the enqueue or during the dequeue.  This is where I’ll detail how we can make our system to be more robust with regard to RabbitMQ and disconnections.  With a subscription model in RabbitMQ, the local queue client basically proactively gathers messages for you and places them in a local cache (itself a queue) and you “enumerate” through the messages in the queue.  One way of doing this is with the Subscrption.Next() method.  One overload of Next simply blocks and returns the next message in the queue.  In my circumstance we simply couldn’t block the thread indeterminately like that, so we used a different overload Subscription.Next(int timeout, out BasicDeliveryEventArgs result).  This is effectively the same as the first Next method but will timeout if there are no messages in the queue.  The calling code is then free to do anything else it needs to do, like abort and/or try again.  During the call to Next the connection to the queue server could have been severed.  If it has been severed one of two things will happen. 

In the case of Next(int, BasicDeliveryEventArgs), the BasicDeliveryEventArgs instance may be null but Next will return true.  (i.e. “succeeded” but there’s no message).  This may happen because the connection was lost prior to the call to Next.  This may also happen if the connection is severed while data is being streamed from the server—resulting internally with an end of stream exception.

Another possibility is OperationInterruptedException will be thrown from Subscription.Next.  I’m not sure why things like end of stream exceptions aren’t translated into an OperationInterruptedException instead of returning true and setting BaiscDeliveryEventArgs to null.

In either case, this tells you that the connection is no longer good.  With RabbitMQ you have to try to recreate your connection and try your dequeue again.

Of course, you’re not sure what the problem may be and thus  don’t know when the server may be available again; so, to simply immediately try to reconnect and retry will most likely mean several failures until it succeeds.  e.g. if the server was rebooted due to an update; we might be able to try thousands of times before we’re successful.  To avoid these needless retries, adding a delay in there limits the number of times we retry and gives other threads a chance at CPU time.  I’ve found that when we don’t delay at all, it’s more common that we never seem to be able to reconnect.

Following is some code that deals with disconnections in a robust way.

while (!abort)
{

	if (subscription == null)
	{
		try
		{
			subscription = CreateSubscription(connectionFactory, "mainqueue", out connection, out model);
		}
		catch (BrokerUnreachableException)
		{
			Thread.Sleep(1000);
			continue;
		}
	}
	try
	{
		BasicDeliverEventArgs basicDeliveryEventArgs;
		if (subscription.Next(500, out basicDeliveryEventArgs))
		{
			if (basicDeliveryEventArgs == null)
			{
				throw new OperationInterruptedException(
					new ShutdownEventArgs(ShutdownInitiator.Application, 0, "null BasicDeliveryEventArgs"));
			}
			// TODO: something with basicDeliveryEventArgs.Body
			subscription.Ack(basicDeliveryEventArgs);
		}
	}
	catch (OperationInterruptedException)
	{
		// don't bother with connection, it will throw IOException due to disconnection
		using (model) using (subscription)
		{
			subscription = null;
			model = null;
			connection = null;
		}
		Thread.Sleep(1000);
	}

}

It, of course, doesn’t go into detail about how to create a subscription instance—I’ve left that up to you to create a subscription how your application needs it, and the disposal of the model and the connection isn’t detailed as your application will implement this code in different ways influencing when and how these two instances need to be disposed.

Certification Caveats

There's lots of organizations and company's that offer "certification"--either on their own products/processes or someone else's.  They'll all try to tell you the virtues of their certification.  But, despite their claims, certification isn't a panacea.  Let's look at some caveats of certification.

The biggest area I've seen is what "certification" means.  "Certified" as what, or able to do what?  Really, certification only certifies that the person can and has taken an exam.  Let's look at software development for a moment.  I'm never going to hire someone whose sole purpose will be to take that exam.  There's the possibility that they used their knowledge and experience to pass the exam; but the exam itself cannot gauge that.  There's a certain amount of faith that needs to be put into that certification.

This leads to the next biggest problem of "certifications": the people who either memorized the answers or only studied exam questions.  These people, at worst, posses only enough knowledge on the technology under certification to pass the test.  It's highly unlikely that anyone with just that knowledge is going to be any use to me.  All joking aside, I have seen people pass certification exams doing only this.  I've worked with "certified" peers, "engineers", "PhD's", etc. that were effectively dysfunctional and ended up being more a burden than a benefit to the team and the project.

This raises the question, what should I be expecting from certifications? Certification alone tells you nothing.  In conjunction with experience and empirical evidence that you yourself have verified about a candidate, certifications can be information that can tell you something more about a candidate; but, rarely should be a mandatory requirement for hire.

Okay, so if certifications are not a panacea, then how do i make them useful?  Well, for one, make them a "nice to have".  At face value they alone can't tell you a candidate meets your criteria.  This assumes, of course, you have criteria and it's more than just "passed certification exam".  So, to get value from whether or not someone is certified, you need to know what this candidate will be doing for you and how to judge her work.  If you cannot answer either of those two questions, whether or not someone is certified does you no good.  Okay, so you do know what you need this candidate to do and come up with some valid business-driven criteria to judge their work once hired...  Is there anything about the certification that would help you verify the candidates work once hired?  e.g. I need to hire someone to write a Windows Phone 7 application.  There isn't a huge pool of resources with loads of experience developing for WP7.  Certification in Windows Phone 7 and experience developing Silverlight or WPF applications would give me the qualitative criteria to make a judgement call over another candidate.

There are circumstances where the certification itself can be the only criteria.  I would argue that it's fairly rare; but, when new technologies come out and there are simply no one with any quantifiable experience with a technology, certifications can be useful.  It's rare in the software industry you want someone simply for their knowledge in one particular technology.  Most of the time you're looking more for someone who has other more general attributes like problem solving, ability to learn, works well under pressure, team player, bleah, bleah, bleah.  My point is, that if they posses none of those other qualities, whether they passed a certification exam is meaningless to you--the candidate is simply unacceptable before even getting to the fact they're certified.  I would also argue that you need to perform some sort of due diligence and either learn about the technology you hope to hire someone for or get cheat sheet—expecting a certification to be a replacement for that is only going to cause you disappointment.

Criteria for Success

I was having a  conversation with a client recently and the topic of “11th hour” came up.  He seemed to think it wasn’t possible to deliver a software project without some sort of “11th hour” panic.  I disagreed

I disagree because I’ve done it on small and large projects.  Don’t get me wrong; it’s very easy to lose sight of things and easily get a project into a state where “11th hour” panic is inevitable.  But, it’s not something that can’t be avoided.

One of the problems with software projects, it seems, (and with other projects, I suppose) is losing sight of things and not having a “destination”.  This is what I refer to by “criteria for success”.

Criteria for success is basically the criteria by which what is expected to be delivered will be evaluated.

This criteria no only gives you a focused destination, it also provides a focus for the team.  Without some sort of criteria it’s hard to gauge whether the team is side-tracked on something that doesn’t add value to the project.  Criteria for success not only gives you a destination to strive for; but also means your efforts can be focused on that destination and be able to get to that destination quicker.  This means the project can be completed quicker and progress can be judged more accurately.

Without defining criteria for success, software projects that do succeed succeed by accident.  I don’t know about the rest of the industry; but I’m not really comfortable with completing my work “by accident”.

Defining “Done”

Defining “Done” is a very similar.  Defining done is another popular mantra; but, I think “criteria for success” is more clear in its goal in that it’s almost explicit that acceptance criteria is what we’re looking for.  “Done” can be a little wishy-washy and open for interpretation.

Defining Criteria

I don’t think it’s enough to complain that people have a problem defining success or “done”; so, I think it’s also important to help people get their own criteria for success.  i.e. people have a problem coming up with criteria for success because no one has defined criteria for success for defining criteria for success—so to speak.

There’s various tools to help with this sort of thing; the most obvious is Agile.  Defining user stories and getting acceptance from stakeholders on these stories is an excellent first step.  Many people simply view user stories as a check-list of items.  This can be extremely useful; but explicitly defining criteria (acceptance criteria is the more general term) isn’t an explicit task for most people.

Often we can’t expect out stakeholders to know how to define the acceptance criteria.  “Just do it” or “make it work” are often the definition of done—which doesn’t leave us much in the way of unambiguous.

Agile kind of get’s away from “requirements” because requirements are often associated with waterfall methodologies and people expect that “requirements” mean a full set of requirements for the project before the project commences and doesn’t change for the life of the project.  This really isn’t what requirements are; but, we’ll continue using terms like “criteria for success” and “defining ‘done’” to avoid these associations.  But, theories and practices for effectively defining requirements can give us some valuable tools for defining good criteria.  Karl Wiegers has a couple of excellent books that has some great criteria that we can reuse for our criteria:

Correct

Is the criteria correct (easy to say)?  Is how we’re judging the deliverable right?  If we’re judging x against y, is y what will actually be used as the criteria?  If it’s really z, we can’t deliver something that satisfies y on purpose.

Feasible

Is the criteria even attainable?  If we simply can’t reach that criteria, we’re going to fail…  This can be a bit tricky because we don’t always know whether what we’re actually working is actually possible.  With Agile, we like to separate what is known to be possible from what is not known to be possible with “spikes”.  We purposely research things we’re not sure are possible (or not entirely sure how long things may take) into spikes so we can gather the information we need to define our criteria for success correctly.  This mitigates the risk and gives is the ability to separate and time-box these aspects of the project because if we’re not really sure if it’s feasible or how long it’s going to take we need to prioritize it differently and explicitly.

Necessary

This is kind of obvious; but without this explicit criteria our criteria may lack focus and our work not be prioritized correctly and we end up working on tasks that are not important (i.e. not necessary).  It’s critical that the criteria actually define necessary qualities.

Prioritized

I’ve already alluded to this; but prioritized criteria is essential for getting a project done properly.  “Prioritized” is a bit ambiguous; so, it’s important to define what we mean by prioritization.  The most essential part is that criteria is prioritized in relation to other criteria.  There’s a tendency for everything to be prioritized as “important” and this has the opposite effect because it also means everything is defined as “unimportant” and the order that tasks get performed is open to interpretation (ease of implementation, interest in the task, coolness of the task, etc. none of which are stakeholder criteria).  If it’s not important enough to prioritize a task correctly, it’s not important enough to actually perform the task.

Unambiguous

It may seem obvious; be we need to make sure our criteria is not ambiguous.  It’s easy for criteria to be open for interpretation and end up being ambiguous.  See the next point…

Verifiable

The criteria that Weigers’ details is that the criteria needs to be verifiable.  It’s not enough to say “done” or “works”, we need to be able to unambiguously and empirically verify our work against specific criteria.  Criteria like “works” just doesn’t cut it. This is stakeholder-driven.  Often we’ll need to help our stakeholders think through what they’re ultimately going to use for verification—we want them to think about this before we begin the work, not after (which is very common).

The Era of Usability

For most of the personal computer generation, success of computers and their operating systems have been ruled by people’s ability to do what they needed to do on or with the computer.  We’re starting to enter an era where the success of a computer and/or it’s operating system is going to hinge on it’s ability to reduce usage friction compared to another.

We’re entering an era where software has effectively become a commodity—where the majority of people don’t differentiate between the major operating systems or the most frequently used software.

To a certain degree computers and computer technology is becoming a bit of a fashion; but that’s due in large part to the fact that their software has become a commodity.  Someone can surf the web, write a document, analyse data in a spreadsheet on any computer.  Most major OSes and other software are largely the same.  They have all the same features, roughly the same non-functional features, etc.  Anyone can go into a computer and pick one of several OSes and begin doing, effectively, anything they need.  The success of an OS is no longer driven by the fact that the software you need/want to use is only available on one platform

For the most part, most of the computer generation has been largely ruled by people willing to put in the time to “build” computers in order to use software.  They started out physically building computers; getting the hardware parts and putting them together to form a workable computer.  That evolved to off-the-shelf computers that people had to get the OS and software they wanted up and running to do what they want.  That evolved to turn-key systems that you could take home, turn on, and be using software in no time—every-day software was already installed and ready to use.

Now we enter a new era.  Anyone and their sister seems to be supplying hardware and software combinations allowing people to do almost 99% of all software-based tasks.  We’ve got Windows-based computers, various Linux-based computers (PCs, phones, tablets, etc.), Mac-based computers, iOS-based computers and phones, etc.  This era is truly about usability.  Whatever OS or software does what you need it to do with the least amount of friction is king.  We’re even seeing hybrid systems.  I can run a Mac running virtual machines to get Mac software and Windows software on the same computer.  I’m not limited to using software for any one particular platform…

So, what does this mean?  This means software developers, manufacturers, and OS designers need to pay a renewed attention to usability.  We had a quantum leap in usability back when Xerox basically created the GUI.  But, that’s largely stagnated in the past 20-30 years.  That’s not to say there needs to be much improvement in the basic GUI; but there needs to be a *huge* quantum leap in basic software’s usability.  We take for granted that our OS provides all the usability we need, piggy-backing, for the most part, on what it has to offer.  But, that time is well on it’s way to moving to the past.

We have put with software that is unresponsive, uses huge amount of power, is flakey, requires weird unintuitive actions, has poor performance, or poor reliability and robustness.

No longer is it acceptable for things to eventually work through some weird series of steps and incantations.  Software needs to work and it needs to work well and in a way we expect it to.  No longer can software just stop responding to our requests, not longer can software require a complex series of actions to do what we want it to do, no longer can one suite of software drive the success of an entire operating system.  No longer is it acceptable to expect people to reinstall, reboot, or reformat.  People are able to do what they want to do with a multitude of options.  The differentiating factor is starting to become usability.  Software, computers, and technology that make it easier for people to do what they want to do (from running their operating system to surfing the web, to using a “word processor”) is quickly rising to the top of the market.

Case in point has been the Mac.  It doesn’t really offer anything new or anything innovative.  It’s just another computer with the same typical software selection.  The differentiating feature of the Mac has been usability.  The Mac makes thing easier to use.  Multi-touch on the Mac, for example, works.  You don’t get a jerky half-assed implementation of scrolling with multi-touch on the Mac, you get something smooth and it’s intuitive.  I can surf the web, write documents, send email, watch videos, listen to music, instant message, etc. on the Mac; and I don’t have to deal with drivers from x number of companies that don’t really give two licks about usability.

In the next little while either software and hardware companies are going to realize that usability is the feature that people are spending their money on and continue to be, or become, differentiating players in the market, or they’re simply going to fall to the wayside.  It’s going to matter if they’ve been in the industry for almost 40 years.  There is no loyalty in business, if there’s something better, easier to use, faster, cheaper, whatever; people are going to use it.  It’s time for some of these companies to wake up and see the writing on the wall.

It’s also time for us, the users of software, to take a stand and be vocal about this stuff.  We don’t have to sit in silence and accept the whims of some software designer who hasn’t seen the light of day in 6 months.  If I purchase a piece of hardware that advertises “multi-touch” I expect to be able to use it, or software that implies common metaphors by how they appear should function like those common metaphors, etc.

Got any gripes about software or computers?  Post them here, let’s start and continue a dialogue on what usability should be…

Mapping to Your Database is a Private Affair

Your mapping to your database is generally coupled to the design of the data and the mapping provider's ability to implement a relational or non-relational model.  This means your ORM influences the design of your mapped classes and/or the design of system that uses these mapped classes.  i.e. your design can be limited in certain aspects by how the mapper is implemented.

Because of these idiosyncrasies, your choice of ORM becomes an implementation detail; so, you want to keep any of its details and generated classes out of your interfaces.  You want to keep your ORM’s classes and the classes it generates private.

This may sound inconsequential, but there are some very real examples of how the limits of your choice of mapping framework can adversely affect your external interfaces.  Let's look at some specific cases of this. 

WCF supports data contracts and collection types.  But, because of the nature of  WCF and SOAP, it deals with serializing collections in a specific way.  If you have an entity class with a collection property on it, it may be perfectly acceptable to be able to add objects to that collection and expect them to make their way back into the database.  But, what does that mean for some entity that’s been serialized over the wire to another remote computer that doesn’t have access to your database?  It means the entity you publish over WCF has a different behaviour that those local to (presumably) the server.  But, you don’t want to modify your entity to work in a different way just for your WCF interface. You’ve published an interface to the outside world, clients are using it and you realise you need to make a change to your database.  Opps, can’t it’s tightly coupled to your WCF interface; if you change your database and regenerate your entities you interface will change; breaking all your clients.

It works the other way around too.  If you want to publish types over WCF, you have to annotate them, potentially, with all sorts of things to get them to serialize the way you want them to.  The simplest is DataContractAttribute and DataMemberAttribute; but, could get more complex with things like IExtensibleObject, things like DataContractSerializer versus XmlSerializer, etc.  Then there’s things like versioning…  Do you really want to add all these annotations to code generated by your mapper?  In some cases you can’t annotate them.

So, classes that map to/from the data effectively should be implementation details of your system; they should be private to your data layer; but, then how do we detail transferring data from our domain to the outside world?  The Facade pattern and the Data Transfer Object pattern are a perfect solution to this.  Your WCF interface is effectively a facade—an interface (usually simplified) to a different body of code—or a type of view to your model. When you have to provide complex data structures in your interface (i.e., other than base types) use Data Transfer Objects (DTOs) to wrap them in.  Should the body of code that the facade wraps needs to change, simply modify the code that generates the DTOs.  (I almost always get this question next: “What if the change means I can no longer produce the same DTO?”.  Well, then you’ll have to decide if you should violate your contract and break all your clients, or find a way to make it work.)

I find it amusing to see all the examples of things like using your LINQ to SQL or LINQ to Entities classes in WCF interfaces.  The examples are simple and often work; but once you try to do that in a complex system quickly realize you’re down a rat hole with really only one way out.  Do yourself a favour and completely abstract your implementation details from your internally accessible interfaces; you’ll thank yourself.