Dispose Pattern and “Set large fields to null”

I was involved in a short side discussion about “should” fields be set to null in the Dispose method(s).  I’m not sure what the impetus of the question was; but, if you read through the dispose pattern MSDN documentation (in most versions I believe) there’s a comment // Set large fields to null. in the implementation of the virtual Dispose method within the if(!disposed) block and after the if(disposing) block.  But, that’s the only reference to setting fields to null during dispose.  There’s nothing else that I’ve been able to find in MSDN with regard to setting fields to null.

At face value, setting a field to null means that the referenced object is now unrooted from the class that owns the field and, if that was the last root of that reference, the Garbage Collector (GC) is now free to release the memory used by the object that was referenced by that field.  Although advanced, this seems all very academic because the amount of time between unrooting the reference and the return from Dispose (and thus the unrooting of the parent object) would seem like a very short amount of time.  Even if the amount of time between these two actions is small, setting a single field to null (i.e. a single assignment) seems like such a minor bit of code to provide no adverse affects.  The prevalent opinion seems to be that the GC “handles” this case and does what is best for you without setting the field to null.

The GC is pretty smart.  There’s a lot of bright people who have worked on the GC over the years; and it improves every release of .NET.  But, that doesn’t answer the question; is there benefit to setting a field to null in the Dispose method?  Considering there isn’t much guidance on the topic; I’d though I’d set aside any faith I have in the GC and throw some science at the problem: take my theory, create some experiments, make observations, and collect some evidence.

What I did was to create two classes, identical except that the Dispose method doesn’t set a reference field to null.  The classes contain an field that could reference a “large” or “small” object: I would experiment with large objects and small objects and observe the differences.  The following are the classes:

	public class First : IDisposable {
		int[] arr = new int[Constants.ArraySize];
		public int[] GetArray() {
			return arr;
		}
		public void Dispose() {
			arr = null;
		}
	}
 
	public class Second : IDisposable {
		int[] arr = new int[Constants.ArraySize];
		public int[] GetArray() {
			return arr;
		}
 
		public void Dispose() {
		}
	}

I would vary Constants.ArraySize constant to make the arr reference a “large” object or a “small” object.  I then created a loop that created several thousand instances of one of these classes then forced a garbage collection at the end; keeping track of the start time and the end time via Stopwatch:

	public class Program {
		private const int Iterations = 10000;
 
		static void Main(string[] args)
		{
			var stopwatch = Stopwatch.StartNew();
			for (int i = 0; i < Iterations; ++i)
			{
				using (var f = new First())
				{
					ConsumeValue(f.GetArray().Length);
				}
			}
			GC.Collect();
			stopwatch.Stop();
			Trace.WriteLine(String.Format("{0} {1}", stopwatch.Elapsed, stopwatch.ElapsedTicks));
			stopwatch = Stopwatch.StartNew();
			for (int i = 0; i < Iterations; ++i)
			{
				using (var s = new Second())
				{
					ConsumeValue(s.GetArray().Length);
				}
			}
			GC.Collect();
			stopwatch.Stop();
			Trace.WriteLine(String.Format("{0} {1}", stopwatch.Elapsed, stopwatch.ElapsedTicks));
		}
 
		static void ConsumeValue(int x) {
		}
	}

I wanted to make sure instanced got optimized away so the GetArray method makes sure the arr field sticks around and the ConsumeValue makes sure the First/Second instances stick around (more a knit-picker circumvention measure :).  Results are the 2nd result from running the application 2 times.

As it turns out, the results were very interesting (at least to me :).  The results are as follows:

Iterations: 10000 ArraySize: 85000 Debug: yes Elapsed time: 00:00:00.0759408 Ticks: 170186.
Iterations: 10000 ArraySize: 85000 Debug: yes Elapsed time: 00:00:00.7449450 Ticks: 1669448.

Iterations: 10000 ArraySize: 85000 Debug: no Elapsed time: 00:00:00.0714526 Ticks: 160128.
Iterations: 10000 ArraySize: 85000 Debug: no Elapsed time: 00:00:00.0753187 Ticks: 168792.

Iterations: 10000 ArraySize: 1 Debug: yes Elapsed time: 00:00:00.0009410 Ticks: 2109.
Iterations: 10000 ArraySize: 1 Debug: yes Elapsed time: 00:00:00.0007179 Ticks: 1609.

Iterations: 10000 ArraySize: 1 Debug: no Elapsed time: 00:00:00.0005225 Ticks: 1171.
Iterations: 10000 ArraySize: 1 Debug: no Elapsed time: 00:00:00.0003908 Ticks: 876.

Iterations: 10000 ArraySize: 1000 Debug: yes Elapsed time: 00:00:00.0088454 Ticks: 19823.
Iterations: 10000 ArraySize: 1000 Debug: yes Elapsed time: 00:00:00.0062082 Ticks: 13913.

Iterations: 10000 ArraySize: 1000 Debug: no Elapsed time: 00:00:00.0096442 Ticks: 21613.
Iterations: 10000 ArraySize: 1000 Debug: no Elapsed time: 00:00:00.0058977 Ticks: 13217.

Iterations: 10000 ArraySize: 10000 Debug: yes Elapsed time: 00:00:00.0527439 Ticks: 118201.
Iterations: 10000 ArraySize: 10000 Debug: yes Elapsed time: 00:00:00.0528719 Ticks: 118488.

Iterations: 10000 ArraySize: 10000 Debug: no Elapsed time: 00:00:00.0478136 Ticks: 107152.
Iterations: 10000 ArraySize: 10000 Debug: no Elapsed time: 00:00:00.0524012 Ticks: 117433.

Iterations: 10000 ArraySize: 40000 Debug: yes Elapsed time: 00:00:00.0491652 Ticks: 110181.
Iterations: 10000 ArraySize: 40000 Debug: yes Elapsed time: 00:00:00.3580011 Ticks: 802293.

Iterations: 10000 ArraySize: 40000 Debug: no Elapsed time: 00:00:00.0467649 Ticks: 104802.
Iterations: 10000 ArraySize: 40000 Debug: no Elapsed time: 00:00:00.0487685 Ticks: 109292.

Iterations: 10000 ArraySize: 30000 Debug: yes Elapsed time: 00:00:00.0446106 Ticks: 99974.
Iterations: 10000 ArraySize: 30000 Debug: yes Elapsed time: 00:00:00.2748007 Ticks: 615838.

Iterations: 10000 ArraySize: 30000 Debug: no Elapsed time: 00:00:00.0411109 Ticks: 92131.
Iterations: 10000 ArraySize: 30000 Debug: no Elapsed time: 00:00:00.0381225 Ticks: 85434.

For the most part, results in debug mode are meaningless.  There’s no point in making design/coding decisions based on perceived benefits in debug mode; so, I don’t the results other than to document them above.

The numbers could go either way, if we look at percentages; release mode, setting a field to null seems to be slower 50% of the time and faster 50% of the time.  When setting a field to null is faster it’s insignificantly faster (5.41%, 9.59%, and 4.28% faster) when it’s slower it’s insignificantly slower but more slow than it is fast (133.68%, 163.52%, and 107.84% slower).  Neither seems to make a whole lot of difference (like 10281 ticks over 10000 iterations in the biggest difference for about 1 tick per iteration—1000 byte array at 10000 iterations).  If we look at just the time values, we start to see that setting a field starts to approach being faster (when it’s slower it’s slower by 295, 8396, and 6697 ticks; when it’s faster it’s faster by 8664, 10281, 4490).  Oddly though, setting “large” fields to null isn’t the biggest of faster setting field to null values.  But, of course, I don’t know what the documentation means by “large”; it could be large-heap objects or some other arbitrary size.

Of course there’s other variables that could affect things here that I haven’t accounted for (server GC, client GC, GC not occurring at specific time, better sample size, better sample range, etc.); so, take the results with a grain of salt.

What should you do with this evidence?  It’s up to you.  I suggest not taking it as gospel and making a decision that is best for you own code based on experimentation and gathered metrics in the circumstances unique to your application and its usage..  i.e. setting a field to null in Dispose is neither bad nor good in the general case.

“Virtual method call from constructor” What Could Go Wrong?

If you’ve used any sort of static analysis on source code you may have seen a message like “Virtual method call from constructor”.  In FxCop/Visual-Studio-Code-Analysis it’s CA2214 “Do not call overridable methods in constructors”.  It’s “syntactically correct”; some devs have said “what could go wrong with that”.  I’ve seen this problem in so many places, I’m compelled to write this post.

I won’t get into one of my many pet peeves about ignoring messages like that and not educating yourself about ticking time bombs and continuing in ignorant bliss; but, I will try to make it more clear and hopefully shine a light on this particular class of warnings that arguably should never have made it into object-oriented languages.

Let’s have a look at a simple, but safe, example of virtual overrides:

public class BaseClass {
	public BaseClass() {
	}
 
	protected virtual void ChangeState() {
		// do nothing in base TODO: consider abstract
	}
 
	public void DoSomething() {
		ChangeState();
	}
}
 
public class DerivedClass : BaseClass {
	private int value = 42;
	private readonly int seed = 13;
 
	public DerivedClass() {
	}
 
	public int Value { get { return value; } }
 
	protected override void ChangeState() {
		value = new Random(seed).Next();
	}
}

With a unit test like this:

[TestMethod]
public void ChangeStateTest() {
	DerivedClass target = new DerivedClass(13);
 
	target.DoSomething();
	Assert.AreEqual(1111907664, target.Value);
}

A silly example that has a virtual method that is used within a public method of the base class.  Let’s look at how we might evolve this code into something that causes a problem.

Let’s say that given what we have now, we wanted our derived class to be “initialized” with what ChangeState does (naïvely: it’s there, it does what we want, and we want to “reuse” it in the constructor); so, we modify BaseClass to do this:

public class BaseClass {
	public BaseClass() {
		DoSomething();
	}
 
	protected virtual void ChangeState() {
		// do nothing in base TODO: consider abstract
	}
 
	private void DoSomething() {
		ChangeState();
	}
}
 
public class DerivedClass : BaseClass {
	private int value = 42;
	private readonly int seed = 13;
 
	public DerivedClass() {
	}
 
	public int Value { get { return value; } }
 
	protected override void ChangeState() {
		value = new Random(seed).Next();
	}
}

and we modify the tests to remove the call to DoSomething, as follows:

[TestMethod]
public void ConstructionTest() {
	DerivedClass target = new DerivedClass();
 
	Assert.AreEqual(1111907664, target.Value);
}

…tests still pass, all is good.

But, now we want to refactor our derived class.  We realize that seed is really a constant and we can get rid of the value field if we use an auto property; so, we go ahead and modify BaseClass as follows:

public class DerivedClass : BaseClass {
	private const int seed = 13;
 
	public DerivedClass() {
		Value = 42;
	}
 
	public int Value { get; private set; }
 
	protected override void ChangeState() {
		Value = new Random(seed).Next();
	}
}

Looks good; but now we having a failing test: Assert.AreEqual failed. Expected:<1111907664>. Actual:<42>.

“Wait, what?” you might be thinking…

What’s happening here is that field initializers are executed before the base class constructor is called which, in turn, is called before the derived class constructor body is executed.  Since we’ve effectively changed the initialization of the “field” (now a hidden backing field for the auto-prop) we’ve switched it from a field initializer to a line in the derived constructor body: trampling all over what the based class constructor did when calling the virtual method.  Similar things happen in other OO languages; but, this particular order might be different.

Now, imagine if we didn’t have a unit test to catch this; you’d have to run the application through some set of specific scenarios to find this error.  Not so much fun.

Unfortunately, the only real solution to this is to not make virtual method calls from your base constructor.  One solution to this is to separate the invocation of ChangeState from the invocation of the constructor.  One way is basically reverting back to what we started with and adding a call to ChangeState in the same code that invokes the constructor.  Without reverting our refactoring, we can change BaseClass to what it was before and invoke the DoSomething method in the test, resulting in the following code:

public class BaseClass {
	public BaseClass() {
	}
 
	protected virtual void ChangeState() {
		// do nothing in base TODO: consider abstract
	}
 
	public void DoSomething() {
		ChangeState();
	}
}
 
public class DerivedClass : BaseClass {
	private const int seed = 13;
 
	public DerivedClass() {
		Value = 42;
	}
 
	public int Value { get; private set; }
 
	protected override void ChangeState() {
		Value = new Random(seed).Next();
	}
}

[TestMethod]
public void ChangeStateTest() {
	DerivedClass target = new DerivedClass();
 
	target.DoSomething();
	Assert.AreEqual(1111907664, target.Value);
}

Issues with virtual member invocations from a constructor are very subtle; if you’re using Code Analysis, I recommend not disabling CA2214 and promoting it to and error.  Oh, and write unit tests so you can catch these things as quickly as possible.

Software Design and First Principles–Part 0: Concepts of Object Orientation

I often compare software development with building houses or woodworking.  I sometimes even compare software development with the vocation of electrician.  In each of these other vocations, craftspeople need to go through a period of apprenticeship and mentoring before being “allowed” to practice their craft.  In each of these vocations there are a series of rules that apply to a lot of the basics of what what they do.  With building houses there are techniques and principles that are regulated by building codes; with electricians there’s techniques and fundamentals that are effectively regulated by electrical codes and standards.  It’s one thing to learn the techniques, principles, and fundamental laws of physics; but, it’s another thing to be able to call yourself an electrician or a carpenter.

Now, don’t get me wrong; I’m not advocating that software development be a licensed trade—that’s an entirely different conversation.  But, I do believe that many of the techniques and principles around software development take a lot of mentorship in order to get right.  Just like electricity, they’re not the most intuitive of techniques and principles.  But, just like electricity, it’s really good to know why you’re doing something so you can know its limits an better judge “correctness” in different scenarios.

To that effect, in order to understand many of the software development design techniques and patterns, I think the principles behind them are being ignored somewhat in a rush to get hands-on experience with certain techniques.  I think it’s important that we remember and understand what—I’m deeming—“first principles”.

A First Principle is a foundational principle about what it applies to.  Some of the principles I’m going to talk about may not all be foundational; but, I view then as almost as important as foundational, so I’m including them in First Principles.

From an object-oriented standpoint, there’s lots of principles that we can apply.  Before I get too deeply into these principles, I think it’s useful to remind ourselves what object-orientation is.  I’m not going to get too deep into OO here; I’ll assume you’ve got some experience writing and designing object-oriented programs.  But, I want to associate the principles to the OO concepts that guide them; so, It’s important you as the reader are on the same page as me.

OO really involves various concepts.  These concepts are typically outlined by: Encapsulation, abstraction, inheritance, Polymorphism (at least subtype, but usually parametric and ad-hoc as well), and “message passing”.  I’m going to ignore message passing in this part; other than to say this is typically implemented as method calls…

You don’t have to use all the OO concepts when you’re using an OO language; but, you could argue that encapsulation is one concept that is fundamental.  Encapsulation is sometimes referred to information hiding; but, I don’t think that term does it justice.  Sure, an object with private fields and methods “hides” information; but, the fact that it hides the privates of the type through a public interface of methods isn’t even alluded to in “information hiding”.  Encapsulation is, thus, a means to keep privates private and to provide a consistent public interface to act upon or access those privates.  The interface is an abstraction of the implementation details (the private data) of the class.

The next biggest part of OO is abstraction.  As we’ve seen, encapsulation is a form of abstraction (data abstraction); but the abstraction we’re focusing on now is one that decouples other implementation details.  Abstraction can be implemented with inheritance in many languages (e.g. code can know now to deal with a Shape, and not care that it’s given a Rectangle) and that inheritance can use abstract types. Some OO languages expand abstraction abilities to include things like interfaces—although you could technically do the same thing with an abstract type that had no implementation.

Inheritance is key to many of other concepts in OO—abstraction, subtype polymorphism, interfaces, etc.  (if we view an interface as an abstract type with no code, then something that “implements” an interface is really just inheriting from an abstract type; but, my focus isn’t these semantics).  We often let our zeal to model artefacts in our design and run into problems with the degree and the depth of our inheritance; a point I hope to revisit in a future post in this series.

Although you could technically use an OO language and not use polymorphism in any way, I think OO languages’ greatest features is polymorphism.  Subtype polymorphism, as I’ve noted, is a form of abstraction (Shape, Rectangle…).  But all other types of polymorphism are also abstractions—they’re replacing something concrete (implementation details) with something less concrete (abstract).  With subtype polymorphism that abstraction is an abstract type or a base type; with parametric polymorphism we generally create an algorithm abstraction that is decoupled from the data involved (Generics in .NET); and ad-hoc polymorphism is overloading—a decoupling of one particular method to one of many.

I quickly realized the scope of this topic is fairly large and that one post on the topic would be too much like drinking from a firehose as well as potentially to be protracted (and risking never getting done at all :).  So, I’ve split up what I wanted to talk about into chunks.  I’m not entirely sure what the scope actually is yet; I’ll kind of figure that out as a I go or let feedback guide me.  Now that we’ve got most of the OO concepts in our head, the next post will begin detailing the principles I wanted to talk about.

The Flawed Eventually-upgrade Software Model

I think Windows XP was the first real release of Windows--it had finally gotten to a usability and stability point that people could accept.  The Microsoft support model changed shortly after Windows XP was released to basically support any piece of software for as long as ten years (if you paid extra for support roughly 2 years after a successive version was released). To paraphrase a famous law: software becomes obsolete every 18 months.  That was true for a long time; but hardware and software isn't improving at that rate any more.   Software has basically caught up with existing hardware design and now has the capability of sustaining itself, without upgrade, for much longer than it did 10 years ago.

To paraphrase once again: you can make some of the people happier all of the time, but you can't make all of the people happier all of the time.  Releasing new versions of software now-a-days is more about attempting to make more people happier than were happier before.  To approach your solution or your technology from a 100% buy-in point of view is unrealistic.  I think we've seen the fallout of that model for at least the last 10 years.  People have said that successors to software like Windows XP, on their own, aren't enough to make people happier than they already are.  To try to force a change is only coming back with push-back.  The friction that once kept people on a particular brand of OS or even particular architecture is gone--people are exercising their options if they’re unable to use what they’re happy with.

I think it’s time for software companies to change their model so customers can buy into an indefinite support model for software.  I think businesses are more than willing to spend more money to get support for some software packages longer than to buy the latest version every x number of years.  If you look at the TCO of upgrading away from XP compared to what a business pays Microsoft for the OS, it's very much more. Companies are willing to offset that cost and buy support for XP rather than upgrade away from XP.  It just so happens that Microsoft extended support for XP rather than change their core model.

I think a the current model effectively giving customers the choice between abandoning XP and going to the latest version of an operating system (because you're effectively forcing them to make that evaluation) the more likely that you end up forcing people away from Windows entirely.  People and businesses are re-evaluating whey they need their computers and thus the operating system installed on it.  There’s much more a need to consume data over the Internet than there was 10 years ago.  People and companies are recognizing that and they’re also recognizing there are many more options for doing just that.

With this model, moving forward, innovation will drive software sales more than they do now.  People will upgrade not because it’s the latest version and not because they have to upgrade their hardware; but because the innovation of the software is pervasive enough to justify upgrading.  Different wouldn’t be enough to sell upgrades.

What do you think?  Do you think the eventually-upgrade software model is out of date?

The Rat-hole of Object-oriented Mapping

Mark Seemann recently had a great post that, as most of his posts seem to do, challenges the average reader to re-think their reality a bit.  The post is titled “Is Layering Worth the Mapping”.  In the post Mark essentially details some of the grief and friction that developers face when they need to decouple two or more concepts by way of an abstraction.

Mark takes the example of layering.  Layering is a one way coupling between two “layers” where the “higher-level” layer takes a dependency on abstractions in a “lower-level” layer.  Part of his example is a UI layer communicates with a domain layer about musical track information.  That track information that is communicated lives in a hand-crafted Track abstraction.  Typically this abstraction would live with the lower-level layer to maintain the unidirectional coupling.  Of course the UI layer needs a Track concretion for it to do its job and must map between the higher-level layer and the lower-level layer.  To further complicate things other decoupling may occur within each layer to manage dependencies.  The UI may implement an MVx pattern in which case there may be a specific “view-model” track abstraction, the data layer may employ object-relational mapping, etc. etc.

Mark goes on to describe some “solutions” that often fall out of scenarios like this in a need to help manage the sheer magnitude of the classes involved: shared DTOs as cross-cutting entities, POCO classes, classes with only automatic properties, etc.

It's not just layering.  Layering lives in this grey area between in-memory modules and out-of-process "tiers".  Layering, I think, is an attempt to get the benefits of out-of-process decoupling without the infrastructure concerns of connecting and communicating between out-of-process processes. Of course, over and above the module/assembly separation, the only thing enforcing this decoupling in layers is skill and being methodical.

I'm convinced layering is often, or often becomes, a "speculative generality" to give some "future proofing" to the application since layering so closely resembles "tiering" (not to be confused with the eventual response of "tearing") as to make it easy to make it tiered should there ever be a need for it.  To be clear, this is the wrong impetus to design an software.  You’re effectively setting yourself up to fail by essentially “making up” requirements that are more than likely going to be wrong.  If the requirements for the design are based on fallacies, they too are designed wrong.  But, you have to continue to maintain this design until you re-write it (ever noticed that anti-pattern?).

But, implementing tiers or any sort of communication between processes often ends up in the same state.  You have internal "domain" entities within the processes (and even within logical boundaries within those processes) that end up spawning the need for "DTO" objects that live at the seams on one or either side of the communication.  Further that, many times that communication is facilitated by frameworks like WCF that create their own DTOs (SOAP envelopes for example).  Except you're mandated by the physical boundaries of processes and you're forced to do things like shared-type assemblies to model the cross-cutting "entities" (if you choose that cross-cutting "optimization”) introducing a whole new level of effort and a massive surface area for attracting human error (you've technically introduced the need for versioning, potentially serialization, deployment issues, etc. etc.).

Creating an object-oriented type to simply act as a one-way container to something that lives on the other size of some logical of physical boundary has appeared to me to be a smell for quite a while.  e.g. the UI layer in your original example has this concept of a "Track" DTO-like type that when used is only used in one direction at a time.  When moving from the UI to the domain layer it's only written to.  If it gets a "track" back from the domain layer the UI layer only reads from it.  Abstracting this into an OO class seems pointless and, as Mark says, not particularly beneficial.

Let’s look specifically at the in-memory representation of something like a “Track”.  We’ll limit our self and say that we need four Track abstractions: one for the view model, one for the domain layer abstraction, one for the data layer abstraction, and one for the object-relational mapping.  (I’ve assumed that the data layer may not have a track “entity” and is only responsible for pushing data around).  So, in effect we have four Track DTO classes in our system (and two or three Track “entities”).  But, if we look at the in-memory representation of instances of these objects they’re effectively identical—each one can’t really have more data than another otherwise there’s something wrong.  If we look at what’s actually happing with this data, we’re really writing a lot of code to copy memory around in a really inefficient way.  The DTO classes in part become the way to copy memory.  To be fair, this is a side-effect of the fact we’re manually mapping from one abstraction to another or from one abstraction to an entity (or vice-versa).

This type of thing isn’t entirely unknown; it sometimes goes by the name of ceremony.

For the most part, I think computer languages are hindering us in our ability to address this.  Languages in general tend to maintain this specific way of messaging called method-calling that limits our ability to communicate only information that can be encapsulated by the language's or platform's type-system.  But, to a certain extent we're also hindered by our myopia of "everything must be a type in language X".  Maybe this is another manifestation of Maslow's Hammer.

Imagine if you removed all the mapping code in a complex system—especially a distributed system—and were left with just the “domain” code.  I’ve done this with one system and I was astounded that over 75% of the code in the system had nothing to do with the systems “domain” (the “value-add”) and was “ceremony” to facilitate data mapping.

I sometimes hear this isn’t so much of a problem with specific frameworks.  I’m often told that these frameworks do all the heavy lifting like this for us.  But, they really don’t.  The frameworks really just introduce another seam.  The issue of Impedance Mismatch isn’t just related to object-relational mapping.  I has to do with any mapping where both sides aren’t constrained by the same rules.  I’ve blogged about this before. but I can use some “data framework” to generate “entities” based on a data model or even based on POCO’s.  Some view this as solving the problem; but it doesn’t.  Each side operates under different rules.  The generated classes can only have as much impedance as what it has to communicate with, and you have to plan that that’s different than the impedance you’ll end up mapping from/to.  The only real solution to this is to introduce another DTO to map between your domain and the classes generated by the framework so you are decoupled from the eventual “gotchas” where your domain has different expectations or rules than the framework you’re communicating with.  When people don’t do this, you see all sorts of complains like “date/time in X isn’t the same as what I need”, etc.

Don’t fall into this rut.  Think about what you’re doing; if you’re got 4 DTOs to hold the same data; maybe there’s a better way of doing it.  Try to come up with something better and blog about it or at least talk about the problem out in the open like Mark.

What Code Comments are Not For.

I deal a lot with other people's and legacy code.  One of the things I see very often is what I call "misuse of code comments".  Let me provide an exhaustive list of what code comments are for:

• Describing why code isn't doing something obvious

There, done.

What code comments are not for (not complete):

The Obvious

// set the value of i
i = value;

It's obvious that the code is doing that; the comment adds no value here.  The compiler provides no validation that the "i" in the comment really applies to any variable in the line the comment applies to.  Nor does it even know what line the comment applies to!  Comments like this actually introduce some technical debt because I can refactor this code to move it around independently of the comment and thus the comment would appear to annotate some other line of code. Refactoring tools help somewhat with this; but they only really do name matching in comments when you rename a variable.  Do you think renaming "i" to "count" really means replacing all "i"'s in all comments with "count"?  Probably not; don’t use refactoring tools as a crutch.

Helping the Reader Learn the Language

You can't possibly know what the reader of your code does and does not know.  This is especially true of what they do and don't know of language syntax.  The language syntax is your common language; you can only assume they know it. You can’t possibly know if a comment about language syntax is going to help the reader or not.  If they don't know it, they should look it up.  Comments that describe syntax are a no-win proposition, you can’t possibly have a comment that helps every reader of the code.

An example:
        /// <summary>
        /// Constructor for class.
        /// </summary>
        public MyClass()

If your reader doesn't know this is a constructor, they probably don’t even know what a c'tor is—this comment isn't going to help them much. 

Slightly different example:

/// <summary>
/// Initializes a new instance of the MyClass class
/// </summary>
public MyClass()

If the reader doesn’t know what a c’tor does, does all your code include comments that will help this reader?  These comments are a waste of time and add no value. Same technical debt as Obvious, it’s not a syntax error to separate the comment from the declaration; there is a risk they will become disconnected or out of synch.  If the comment has no value having to manage it also has no value and therefore adds work to the project.

Another example verging on Obvious:
public MyClass()
{
  // Empty
}

As this stands, it seems benign. But one, it should be Obvious.  Two, if it's not, the reader should be brushing up on language syntax.  Three, it's not verified.  I can edit this c'tor to make it do something else this is perfectly syntactically correct:

public MyClass()
{
  x = 42;
  // Empty
}

Now, the comment is meaningless and potentially confusing.  Reading this for the first time makes you wonder did the class just have // Empty in it in the past and x = 42 was added, or does "empty" mean something different to the author, or did the author suffer from a stroke and possibly need medical attention?

You can assume the reader of your code doesn’t know anything about the code.  If the language can’t express the concepts in the code properly (if it can, you should be writing it that way; if you choose not to, comment why.) then comment why the language isn’t describing the concepts.

WHY not HOW

Writing comments to aid the reader in the understanding of the language is sometimes describing HOW the language is working.  That's not describing the code but describing the language.  Comments should describe WHY the code was written that way if it’s not obvious.  Again, the language is the common denominator between the reader and the author.  There's many references the reader can refer to to learn the language--let them do that; you may not be the best person to help someone learn the language; at the very least you don’t know the degree to which they don’t know the language.  Let the code clearly describe HOW.

Use of comments is often a form of religion; people are very opinionated about them in one way or another.   Robert Martin pulls no punches in Clean Code by saying:

“The proper use of comments is to compensate for our failure to express yourself in code. Note that I used the word failure. I meant it. Comments are always failures.”

Martin has previous described comments as “apologies” for “making the code unmaintainable”…

If you want to use concepts like XMLDOC or tools like JavaDoc to document members and classes, that's fine; just make sure your comments are meaningful and can stand alone.

For what it’s worth, these are comments that I have encountered; more than once and on more than one project.

Do you have some code comment pet peeves?  We’d love to hear them!

Avoid the volatile Modifier

[Update: 25-Jan-12 5:45 pm; fixed typo]

I was reminded recently of the misconceptions of the volatile modifier in C#, and I’d thought I’d pass along the recommendations of other’s that is tantamount to “avoid the volatile modifier”.  The volatile modifier in C# “indicates that a field might be modified by multiple threads that are executing at the same time” [1].

The first problem is that documentation.  What does that really mean to someone developing code that uses multiple threads?  Does that make the code “thread-safe”?  The answer is “maybe”; the real answer is “rarely”. Most people just stick volatile on a field because they think that’s what they need to do.

What volatile modifier does do to a field in C# is make all reads to the field use “acquire semantics” and all readswrites use “release semantics”.  Much clearer, right?  Acquire semantics means access is “guaranteed to occur prior to any references to memory that occur after it in the instruction sequence” and release semantics means access is “guaranteed to happen after any memory references prior to the write instruction in the instruction sequence” [6,2]

One of the problems with modern compilers [3] and processers and multithreaded code is optimization.  Within a block of code with no externally visible side effects the compiler is free to re-order instructions and remove instructions to result in the same visible side-effect.  “x+=1;x+=1;” can be freely optimized to “x+=2;” and “x+=1;y+=2;” can be reordered so that order of execution is effectively “y+=2;x+=1;”, for example.  The processor doesn’t optimize multiple instructions into one; but can re-order instructions and make the results of executed instructions visible to other cores/processors visible well after the instructions were executed (processor caching).  With fields, the compiler[3] has less freedom to optimize because side-effects to fields have more visibility—but the processor doesn’t discern between C# fields and any other bit of memory.

So, what does volatile really do for a field?  Realistically it prevents processor re-ordering and caching.  It tells the processor that all accesses to the value of that variable should come from or be made directly to memory, not the cache.

Not having a value cached by the processor so all other processors (and thus all other threads) can see what happens to the value of a variable seems like a really good thing though, doesn’t it?  In that respect, yes, it is a good thing.  But, let’s look at some of the drawbacks of volatile.

The syntax of volatile is that it just annotates a memory location.  In reality it really modifies the operations that occur on that memory location.  Code that operates on that memory location looks the same, regardless of whether it’s modified by volatile or not.  i.e. it’s unclear the code has different side-effects for multithreaded scenarios.  Another problem is that “volatile” is a very overridden word.  It’s used in C++ and Java; but it means something different (sometimes slightly) in each.  Moving back and forth from C++ to C# to Java could lead to using volatile incorrectly.  Volatile also assumes that all accesses to the field need to be protected from re-ordering.  Most of the time, there’s very specific times you want to force side-effects to a field to be made “visible” and, effectively, flush the processors cache to/from memory.  Flushing to the cache on every access is not very performant if you don’t need side-effects truly visible until specific times.  Plus, the volatile modifier means nothing if you pass the field by reference somewhere else. [4,5] There’s also limitations to what you can apply volatile to (e.g. you can’t have a volatile long).

What does volatile really mean from a code perspective?  Well, when you read from a volatile field effectively the following code is executed:

Thread.VolatileRead(ref field);

And when you write to a volatile field effectively the following code is executed:

Thread.VolatileWrite(ref field, newValue);

Both of these methods really just translate to a memory access and a call to Thread.MemoryBarrier (before the memory access for read, and after for write).  What MemoryBarrier does for is is ensure that all cached writes are flushed to memory at the call to MemoryBarrier (i.e after MemoryBarrier, any cached writes that occurred prior to MemoryBarrier are flushed and any cached reads are abandoned).  So, if you’ve noticed there’s nothing field-specific about Volatile[Read|Write], it makes everything flush, not just the field in question.  The performance aspect of the volatile modifier comes into play when you have multiple volatile fields within a class, it’s really only the last one that needs to use VolatileWrite when writing and the first one to use VolatileRead when reading. To steal some code from Jeffery Richter:

m_value = 5;
Thread.VolatileWrite(ref m_flag, 1);

is just as thread-safe as:

m_value = 5;
m_flag = 1;

if m_value and m_flag were volatile.

Same holds true for VolatileRead:

if(Thread.VolatileRead(ref m_flag) == 1)
	Display(m_value);

is just as thread-safe as the following with volatile fields:

if(m_flag == 1)
	Display(m_value);

I feel compelled to bring up invariants here.  When you’re dealing with modifications or accesses of multiple things that can’t operate atomically (i.e. the “validity” of m_value depends on the value of m_flag, making the two fields an “invariant” that can be accessed by a single instruction—i.e. not atomically).  While the above example ensures that changes to m_flag and m_value are made visible to other threads “at the same time” it doesn’t do anything to stop another thread from accessing the m_value before m_flag has been updated.  This may or many not be “correct”.  If it’s not correct, using lock or Monitor to model atomic access to such an “invariant” that isn’t natively atomic is a better choice.

On the topic of lock/Monitor.  It’s important to note that the end of a lock or the call to Monitor.Exit has release semantics and the start of the lock or Monitor.Enter (and variants) have acquire semantics.  So, if all access to a field is guarded with lock or Monitor, there’s no need for volatile or Thread.Volatile[Read|Write].

Using Monitor, lock, volatile, VolatileRead and VolatileWrite correctly shows that you understand what it means to be thread-safe and that you understand what it means to be externally visible and when in the context of your fields and invariants.

There have apparently been some discussions about the usefulness and applicability of volatile in C# as well as VB (and I assume C++11); but, I’m was not privy to those discussions and haven’t been able to find much in the way of reference to those discussions other than third-hand information…  Needless to say, VB doesn’t have anything similar to “volatile”, apparently for a reason.

I’m not saying there aren’t perfectly valid scenarios for volatile; but, look carefully at what you need; you probably could make better use of VolatileRead or VolatileWrite.  Just understand your needs and use what is correct—code on purpose.

[1] volatile (C# Reference)

[2] Acquire and Release Semantics

[3] CSC and JIT compilers.

[4] Sayonara volatile

[5] Aug 2011 Getting Started with Threading webcast with Jeffrey Richter (45MB)

[6] C# Language Specification (ECMA-334)

C#, Async, Limits, oh my!

One of the great sessions at Codemash was a dual-speaker session with Bill Wagner and Jon Skeet—Async from the Inside.

In that session Bill and Jon describe (in great detail) the state machine that the C# compiler generates when it compiles async code involving the await keyword.  When the Async CTP was released this state machine was one of the first things I noticed when I was reflecting through some generated code.  I too noticed the type of the state variable (int) and wondered, at the time, if that would be an issue.  All the information Bill, Jon and I portray is what we’ve learned from reflection, none of this is “insider information”.

Some background: a state machine is generated for each method that uses the await keyword.  It manages weaving back-together the asynchronous calls that you so lovingly linearly describe in an async method.  The state machine not only weaves together the various “callbacks” that it generates, but it also (now) manages the state changes of the data being modified between and by asynchronous calls.  Wonderfully complex stuff.

The state variable is basically an enum of the various states between async calls that the generated code can be in.  I won’t get into too much detail about that (you should have gone Codemash like all the other cool people :) but needless to say having a “limit” on the number of states a method can be in sounds a bit scary.

Of course Jon, at one point, brought the same thing up in the session about the “int” state variable.  This reminded me that I wanted to look into it further—not because I wanted to break the compiler (directly) but to know what the limits are, if any.

A couple of days later while I had some time waiting around at airports on my way home.  I thought I’d test my theory.  If you follow me on Twitter you probably saw some of my discoveries in real time.  For those of you who didn’t, here’s an aggregation of what I found.

First of all, the state machine represents the async states a single method can be in.  This is represented by a signed int.  the only negative value that seems [1] to mean anything is –1, leaving 2,147,483,647 states (or, roughly, await invocations) that can occur in an async method.

First glance, this seems disconcerting.  I quickly wrote some code to generate a method with 2,147,483,648 await invocations in it (that’s Int32.MaxValue + 1 for those without OCD).  Needless to say, that took a few minutes to generate on my laptop, and I have an SSD (which I’ve clocked at increasing my IO by about 7 times on average).  That generated a 45 gigabyte file (one class, one method).

Problems with the type int for the state variable started to seem ridiculous.  But, I didn’t stop there.

Now I’m doing everything outside of Visual Studio (VS) from this point on.  I’m running the app to generate the code from within VS; but everything else is from the command-line.  So, I run CSC on the 45 gig file honestly expecting and internal compiler error.  But, what happens is that I get a CS1504 error ‘file.cs’ could not be opened (‘Arithmetic result exceeded 32-bits. ‘).  Now, the state variable is 32-bits so, it sounds like that could be the culprit.  But, if you look at the error message, it can’t even open the file.  I tried opening the file in VS and it told me the file couldn’t be found…

Okay, at this point it’s seeming even more unlikely that a type of int for the state variable is even remotely going to be an issue.  But, now I’m curious about what 32-bit value has been exceeded.  My theory is now the number of lines in the file…  The compiler has to keep track of each line in the file in case it has to dump an error about it, maybe that’s the 32-bit value? I modify my code generation to limit the the number of await invocations so the number of lines in the file is 2,147,483,647 (kind of a binary search, if this works then I know it still could be the number of lines in the file). Same error.

The error isn’t from the # of lines.  Now my theory is that the overflow is from trying to allocate enough memory load the file (keep in mind, I’ve got 8 GB of RAM and I’m trying to load a 45GB file; but, I have yet to get an out of memory error).  So, I modify my code to generate a file that is approaching 2,147,483,647 bytes in size.  Things are much faster now…  I try again. Now I get the same CS1504 error but the message is ‘file.cs’ could not be opened (‘Not enough storage available to complete this operation’) (I’ve got 100 GB free space on the drive…).  Interesting.  I’ve lessened the data requirement and only now effectively getting “out of memory” errors.

Now I’m just looking for a file that the compiler will load—I’ve given up on some super-large number of await statements…  Longer story, short, I kept halving the size of the file until I reached about 270 megabytes in size then the compiler finally succeeded (meaning ~540 Megabytes failed).

At this point, I’ve successfully proven that a type of int for the state variable is not an issue.  If the compiler could load the 540 megabyte file and I somehow could use 8 bytes per await invocation (“await x;” for example) then I could never reach more than about 70,778,880 await calls in a single method.  Of course, I’m way off here; that number is even much lower; but 70,778,880 is about 3% of 2,147,483,647.  Clearly int is the smallest type that could store anything close to 70,778,880 or less…

Of course, I’m completely ignoring the fact that a 540 MB cs file is completely unmanageable in VS or a project in general; but, why get caught up in silly realities like that.

This state machine is almost identical to those generated by enumerator methods (yield return).  If we assumed that the async state machine generation code is “inherited” (by pragmatic reuse) from the enumerator method state machine generator, we can assume it has very similar limits (but even smaller)—meaning you’d never get close to overflowing its int state variable.

HTMYL.

[1] Again, this all observed; I’m surmising the compiler never uses any other negative value.

The TeamCity Database Migration Documenation Could Use Some Work

I have a client who wants all the benefits of ALM but doesn’t really want to spend the money on them.  They’re a start-up; so, I can respect that. As a result, I’ve been getting them up and running with some of the freely-available and open-source tools.  They’re also cross-platform, so getting on some of the great Microsoft programs like BizSpark and WebsiteSpark wasn’t really in the cards.

Anywho…  One of those tools was TeamCity.  It’s a great too for Continuous Integration and Deployment.  I’ve used it with a couple of clients now and have nothing but good things to say about it.  When I installed it, I probably read some yellow box talking about HSQLDB and External Database and evaluating the system and production; but, honestly, just follow the steps. :).  I created several configurations and had artifacts and build logs, so upgrading needed to include backup and restore…  I had a couple spare cycles to look into migrating to an external database (as well as run through a backup/restore cycle) so I thought I might as well bite the bullet.

Well, the title of the post explains it.  I’m not going to go too much into the existing documentation and how it jumps from page to page before you actually get a migration done; I’m just going to say there wasn’t just a single list of steps (and, I think a couple things were missing).  There’s support for multiple platforms and multiple databases, so I get it’s hard.  It’s seemed a bit too hard; so, I’m going to detail the process of migrating a TeamCity database to a SQL Server Express database here in case anyone else might find it useful.

[NOTE: I did this with TeamCity 6.5.4 build 18046, YMMV]

First off, backup.  I did this from the web interface.  I clicked on the Administration link near the top right, clicked the Backup Data link on the right-hand list, selected “Custom” scope and checked everything.  This created a zip file in the backup directory in the TeamCity data directory. For me, this was C:\TeamCity\.BuildServer\backup\TeamCity_Backup_20111130_210020.zip—but, yours will be different.

After doing a TeamCity backup, stop the TeamCity services.  (Start menu, Control Panel, … you know the drill … or net stop TCBuildAgent and net stop TeamCity from a command prompt).

Next, I did a file system backup.  I just copied the TeamCity directory to another folder.  I had installed TeamCity to C:\TeamCity.  I think the default was my user folder; but, I’m only at my client for a limited time and when I’m gone that’s going to cause a bit of grief (yeah, it happened :), so that meant copying C:\TeamCity to somewhere else on C:\.  I’ve digressed from the the Jetbrains instructions already…

Next, I set up the external SQL Server Express database.  I already had SQL Server 2005 Express installed, so, I won’t go through that installation process.  In SQL Server Management Studio I created a “TeamCity” database, and added a “TeamCity” login—you’ll need those later.

Next, I installed some JDBC drivers for SQL Server.  I just grabbed the Microsoft ones—the “Native” driver—I figured they might know a thing or two about writing a SQL Server driver.  I downloaded from here. I unzipped (or ran the EXE) into a directory then copied the sqljdbc4.jar file into the lib\jdbc TeamCity data sub-directory (which was C:\TeamCity\.BuildServer\lib\JDBC for me).

Next, I configured TeamCity to use SQL Server.  I created a database.properties file in the config data folder (C:\TeamCity\.BuildServer\config, for me).  It’s later: the file was similar to:

connectionUrl=jdbc:sqlserver://localhost\\sqlexpress:1433;databaseName=TeamCity
connectionProperties.user=TeamCity
connectionProperties.password=P@nda$

…names and passwords might have been changed to protect the pandas.  Since I’m using SQL Express I had to throw the \\sqlexpress named-instance stuff in there.  If you’re using a different named instance, the”sqlexpress” part will be different.  If you’re using a “real” install of SQL Server, you don’t need the “\\sqlexpress” part at all, e.g. “//localhost:1433”.  I also had to throw the :1433 stuff in there (long story), you may not need that.

At this point, be sure not to be running the TeamCity services and don’t even think about logging into the TeamCity web interface (well, you can’t if the server isn’t running).  As far as TeamCity is concerned at this point, you’ve got a fresh install.  If you log in now it will ask you to create an Administrator login and create all the database tables.  The next step requires an empty database and will fail if you do create that login…

Now, restore.  Okay, I lied, this is one of those parts where the documentation has misled you.  The restore requires that you have an empty config directory—i.e. you can’t have database.properties in the config directory.  You also need an empty system directory (C\TeamCity\.BuildServer\system\ for me)—a fresh install might mean those directories are empty, I don’t know.  I simply renamed config to config-old and system to system-old and created new config and system directories.  Then, I had to add the Java bin directory to the PATH.  TeamCity has all the Java binaries included with it—it doesn’t actually install Java (oh, BTW, TeamCity uses Java).  For me, I just ran path=%path%;c:\TeamCity\jre\bin to let the command prompt know where java.exe is.  If you have Java installed, you might not need to do this.  Then, I performed a restore; which is just a matter of running maintainDB with the restore command from the TeamCity bin directory (C:\TeamCity\bin directory—note no “.BuildServer”—for me):

…which tells TeamCity to restore from my backup (zip) file, where the data directory is, and what database configuration to use (note the config-old business).  Either TeamCity or Java doesn’t grok relative directories; so, note that I used “C:\TeamCity” instead of “..\”.

Once the restore is done (and it’s kind enough to remind you :), copy the database.properties file that you used to the config directory (for me, this was just copy ..\.BuildServer\config-old\database.properties ..\.BuildServer\config ).

Then, restart the TeamCity services:

And you’re done.  If you can now log into the TeamCity web interface with all your old logins and all your old configurations are still there…  I did a bit of housework by deleting config-old, system-old, and went ahead and deleted the C:\TeamCity copy that I made. (I don’t need that, now that I’ve migrated to SQL Server and verified that everything works).

If you ran into a problem along the way, you could simply copy the backed-up TeamCity directory over-top the old one (making sure the services were stopped first) and you should be back to where you were before.  I didn’t have a problem, so I can’t confirm that actually works; but the docs detail that it does.

Since you have to have an empty database and empty config/system directories, I gather the actual restore process would be identical (minus dropping all the tables in the TeamCity SQL Server database).

If You’re Using “#if DEBUG”, You’re Doing it Wrong

I was going through some legacy code the other day, refactoring it all over the place and I ran into many blocks of code wrapped in “#if DEBUG”.  Of course, after a bit of refactoring in a RELEASE configuration these blocks of code were quickly out of date (and by out of date, I mean no longer compiling).  A huge PITA.

For example, take the following code:

	public class MyCommand
	{
		public DateTime DateAndTimeOfTransaction;
	}
 
	public class Test
	{
		//...
		public void ProcessCommand(MyCommand myCommand)
		{
#if DEBUG
			if (myCommand.DateAndTimeOfTransaction > DateTime.Now)
				throw new InvalidOperationException("DateTime expected to be in the past");
#endif
			// do more stuff with myCommand...
		}
		//...
	}

If, while my active configuration is RELEASE, I rename-refactor DateAndTimeOfTransaction to TransactionDateTime then the block of code in the #if DEBUG is now invalid—DateAndTimeOfTransaction will not be renamed within this block.  I will now get compile errors if I switch to DEBUG configuration (or worse still, if I check in an my continuous integration environment does a debug build).

I would have run into the same problem had I been in a DEBUG configuration with all the “#if RELEASE” blocks.  Yes, I could create a configuration and define DEBUG and RELEASE and do my work in there.  Yeah, no, not going down that twisted path… i.e. try doing it without manually adding a conditional compilation symbol.

I got to thinking about a better way.  It dawned on me that #if blocks are effectively comments in other configurations (assuming DEBUG or RELEASE; but true with any symbol) and that comments are apologies, and it quickly became clear to me how to fix this.

Enter Extract Method refactoring and Conditional Methods.  If you’ve been living under a rock or have simply forgotten, there exists a type in the BCL: System.Diagnostics.ConditionalAttribute.  You put this attribute on methods that[whose calls] may or may not be included in the resulting binary (IL).  But, they are still compiled (and thus syntax checked).  So, if I followed the basic tenet for code comment smells and performed an Extract Method refactoring and applied the ConditionalAttribute to the resulting method, I’d end up with something like this:

		//...
		public void ProcessCommand(MyCommand myCommand)
		{
			CheckMyCommandPreconditions(myCommand);
			// do more stuff with myCommand...
		}
 
		[Conditional("DEBUG")]
		private static void CheckMyCommandPreconditions(MyCommand myCommand)
		{
			if (myCommand.DateAndTimeOfTransaction > DateTime.Now)
				throw new InvalidOperationException("DateTime expected to be in the past");
		}

Now, if I perform a rename refactoring on MyCommand.DateAndTimeOfTransaction, all usages of DateAndTimeOfTransaction get renamed and I no longer introduce compile errors.

Code Contracts

If you look closely at the name I chose for the extracted method you’ll notice that what this DEBUG code is actually doing is asserting method preconditions.  This type of thing is actually supported in Code Contracts.  Code Contracts implement a concept called Design by Contract (DbC).  One benefit to DbC is that these checks are effectively proved at compile time so there’s no reason to perform this check at runtime or even write unit tests to check them because code to violate them becomes a “compile error” with Code Contracts.  But, for my example I didn’t use DbC despite potentially being a better way of implementing this particular code.

Anyone know what we call a refactoring that introduces unwanted side-effects like compile errors?

[UPDATE: small correction to the description of conditional methods and what does/doesn't get generated to IL based on comment from Motti Shaked]

Working with Subversion Part 2 (or Subversion for the Less Subversive)

In my previous Subversion (SVN) post I detailed some basic commands for using SVN for source code control.  If you’re working alone you could get by with Part 1.  But, much of the time you’re working in a team of developers working on versioned software that will be deployed to multiple places.  This post will detail some more advanced commands for systematically using SVN to simultaneously work on multiple editions and versions of a software project. (hence the “less subversive”).

Practices

The basic concept of working on two editions or two versions of a software project at the same time is considered “branching”.  Simultaneous work is each done on it’s own “branch” (or “fork”) of the project.  Branch can be considered “copy”; but that’s generally only true at the very start of the branch.  With any project you can add and remove files from it; at which point a branch ceases to become a “copy”.  Semantics aside, SVN doesn’t really have a concept of a “branch”.  This is a practice used by SVN users.  SVN really only knows how to “copy” and “merge”.  The practice is the create a standard location to control branches, copy existing files and directories to a branch location, and merge changes from a branch to another location (like the trunk).  To support branching many repositories (repos) have a “trunk” and “branches” folder in the project root.  Trunk work is obviously done in the “trunk” folder and branches are recreated in the “branches” folder.

Branching Strategies

Before getting too much farther taking about things like “trunk”; it’s a good idea to briefly talk about branching strategies.  The overwhelming branching strategy for SVN users has been the “trunk” or “main line” branching strategy.  This strategy basically assumes there is really one “main” software product that evolves over time and that work may spawn off from this main line every so often to work independently and potentially merge back in later.  In this strategy the “current” project is in the “trunk”.

Another strategy is sometimes called the Branch per Release strategy which means there’s a branch that represents the current version of the project and when that version is “released” most work transitions to a different branch.  Work can always more from one branch to another; but there’s no consistent location that project files are located over time.  This a perfectly acceptable strategy and almost all Source Code Control (SCC) systems support it.  The lack of a consistent location of files makes discovery difficult and it really forces the concept of branching onto a team.  I’ve never really found this strategy to be very successful with the teams I’ve worked on, so I prefer the trunk strategy.

Branching

Branching is fairly easy in SVN.  The recommended practice is the perform a copy on the server then pull down a working copy of that branch to perform work on.  This can be done with the svn copy command, for example:

svn copy http://svnserver/repos/projectum/trunk http://svnserver/repos/projectum/branches/v1.0 -m "Created v1.0 branch"

…which makes a copy of the trunk into the branches/v1.0 folder.

Now, if you checkout the root of the project (e.g. svn checkout http://svnserver/repos/projectum) you’ll have all the files the trunk and all the branches.  You can now go edit a file in branches/v1.0 in your working copy and svn commit will commit that to the branch.  If you want to just work with a branch in your working copy you can checkout just the branch.  For example:

svn checkout http://svnserver/repos/projectum/branches/v1.0 .

..which makes a copy of all the files/directories in v.1.0 to the current local directory.  So, if you had …/branches/v1.0/readme.txt in the repo, you’d now have readme.txt in the current local directory.

Same holds true for the trunk, if you want to work on files in the trunk independently of any other branches checkout just the trunk, for example:

svn checkout http://svnserver/repos/projectum/trunk .

It’s useful to work just with the trunk or just with a branch because over time you may have many branches.  Pulling down the trunk and all branches will get time-consuming over time.

While SVN doesn’t really have the concept of a “branch”, it does know about copies of files and tracks changes to those copies.  So, if you shows a log of the changes to a file you’ll see the commit comments for all the branches too.  For example, edit readme.txt in the branch directory and commit the change (svn commit –m "changed readme.txt" for example) then go back to the trunk directory and show the log of readme.txt, for example:

svn log –v readme.txt

…you’ll see the commit comments for both trunk/readme.txt and branches/v1.0/readme.txt.  For example":

------------------------------------------------------------------------
r2 | PRitchie | 2011-11-17 10:23:36 -0500 (Thu, 17 Nov 2011) | 1 line
Changed paths:
   A /trunk/readme.txt
initial commit
------------------------------------------------------------------------

Merging

Okay, you’ve been working in v1.0 for a few days now, committing changes for v1.0.  One of those changes was a bug fix that someone reported in v1.0.  You know that bug is still in the trunk and it’s time to fix it in the trunk too.  Rather than go perform the same editing steps in the trunk, you can merge that change from the v1.0 branch into the trunk. For example, a typo fixed in the readme.txt needs to be merged into the trunk, from a clean trunk working copy (no local modifications):

svn merge http://svnserver/repos/projectum/branches/v1.0

This merges the changes from v1.0 into the working copy of trunk.  You can now review the merges to make sure they’re want you want, then commit them":

svn commit –m "merged changes from v1.0 into trunk"

Merge Conflicts

Of course, sometimes changes are made in the trunk and a branch that conflict with each other (same line was changed in both copies).  If that happens SVN will give you a message saying there’s a conflict:

Conflict discovered in 'C:/dev/projectum/trunk/readme.txt'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options:

You’re presented with several options.  One is to postpone, which means the conflict is recorded and you’ll resolve it later.  You can see the differences with diff-full.  Or you can accept the left or the right file as iss with mine-conflict or theirs-conflict—to accept the local file, use mine-conflict.  There’s also edit, which will allow you to edit the merged file showing the conflicts.  For example:

one
<<<<<<< .working
three=======
four >>>>>>> .merge-right.r9

This details that your working file has a changed second line “three” but the remote version has a changed second line “four “.  The lines wrapped  with <<<<<<< .working and ======= is the local change, which is followed by the remote change and >>>>>>> .merge-right.r9.  The “merge-right.r9” is diff syntax telling you which side the change came from and which revision (r9 in this case).  You can edit all that diff syntax out and get the file the way you want to merge it, then save it.  SVN will notice the change and present options again:

Select: (p) postpone, (df) diff-full, (e) edit, (r) resolved,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options:

Notice you now have a resolved option.  If your edits fixed the conflict you can choose resolved to tell SVN the conflict is gone.  You can then commit the changes and the merges will be committed into the repo.

Merging Without Branches

Of course branches aren’t the only source for merges.  You might be working on a file in the trunk that a team member is also working on.  If you want to merge any changes they’ve committed into your working copy, you can use the update command.  For example:

svn update

This will merge any changes files with your local files.  Any conflicts will appear the same way they did with svn merge.

It’s important to note that with SVN you can’t commit changes if your working copy was out of date with the repo.  (e.g. someone committed a change after you performed checkout).  If this happens you’ll be presented with a message similar to:

Transmitting file data .svn: E155011: Commit failed (details follow):
svn: E155011: File 'C:\dev\projectum\trunk\readme.txt' is out of date
svn: E170004: Item '/trunk/readme.txt' is out of date

This basically just means you need to run svn update before you commit to perform a merge and resolve any conflicts.

Mouse Junkies

If you don’t generally work with command-line applications and don’t care for the speed increase of not using the mouse, there’s some options you can use to work with SVN.

TortoiseSVN

TortoiseSVN is a Windows Explorer extension that shows directories/files controlled with SVN differently within Explorer.  It will show directories and files that have been modified or an untracked with different icons (well, icon overlays) within Explorer.  It will also let you perform almost all SVN commands from with explorer (via context menu).

VisualSVN

VisualSVN provides much of the same functionality as TortoiseSVN but does it within Visual Studio.  It’s not Source Code Control Visual Studio extension; which is interesting because you can use VisualSVN and another source code control extension (like TFS) at the same time.  VisualSVN requires TortoiseSVN to work correctly, so install that first.

Both TortoiseSVN and VisualSVN make dealing with merge conflicts easier.  I recommend using these for merging instead of the command-line.

Resources

http://svnbook.org/

“Explicit” Tests with Resharper

NUnit introduced a feature called Explicit Tests (a long time ago, I believe) that basically mean a test is considered tagged as Ignore unless the test name is explicitly given to the NUnit runner.

This is useful if you have tests that you don’t want run all the time.  Integration tests or tests highly coupled to infrastructure or circumstance come to mind…  But, it’s difficult to automate these types of tests because you always have to maintain a list of test names to give to the runner.

The ability of NUnit to run explicit tests aside, I don’t generally use the NUnit runner directly; I use other tools that run my tests.  I use tools like Resharper to run my tests within Visual Studio, ContinuousTests for continuous testing, and TeamCity to run my tests for continuous integration.

Continuous integration is one thing, I can configure that to run specific test assemblies and with specific configuration and get it to run whatever unit tests I need it to for whatever scenario.

Within Visual Studio is another story.  I sometimes want to run tests in a class or an assembly but not all the tests.  At the same time I want the ability of the runner to run tests it wouldn’t normally run without having to edit and re-compile code.

With Resharper there are several ways you can do this.  One way is to use the Ignore attribute on a test.  This is effectively the same as the NUnit Explicit attribute.  If I run the test specifically (like having the cursor within the test and pressing Ctrl+U+Ctrl+R) Resharper will still run the test.  If I run all tests in solution/project (Ctrl+U+Ctrl+L/right-click project, click Run Unit Tests) the test is ignored.  This is great; but now this test is ignored in all of my continuous integration environments.  Sad Panda.

If you’re using NUnit or MSTest (amongst others that I’m not as familiar with) as your testing framework you can tag tests with a category attribute (MS Test is “TestCategory”, NUnit is “Category”).  Once tests are categorized I can then go into Resharper and tell it what category of tests to “ignore” (Resharper/Options, under Tools, select Unit Testing and change the “Don’t run tests from categories…” section in Resharper 6.x).  Now, when I run tests, tests with that category are effectively ignored.  If I explicitly run a test (cursor somewhere in test and I press Ctrl+U+Ctrl+R) with an “ignored” category Resharper will still run it.  I now get the same ability as I did with the Ignore attribute but don’t impact my continuous integration environment.  I’ve effectively switched from an opt-in scenario to an opt-out scenario.

With the advent of ContinuousTests, you might be wondering why bother?  That’s a good question. With ContinousTests only the tests that are affected by the changes you’ve just saved—automatically, in the background.  In fact, having any of your tests run whenever you make a change that affects the test is one reason why I make some tests “explicit”.  I tend to use test runners as hosts to run experimental code, code that often will become unit tests.  But, while I’m fiddling with the code I need to make sure it’s only run when I explicitly run it—having it run in the background because something that affects the test isn’t always what I want to do.  So, I do the same thing with ContinousTests: have it ignore certain test categories (ContinuousTess/Configuration (Solution), Various tab, Tests categories to ignore).

Test Categorization Recommended Practices

Of course, there’s nothing out there that really conveys any recommendations about test categorization.  It’s more or less “here’s a gun, don’t shoot yourself”…  And for the most part, that’s fine.  But, here’s how I like to approach categorizing tests:

First principle: don’t go overboard.

Testing frameworks are typically about unit testing—that’s what people think of first with automated testing.  So,  I don’t categorize unit tests.  These are highly decoupled tests that are quick to run and I almost always want to run these tests.  If the tests can’t always run or I don’t want them run at any point in time, they’re probably not unit tests.

Next, I categorize non-unit tests by type.  There’s several other types of tests like Integration, Performance, System, UI, Infrastructure, etc.  Not all projects need all these types of tests; but these other tests have specific scenarios where you may or may not want them run.  The most common, that I’ve noticed, is Integration.  If the test has a large amount of setup, requires lots of mocks, is coupled to more than couple modules, or takes a long time to run, it’s likely not a unit test.

Do you categorize your tests?  If so, what’s your pattern?

Working with Subversion, Part 1

Working with multiple client projects and keeping abreast of the industry through browsing and committing to open source and other people's libraries means working with multiple source code control (SCC) systems.  One of the systems I use is Subversion (SVN).  It's no longer one of the SCCs I use most often so I tend to come back to it after long pauses and my SVN fu is no longer what it used to be.  I'm sure my brain is damaged from this form of "task switching", not to mention the time I spend trying to figure out the less common actions I need to perform on a repository (repo).  I usually spend more than few minutes digging up the commands I need for the once-in-a-decade actions I need to perform. 

I don't foresee getting away from SVN in the near future; so, I'd thought I'd aggregate some of these commands into one place.  My blog is the perfect place to do that (because it's just for me, right? :)

Backing Up

Outside of adding/committing, the most common action to be performed is backing up the repository.  Unfortunately for my brain this is automated and I don't see it for months at a time. To back up an SVN repository that you're not hosting or being hosted by third-party software (like VisualSVN Server) then I like dump/load:

svnadmin dump repo-local-path > repo-bkp-path

This let's you restore to the host that contains all the configuration data like permissions, users, and hooks.

If the repository is completely autonomous (i.e, just a directory on your hard drive and maybe an SVN daemon) then hotcopy is better:

svnadmin hotcopy local-path destination-path

Restoring

If you used the dump method of backing up, you need to use the load command to put the backup into an existing repository.  If you're not using a hosted repository, you'll first need to create the repository (svn create repo-local-path) to run load (in which case I'd recommend using hotcopy instead).  To load the dump into the existing repository:

svnadmin load repo-local-path < repo-bkp-path

If you’ve used hotcopy then the backup is the fully functional repository; just make it available to users (i.e. put it where you want it :).

Migrating

Migrating is basically just a backup and restore.  If you're backing up one repository and putting into an existing repository, use dump/load.  On System A

svnadmin dump repo-local-path > repo-bkp-path

On System B after copying repo-bkp-path from System A

svnadmin load repo-local-path < repo-bkp-path

Even if you weren't migrating to an existing repo, you could use this method; just add svnadmin create repo-local-path before svnadmin load.  The dump/load method has the added benefit of upgrading the data from one format to another if both systems don’t have the same version of SVN running.  The drawback of migrating with dump/load is that you’ll have to configure manually (or manually copy from the old repo) to get permissions, hooks, etc…

Now you've migrated your repo to another computer, existing working copies will be referencing the old URL.  To switch them to the new URL perform the following:

svn switch --relocate repo-remote-URL

Creating Working Copy

If you don’t already have a local copy of the repo to work with, the following command

svn checkout repo-remote-URL working-copy-path

Committing Changes

I added this section because I’ve become used to GIT.  GIT has a working directory and staging area model; so you tag files in the working directory for staging before committing.  This allows you to selectively commit modifications/additions.  SVN is different in that the working directory is the staging area so you effectively have to commit all modifications at once.  (you can stage adds because you manually tell SVN which files to start controlling).

svn status will tell you what’s modified (M) and what’s untracked (?).  To commit all modified

svn commit –m "description of changings included in the commit"

Undoing Add

Sometimes you a schedule something to add on commit by mistake or it’s easier to add by wildcard and remove the files that you don’t want to commit on your next commit.  To remove them from the next commit:

svn revert file-path

Be careful with revert because if the file is already controlled this will revert your local modifications.

Undoing Modifications

To revert the modifications you’ve made locally and restore a file to current revision in the repo:

svn revert file-path

Resources

    http://svnbook.org/

What’s your favourite SVN workflow?

Getting a Quick Feel for a New Software Team

I deal with many different teams and many different companies.  I see a lot of teams not get the benefit of known techniques and fail to evaluate and improve.  There's some check-lists out there that are used to evaluate a company/team before joining it; but I find the lists to be deeply rooted in the past.  They detail such fundamental things that knowing the criteria on the list really doesn't tell you much about how your time with this team will be or your role within it.

There's so many low-level features within software development team to aid in successfully delivering software.  But these low-level features are part of higher-level initiatives that if absent make the lower-level features almost moot.  "Do you use source code control?" for example.  Sounds like this is a fundamental part of being able to delivery quality software; and it is.  But, on its own it doesn't really do a whole lot to help the team deliver software.  Is the code put into source code control using accepted design practices?  Is the code put into source code control in a timely fashion.  Is the code put into source code control not impeding other people?  Etc. etc. etc.  "Yes" to "do you use source code control" without any of other initiatives that follow-on doesn't give me a warm and fuzzy feeling about getting on the team and focusing on software value and not spending a lot of time on, or stuck in, process.

Over the years I've been on many teams hired for a specific role that changed drastically almost as I began working with the team.  I've observed many things about many teams and have come up with some things I like to find out about a team before I start so I can better gauge the type of work that I'll be doing and how successful the team will be fulfilling their goals.

Does the team:

• have a known product owner/sponsor,
• have a cross-functional team 6-9 people in size,
• use appropriate tools,
• foster SOLID design principles,
• use continuous integration,
• use continuous deployment,
• foster communications with team members and stakeholders,
• have a known and managed process and visualize workflow,
• evaluate and improve process as a team,
• have new candidates write code in their interviews,
• have a plan that limits work in progress,
• have a plan that orders or prioritized tasks;

and, how easy is it to change to start doing any of the above items?

“Do you use source code control” is covered by “Does the team use continuous integration” as it’s not just about using source code control, it’s about a process that can’t function properly without source code control.

And, for what it's worth, this list doesn't tell you whether you should work on a team or not; it just tells you the type of work you will be doing.  It's up to you to dig deeper and decide whether you unrealistically want to limit your work to a specific or a small set of roles.  I would only use the last question as a acid test of whether I would join the team or not.  If they are not willing to improve, there's not much I'm going to be able to do for them.

What do you look for in a team/project?

DevTeach Comes to Ottawa!

DevTeach is coming to Ottawa November 2-4 2011.

DevTeach has had some of the best speakers in the industry and some of the best sessions/tracks that I’ve ever attended.

This year is no exception, the speaker line-up is full of experts and MVPs like David Starr, Mitch Garvis, James Kovacs, Laurent Duveau, Mario Cardinal, Guy Barrette, Paul Neilson, Rob Windsor, Benjamin Day, Erik Renaud, Joel Semeniuk, etc. and locals like Garth Jones, Wes MacDonald, Joel Hebert, Colin Smith, Colin Melia, Brad Bird, Rick Claus, etc.

Tracks include Agile, Web Dev, Rich Client, SQL Server Dev, SQL Server BI, and for the first time at DevTeach, an IT Pro track.

If you think you might have trouble convincing your boss you need to go, have a look at http://www.devteach.com/WhoShouldAttend.aspx.

To register: http://www.devteach.com/Register.aspx

Ottawa Speaker’s Idol Nov. 2nd at DevTeach Ottawa

Do you like to share your IT knowledge with others? Do you know a tip, trick or a feature that can save time for others and you would like to share it?

Well, Ottawa Speaker Idol is your solution and there is still time to submit your session proposal. The deadline is Wednesday Oct. 26^th.

The Ottawa IT Community is organizing a contest where you can win a free registration to DevTeach Nov. 3-4 (value of $649). On November 2nd, at the venue for DevTeach Ottawa (Ottawa Conference Center), you will be invited to present your tip, trick or a feature for 10 minutes. After all presentations have completed, the attendees will vote for the winners. Prizes will be distributed to the best presenters. This is a free event and everyone is welcome to attend.

Date: Wednesday November 2^nd , at 18:30
Location: Ottawa Conference Center (room 103)

55 Colonel By Drive

Ottawa , ON, K1N 9J2 Canada
Rules:

• You must not be a regular speaker in the IT Community. This contest is to encourage new speakers.
• It must be the first time you participate to the Ottawa IT Speaker Idol
• No need to be register to DevTeach to participate in the contest.
• You must submit the description of your presentation (no more than 200 words) by October 26^th to jrroy@DevTeach.com
  
• We will accept 10 presenters
• If more than 10 submissions, a committee will select 10 presentations
• The list of presentations will be announced on October 31st
• A laptop will be available for your presentation or you can use your own laptop
• You will need to bring your PowerPoint and code on a USB key. All presentations will be posted on our web site after the contest.

Prizes:

First place:    Registration to the main conference DevTeach Nov. 3-4 (value of $649) from DevTeach Inc.

Second place:    Win 7 Phone (Value of $549) from LIKE 10 Inc.

Third place:    TBA from Microsoft

Windows 8, What Does It Mean to Me?

Well, unless you’ve been hiding under a rock, you’ve probably heard about Microsoft Build conference that happened in Anaheim last week.  It was during this conference that Microsoft finally released details about the successor to Windows 7.  This event is solely developer-focused and, in my opinion, tablet-specific.  It went into a lot of detail about the added APIs and usability changes in Windows 8 to better support tablet and touch-based computers.

First off, the new touch-first usability changes are called “Metro” and applications written for touch are called “Metro-style apps”.

The term “touch-first” is used because it’s not touch-only.  Metro will work fine with a mouse and a keyboard; but the user experience is obviously designed for a multi-touch device and that you can “get by” adequately with a mouse and a keyboard.  This was done for various reasons, one being that touch isn’t good for entering reams of text and that you want to dock your tablet and use a keyboard and mouse when you want to do something that would be better served by a keyboard or a mouse.  e.g. tablet/touch-based UX is better for consuming information, whereas keyboard is better for producing information.

So, with all the focus was on tablet and touch, does that mean Windows 8 is all about Metro?  The focus of the conference was to show off (likely in answer to other vendors producing tablet-based hardware and systems) tablet- and touch-based user experiences.  In fact, paying attendees received a custom-made Samsung tablet computer with a version of Windows 8 (called Developer Preview) and the next version of Visual Studio (called Dev11, for now) pre-installed.  This puts the focus of Developer Preview almost solely on touch.  But, does this mean that mouse/keyboard was ignored or that we need a touch screen on our computers to be able to use Windows 8 when it is released?  No, of course not.  Discussing various things with numerous Softies, the message time and time again was that if something isn’t mentioned, or doesn’t have focus, that doesn’t mean it won’t exist in the release version or in the near future.  It was clear that this version of Windows 8 was produced strictly for this event, and since it was effectively deployed on tablet, it had a tablet edition of the OS.  I.e. if you by a desktop without a multi-touch monitor after Windows 8 comes out, the edition of Windows 8 that will be deployed will not likely have Metro on by default.  But, that’s my opinion.

We now have Metro apps, and they’re XAML-based and very similar to Silverlight.  You initialize the runtime slightly differently and do some other things slightly differently; but if you’re a Silverlight developer you’ll be right at home in Metro.  Does that mean all Windows 8 development is Metro?  Definitely not.  It’s not C++ based and it is still managed code backed by the CLR (they shows using VB and C#; but F# support should be there for RTM).  The CLR backing Metro apps is the same CLR that backs “Desktop” apps written as WinForms (yes, they’re still supporting that) traditional Silverlight, Console apps, Windows services apps, etc. etc. etc.

Does this mean Silverlight and .NET is dead?  Well, no; but you could interpret it that way.  Metro apps are driven by a new Windows API called WinRT—which is basically a object-oriented alternative for the Win32 API and removes blocking methods leaving, or replacing them with, asynchronous methods.  This and the CLR that backs it are inherent parts of Windows 8.  So, it’s technically not it’s own product and simply just another part of Windows.  So, you could say that .NET and Silverlight are just bits of Windows 8 and thus no longer exist as “.NET” and “Silverlight”.  Just like Hydra or Terminal Services really doesn’t exist any more because it’s baked right into Windows.  But, that doesn’t mean they don’t exist in this new form.

Some discussion has occurred as to whether WinRT is truly CLI-compliant because it doesn’t support all the types and methods that .NET does now.  This isn’t true.  We had these same discussions when Silverlight and Windows Phone came out.  WinRT is just another “profile”.  It has a subset .NET API.  Just like how Silverlight doesn’t include classes like System.Windows.Forms.Form and Windows Phone doesn’t include classes like System.Windows.Browser.HtmlDocument.  The depth of the .NET classes that are available to WinRT developers is restricted, just restricted differently than Silverlight and Windows Phone.

Just how restricted is the WinRT profile?  Well, that remains to be seen at this point; but, from the looks of it it is minimally CLI compliant.  This means it doesn’t include things like the reflection namespaces.  WinRT is, after all, for producing touch-based applications that may run on very minimal processors and hardware.

One thing of note, to finish, is that the theme of WinRT is asynchronous.  Anything that could take a noticeable amount of time (“blocking”) is not available in the WinRT profile; so, get used to asynchronous programming.

Enterprise Service Buses, Brokers, Message Queues, Oh My!

Continuing on my theme of message-oriented, I thought I’d get into a bit of detail about the various middleware options and some details about these options.

Regardless of what type of middleware is chosen, one thing is common to all types of message-oriented middleware—the fact that they broker messages from producers to consumers.

Impetus

Before I get too far into the options, let’s look at a couple of the common reasons behind moving to a message-oriented architecture.

One impetus towards MOM is to integrate various legacy or third-party systems into a new or exiting enterprise system.  This type of MOM has been know as Enterprise Application Integration (or EAI).  There’s all sorts of information about this architectural style.

The other major impetus towards MOM is to facilitate distribution.

Technically EAI is also distributed; but that’s more because the individual components are completely independent of one another and thus executing on different computers.  The purpose of EAI isn’t to get distribution but to integrate distributed components.

Enterprise Application Integration

Enterprise Service Bus

The technology-du-jour in the MOM world seems to be Enterprise Service Buses (ESB). Enterprise Service Buses, by definition are are horizontally scalable MOM that integrate disparate (including legacy) loosely coupled SOA components using web services. Not only do ESBs technically broker communication between producers and consumers, they also deal with routing, transformation and translation. It’s hard for ESBs to do what they’re good at without that specific loose coupling you get from web services, where interface definition is discoverable and service-oriented (web services, over http, etc. etc.).

From a .NET standpoint there’s a few options out there calling themselves ESBs like NServiceBus and MassTransit.

Brokers

One form of EAI is a Broker.  “Broker” is considered an anti-pattern by more than a few people in the industry given what we now know because of it’s inherent hub-and-spoke shape.  A broker is a single component (hub) that brokers messages from producers to consumers (spokes).  It’s frowned upon because of its inherent scalability problems.  A single broker is constricted to a single computer and to gain scalability it’s forced to scale vertically so it has finite and expensive scalability options.  It’s unfortunate that this architecture is frowned upon because it’s still more than adequate when used where appropriate and technically a subset of all the other MOM options.

There isn’t technically a whole lot of options in NET for hub-and-spoke integration.  .NET became popular just as the many of the problems with hub-and-spoke were coming to light.  But, the closest thing to hub-and-spoke seems to be BizTalk.  It’s message box would technically be the hub and components integrated into BizTalk be spokes.  But BizTalk is a bit of hybrid of sorts (it supports translation, SOA, etc) so it’s a bit of a matter of opinion.

Distribution

Message Queues

Almost all EAI middleware uses some form of queue BizTalk: MSMQ, MassTransit: MSMQ, RabbitMQ, ActiveMQ, etc, NServiceBus MSMQ, Azure Queue, etc.

Other

Technically the other options are considered message queues; but they’re not message queues in and of themselves; they just operate like a message queue.  Like databases, RPC, web services, etc.

When to use which option

So now that I’ve confused the horizon with a bunch more terminology and a fair share of technobabble, why would you choose one over another?

The reason I divided the options in to two is because the choice on which to use should really be based on the impetus behind the composition of the system.

If you need to integrate legacy or disparate 3rd party services together, enterprise service buses are definitely the way to go.  If you’re dealing with integrating many components together with much routing and translation requirements, and a high number of messages, ESBs will help you immensely.  If your scalability requirements are very low, going the broker route is probably more than adequate.  The impetus here is the integration; but you don’t want to make a system completely unusable because it can’t scale.

But, if you have control over all the components in your system and your overall impetus is the distribution of work and horizontal scalability, then your focus is on performance not integration.  You want your endpoints to communicate in the most efficient manner and not be bogged down by unnecessary translations and routing. In which case you most likely want to avoid ESBs.

ESBs are kind of a Golden Hammer at the moment. ESB implies a SOA, web services, certain discoverability, lack of control of some or all components, etc.  If you *do* have control over all the components and don’t specifically require SOA, Web Services, and discoverability; implementing specifically to work with an ESB seems overkill.  If your have control over all components in your system and every message coming from a producer can be directly accepted by the consumer, you don’t need any of the translation aspects of ESBs.  You end up really only gaining in the routing aspect of ESB—which if you control all the components in your system, should be fairly static.

Message-Oriented Architecture is a Misleading Term

I recently blogged about Message-Oriented Architecture (and Message-Oriented Middleware, MOM).  The technology leads to extremely flexible and scalable applications; but it really distracts from the system.  It’s like calling a standard single-process application as Control-Unit Oriented Architecture.  Sure, the application is divided into instructions that eventually make their way to the control unit of the CPU; but that’s not only a implementation detail of the application; it’s an essential implementation detail. It’s pretty hard to write an application that doesn’t eventually distil down to instructions given to a control unit of a CPU.

And that’s kind of the problem with Message-Oriented Architecture.  Messaging facilitates distributed systems; but you can’t implement a distributed system without some sort of messaging.  MOM comes into play when you don’t want to have to implement your own layer or code to broker communications.  MOM is middleware that you acquire the distribute messages from producers to consumers.  You don’t have to use MOM to do messaging; but, you can’t do distributed without messaging—at least in some form.

What, you say?  If that’s the case then why are things like MOM and message-oriented architectures just now coming into focus?  Well, it’s true, MOM is a fairly recent technology on the software scene.  Mostly because we’ve been simply accepting Moore’s law as a way of scaling our applications.  Now that Moore’s law is slowing down, we have to scale our applications much more differently.  Technologies like Map/Reduce, cloud, etc. are mechanisms for horizontally scaling our systems rather than upgrading a computer to make our software go faster (vertical scaling).  Technologies like cloud and map/reduce give us ways to break up the work and execute it elastically on multiple computers; but it doesn’t give us a way for our individual components of our system to communicate between each other.

I think message-orientation, although not really new, has come to the forefront simply because it’s so inherent to designing and implementing distributed systems.

A distributed system is composed of many processes—it’s a composite system.  Writing distributed systems is the art of creating composable systems.

I’m going to continue with my series of posts relating to messaging between processes in a distributed system with a focus on message-oriented middleware; but, keep implementation details in mind.

Message-oriented Architectures (Updated)

I’ve had a few people ask me about message-oriented architectures and my knowledge and experience of it has evolved over time; so, I figured it would be a good idea to collect and publish some information about it.  It’s not one of those topics on the Internet that has gotten a lot of content, so I’ll try and change that.

A Message-Oriented Architecture is sometimes called Message-Oriented Middleware or MOM because there’s a middle tier or middleware that acts as the broker to relay messages from producers to consumers.  At the most succinct definition MOM is simply an architecture that supports sending and receiving messages.  That’s an extreme simplification and when implementing MOM many more facets of it’s design come into play, so I’m going to expand out into some of the common ways of implementing a message-oriented architecture over time.  So, I may go outside of MOM from a purists point of view.

This will be the first post in a series of posts so it will attempt to be very introductory.  To have a stable base on which to discuss this topic, let me first outline some terms relating to messaging.

Terminology

Producer

Something that produces a message. A message could be a command sent to a single command handler or an event published to zero or more event handlers.

Consumer

Something that consumes a message

Handler

Synonym for consumer

Message Broker

A component, often in an ESB, that transforms/translates and/or routes messages from one subsystem to another.

Queue

A storage mechanism for messages.  Depending in the implementation, a queue may be where messages are received from. Generally a separate process that receives messages through a known protocol like HTTP or TCP.  Generally used to implement load balancing semantics.

Aggregator

Something that consumes messages and contains them in some way and directs the message to another consumers.  Used, partially, to decouple producers from consumers.  A Queue is a form of aggregator.  Often lives within the a process.  Sometimes used to to merge pipelines.

Topic

Metadata associated with one or more messages that is generally used for publish and subscribe semantics.  Something publishes to or subscribes to topics rather than having to process all messages.

Exchange

In messaging systems that implement exchanges, exchanges are where messages are sent (compared to a queue, which is technically where messages are received from).

Service Bus

A integration platform that generally combines web services, messaging, transformations, and routing, to transactionally integrate many disparagedisparate applications or systems.  Often referred to an Enterprise Service Bus or ESB.

Pipeline

Often used in the context of messaging to describe a particular stream of messages or the fact that messages arrive or are processed one-by-one in a particular order.

Event

A message that informs subscribers of something that occurred in the past (albeit possibly not in the very distant past).

Command

A message that requests a change in state or data.

Saga

A series of related long-lived transactions that are executed as a non-atomic unit.

Idempotent

A operation that when repeated gives the same result.  This concept is used in mathematics as well; but becomes an important concept in messaging when we deal with things like guaranteed delivery.

Guaranteed delivery

Reliable messaging that guarantees a message is delivered to its intended recipients.  This often involves store-and-forward techniques to ensure message are not lost before delivery.

A least once delivery

A form of delivery reliability that involves concessions which may mean a message is delivered more than once.  See Idempotent.

Pub/Sub

AKA Publish/Subscribe.  A form of messaging where a publish publishes an event that is subscribed to by zero or more subscribers.  Publishers don’t can who, if anything, subscribes to events because an event is a representation of something that occurred in the past and requires no response.

Publisher

A producer that produces messages that are considered events.  A producer could produce events and commands.

Subscriber

A consumer that consumes message that are considered events.  There nothing stopping someone from implementing a consumer that subscribes to events and handlers commands; but they are generally kept separate, ala CQRS.

There’s all sorts of examples of messaging that have been around quite a while.  Sockets, HTTP, Web Services, WCF, etc are all examples of messaging techniques or technologies.  I’m going to discuss things at a slightly higher level where we deal with some of the application-level details of a message-oriented system.  i.e. things like Sockets, HTTP, Web Services, WCF, etc are implementation details—they may facilitate our messaging, but I’m not really going to get into that level of detail.

Messages are sent by producers asynchronously.  This means the producer is decoupled from consumers in every way except the fact that they share knowledge of one or more messages.  This also means that producers don’t know whether or not consumers have or will consume a message that it sends.  This sounds a bit scary at first; but it opens up a world of possibilities through this decoupling.  You have to account for the fact that producers don’t know about what consumers are up to.

Features

Very loosely coupled

Consumers and produces are separated by messages.  These messages are simply data.  They could be a binary serialized blob of data that a producer needs to be able to understand how to deserialize (often limiting producer/consumer to the same platform), or the message could be in the form of XML; where the consumer need only know how to process XML (not limiting consumer/producer to the same platform).

Platform agnostic

Messages can be sent and received on any platform.  Much in the same way an HTML page can be downloaded and viewed on any platform.  There are degrees to which a message-oriented system can support multiplatform (see above w.r.t. binary serialization); but the architecture itself imposes no real platform specifics.

Asynchronous

Consumers give messages to a broker, aggregator, or queue of some sort and that message is processed and delivered asynchronous.  The consumer is free to go about its business before effectively before the message has even left the broker.  Of course when consumer and producer and living in the same memory space using certain types of aggregators, this may not be the case; but typical message-oriented architectures use middleware that processes messages asynchronously.

Benefits

Flexibility is one of the important benefits of message-oriented architectures.  A message-oriented architecture is limited only by the coupling you impose.  If you some sort of middleware and don’t impose any requirements on how your message needs to be processes, you messages can be processed on any platform and at any point in time.  Need to publish a message from Windows and have it consumed in a Windows 7 Phone?  No problem.  Need that processing to occur almost instantly?  It’s entirely up to the consumer how quickly they need to consume a message.  Message doesn’t need to be processed right away?  Again, no problem the consumer can take as long as it needs to process the message.

The flexibility of the system results in many indirect benefits.  The producer doesn’t know anything about the consumer of a message (or how many consumers there are, if any) which means the message is free to be routed, aggregated and delivered completely independently of the producer.

One of the most important aspect, in my opinion, of message-oriented architectures is scalability.  By separating producers and consumers at the process level from one another by way of a message you introduce the ability to horizontally scale almost infinitely.  Because a producer asynchronously produces a message and one or more consumers may process the messages of a consumer, the broker is free to load balance or scale out to multiple consumers.  This is not without purposeful, and correct, design; but offers a system-level scalability the supports vertical and horizontal scaling.  For example, multiple consumers can be running on a computer to take advantage of each processor on that computer (vertical) or multiple consumers can be running on multiple computers to take advantage of any number or type of computers (horizontal) resulting in an architecture that supports near infinite scalability.

Drawbacks

One of the biggest drawbacks of a message-oriented architecture is the topology.  You’re generally looking at least at a publishing computer, a computer to house the messaging middleware, and another computer handle the consuming.  Each of these computers needs to be configured and publishers/consumers/middleware needs to be deployed and working correctly.  This can be cost prohibitive on smaller systems and implies much management.

If a message-oriented architecture is used in a situation it is not intended, there can be drawbacks to using it.  If a producer needs to synchronously produce a message (i.e, now when the consumer handles the message before it continues on with anything else), this can be a difficult scenario to implement.  If the producer needs to know about the consumer or consumers, this can also be a difficult scenario to implement.  Both of the scenarios can cause various headaches in a particular deployment.  I don’t really consider these drawbacks because MOM is not truly being used correctly; but the technology is not widely understood and the concepts it requires are often misunderstood so I mention it here because an implementation of MOM may encounter them early in the implementation.

Standards have come a long way recently, but—depending on your choice of implementation—there may be no real messaging standard.  This could mean you end up being tied to a particular vendor or even a particular platform with no means of extending to other platforms or other components using another messaging mechanism.

References

Wikipedia’s Message-oriented Middleware

CQRS