FxCop backlog tools: FxCop
If you're considering tackling an FxCop backlog, you're going to need a few tools. Obviously, the most important of these is FxCop itself, but that still leaves you with a potential choice to make between stand-alone FxCop and Visual Studio Static Analysis. If your target code base is built against .NET 1.1 or if you're not willing/able to shell out for Team Edition for Software Developers or Team Suite, the decision is pretty trivial. However, what if all your developers already have Visual Studio 2005 editions that include static analysis? You've paid for it, so your first instinct will probably be to use it. Unfortunately, it's probably not the best tool for this particular job.
While VStudio Static Analysis is great for dealing with a small volume of problems that should be addressed immediately, it kind of sucks the big wazoo with respect to handling the much larger volume of violations that one needs to deal with while tackling a backlog. This is partly a UI issue, and it shouldn't come as much as a surprise given that the backlog scenario probably wasn't exactly a major priority during the development of the VStudio integration piece. However, there are also some issues that go beyond the limited capabilities of the Visual Studio Error List window, and it's important to be aware of these as you make your tooling decisions.
In order to help understand why the stand-alone tool will likely be a better choice, let's take a closer look at some of the tasks in the backlog winnowing process...
You've decided to activate a given rule. Your code base has 400 existing violations of the rule, and you want your team to start fixing those right away. However, you certainly don't want to have to specifically assign each and every fix to a given developer. How do you create a "pool" of violations from which developers can draw?
If you're using Static Analysis and you also happen to be using Team Foundation Server, you might want create work items for the violations, then let developers assign the work items to themselves before starting to work on the fixes. However, from the VStudio Error List, you're going to able to generate work items in only one of two ways:
- Select all the violations, then generate a single work item, or
- Generate a work item for each violation individually.
Generating the individual work items would be ridiculously time consuming and tedious, so a single work item would usually be the only practical choice. However, in order to ensure that the more than one developer isn't working on any given violation at the same time, there's still going to have to be some manual work in terms of copying and editing work items before a developer will even start fixing a violation. Unfortunately, this procedural overhead doesn't really have much added value besides preventing overlapping work, and the additional effort may end up accounting for a very large proportion of the time spent on any given fix. Wouldn't you rather use that time for fixing other violations instead? (Of course, you could create a tool that automatically creates individual work items for you based on an FxCop output report, but that's again time that might be better spent doing other things.)
Another problem with adding backlog work items to TFS from VStudio Static Analysis results is that they're going to end up in the same TFS project as all your "real" work items for the same code base. Assuming your Static Analysis backlog is non-trivial, these backlog work items will start to swamp out your other feature and bug fix work items pretty quickly. Sure, you can isolate them in a category of their own, but sooner or later someone is going to include those beasties in a report where they shouldn't belong, and your stats are going to be skewed halway to heck.
OK, so if those are problems with using Static Analysis for violation reservation, what can the cheapskate's standalone edition do better? For starters, we can mark all the violations for fixing in one step without requiring later splitting. After activating a new rule and executing an analysis, simply select all the newly active violations of the rule and exclude them with a "TODO" note. If you're feeling fancy, you can even add a priority level (e.g.: "TODO 1" or "TODO 2"). After excluding the pre-existing violations, simply check the FxCop project into source control, and all the developers on your team can instantly access the items.
Under the "TODO" exclusion scheme, it's equally easy for any given developer to grab himself a chunk of new backlog work. He simply selects a bunch of violations from those that still have "TODO" notes, adds a new note (in one step) to claim all of them, then checks the modified FxCop project back into source control. What's this new note that gets added for the claiming? At FinRad, we simply add our initials to the note. For example, I claim violations by adding the note "TODO - NC" to an unclaimed TODO exclusion. If you've got a bigger team, you might need to identify developers by somewhat longer strings than their initials, but the underlying technique should still work.
Using stand-alone FxCop and TODO exclusions, I can both create and reserve violation "work items" in seconds, so I can spend more time on actually fixing violations rather than on managing the workload itself.
Don't break that
Identifying the current workload is only part of managing the violation backlog. Another important part is ensuring that the backlog violations that have not yet been fixed don't break your builds (assuming, of course, that you're using automated builds and that you've set FxCop rule violations to break those builds). If you're using VStudio Static Analysis, you've got two basic options for doing this:
- Set the rule to not break the build until after all the backlog violations are fixed or
- Use SuppressMessageAttribute to exclude the pre-existing violations.
If you don't allow violations of the newly activated rule to break the build, you won't start detecting new violations of the rule until after you've cleaned up the entire backlog, and that's not pretty. On the other hand, if you add a SuppressMessageAttribute to any given pre-existing violation, that violation will no longer show up in the Error List, so it's going to start being difficult for your team members to find and fix the problem.
Another problem with the SuppressMessageAttribute exclusions approach is that VStudio Static Analysis doesn't let you know when you don't need a suppression anymore.1 This means that if you fix the original violation but forget to remove the attribute, you could later create another violation of the same rule in the same code element, and you'll never notice it because the old placeholder exclusion will hide the new problem. That's a nasty enough problem even when you have a small number of "real" exclusions, but it's asking for big trouble when you consider the very large number of suppressions that you would need to create for your backlog violations.
This is another problem for which a much cleaner, simpler solution is possible when using the FxCop stand-alone UI. In fact, we've already covered the approach above. Since we create a "TODO" exclusion for every violation in the immediate backlog, we can keep the rule activated and build-breaking while still readily identifying the backlog violations. The stand-alone UI also allows us to detect exclusions that are no longer needed, so we can remove the unecessary exclusions in batches even if individual developers forget to remove them when fixing a violation.
On a bit of a side note, if you happen to be running a backlog cleanup of a .NET 2.0 application, you will probably want to store only your TODO exclusions in the FxCop project file. If you happen to resolve a violation by creating a "real" exclusion, that exclusion should probably be created with SuppressMessageAttribute rather than in the FxCop project file. After all, one of these days, you'll actually finish tackling the backlog, and you may want to switch over to using VStudio Static Analysis for finding and addressing the piddly few rule violations you'll be generating in your day-to-day work.
Sorting, filtering, and other fancy-pants advanced techniques
So let's say you're a developer who is ready to give an hour or so to fixing some of the backlog problems. How to you select which ones to work on? If you simply grab a random assortment, you'll probably end up working on violations of a variety of different rules that are widely separated with respect to location in the source code. You'll very likely end up wasting time on context switching that could have been put to much better use actually fixing violations.
As a more concrete example, let's say you decide that you have time to fix about 30 violations of the Performance.RemoveUnusedLocals rule. Trivial as these might seem, chances are that you'll probably find that somewhere between 10% and 20% of the violations actually require some decisions and maybe even a bit of code spelunking. This is why you won't try tackling 3 or 4 violations per minute, but it also has another consequence. If you're going to be encountering "weird" violations, wouldn't it be nice if they're mostly of the same style? One way of trying to increase the probably of this would be to draw the violations from a smallish area of the code base, where you'll at least stand a reasonable chance of encountering mostly code authored by a single developer.
While selecting violations using the above pattern is certainly possible from the VStudio Error List, it's not terribly convenient, and it's certainly not something that most of your developers are likely to enjoy doing several times a week. Developer patience with the selection process is likely to fray even faster if you're using TFS to store the backlog work items. To make things worse, you'll again be wasting time that could be better spent on actually fixing violations.
On the other hand, making the same issue selection in the stand-alone FxCop UI is trivial. First, I select the Performance.RemoveUnusedLocals rule in the rule treeview to hide violations of any other rule. Then I sort the "Excluded In Project" list by the item column. Since I can see availability of items in the "Last Note" column, all I have to do is select a bunch of violations that are clustered close together by item path in order to increase the chances of similar types of weirdness in the problems that I'm going to address.2
But I don't like the extra step involved in running FxCop...
Let's step back a moment and look at the usual reason for preferring VStudio Static Analysis: it runs seemlessly as part of the build process. Even ignoring the backlog problem, there's another nasty gotcha here: duration of a static analysis run. It takes over 3 minutes to analyze the assemblies in the product on which I work day-to-day. Given this, I'm certainly not going to run an analysis as part of my usual debug builds, whether it's via the integrated Static Analysis tool or by launching FxCop via a post-build event. If I really want the analysis to be associated with a build, I'm going to have to create an alternate build configuration at the very least.
This means that, even if I'm targeting the integrated tool, I'm still going to be launching analyses via a conscious, at least partially manual step. With one little extra step (clicking the "Analyze" button within the FxCop UI), I can run the analysis in the richer, otherwise more convenient stand-alone UI instead. To make this rapid launch possible, I simply add my FxCop project file to my solution as a solution item, then configure Visual Studio to open FxCop files using FxCop.exe. Once this initial setup is done, all I need to do to run my FxCop analysis is open the FxCop project from within Visual Studio then click the "Analyze" button once the FxCop UI has opened (to watch a 700K demo video, click here). That's amply convenient for me, although I guess your mileage may vary (but only if you're a complete nut <gdr>).
But that Static Analysis thingy cost me 5000$ (per seat) !!!
If you've bought one of the Team System editions that includes static analysis, I very much hope that you're taking full advantage of it for your new development work. I also hope that static analysis isn't the only "extra" feature that you're using, or perhaps you did spend rather a lot for one feature. ;) Either way, the fact that you happen to have the integrated static analysis tool available to you shouldn't be an important factor in making a decision regarding which FxCop tool to use for your backlog cleanup project.
On the other hand, the integrated tool ships with some rules that are not available in the free version of FxCop, and you might be swayed toward using that tool in order to take advantage of the additional rules, particularly given that FxCop 1.35 and Visual Studio 2005 Static Analysis cannot consume each other's rules. However, what you may not know is that FxCop 1.35 also shipped with some rules that aren't present in the integrated static analysis rule set3. The following table lists the rule set differences:
||Visual Studio 2005|
If you're seeking to maximize your rule coverage, jumping on static analysis because it supposedly contains more rules isn't exactly looking like an optimal approach... If you want to cover the combined rule set offered by the two tools, I'd recommend using FxCop 1.35 for your backlog cleanup until you've exhausted all of its rules. Once that's done, you would potentially be faced with a decision of switching over to the integrated tool for the remaining rules or coding custom rules for FxCop 1.35 that check for the same problems as the additional rules shipped with Visual Studio.
Then again, is that really a decision you're likely to end up facing? Assuming you start a backlog cleanup now, when do you think you're likely to be almost done? Unless your backlog is trivial, chances are excellent that both Orcas and FxCop vNext will have shipped by the time you're even half way done. Personally, I'm hoping that Orcas static analysis and FxCop vNext will be able to consume each other's rules, thereby making it possible to use the richer UI when addressing a backlog of the Visual Studio rules, as well as easing some of the pain around custom rule authoring. There's no technical reason why they shouldn't be able to consume the same rules, but that doesn't mean that a difference couldn't be introduced explicitly in order to prevent using the expensive rules from the free tool. I'm keeping my fingers crossed that this won't happen, although I won't start counting any chickens until Orcas and FxCop vNext have both RTMed...
1If you wish VStudio Static Analysis and/or FxCopCmd.exe would let you know when you've got unecessary suppressions, vote here.
2If FxCop wouldn't "lose" the rule selection when switching between the rules and targets treeviews, it would be possible to make an even more convenient selection by "climbing" the target treeview from the target associated with any given violations. Unfortunately, the rule selection is lost when switching to the other treeview tab in FxCop 1.35. I'm keeping my fingers crossed that this won't be the case in the vNext UI, but there seems little point in creating a bug report until I've actually tried the beastie. I'm also hoping that the vNext UI will allowing filtering by column values within the violation lists, which would also make working with large backlog lists more convenient, but I won't be holding my breath for that change.
3In case you're not already aware of this, the Visual Studio rule set is supposed to be a superset of the stand-alone FxCop rule set. FxCop 1.35 only contains rules that aren't present in Visual Studio 2005 because of time constraints during the preparation of Visual Studio 2005. (Sorry, but the best reference I seem to be able to find regarding this at the moment is a post on the FxCop forum.) The extra FxCop 1.35 rules are present in Orcas, with the exception of Usage.LiteralsShouldBeSpelledCorrectly, which is a control flow rule.