Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

I am no more

I have moved my blog to http://omaralzabir.com and I am no longer blogging here. Please update your readers and subscribe to the feed on the new site.

C# with keyword equivalent

There’s no with keyword in C#, like Visual Basic. So you end up writing code like this:

this.StatusProgressBar.IsIndeterminate = false;
this.StatusProgressBar.Visibility = Visibility.Visible;
this.StatusProgressBar.Minimum = 0;
this.StatusProgressBar.Maximum = 100;
this.StatusProgressBar.Value = percentage;

Here’s a work around to this:

this.StatusProgressBar.Use(p =>
{
  p.IsIndeterminate = false;
  p.Visibility = Visibility.Visible;
  p.Minimum = 0;
  p.Maximum = 100;
  p.Value = percentage;
});

Saves you repeatedly typing the same class instance or control name over and over again. It also makes code more readable since it clearly says that you are working with a progress bar control within the block. It you are setting properties of several controls one after another, it’s easier to read such code this way since you will have dedicated block for each control.

It’s a very simple one line function that does it:

public static void Use<T>(this T item, Action<T> work)
{
    work(item);
}

You could argue that you can just do this:

var p = this.StatusProgressBar;
p.IsIndeterminate = false;
p.Visibility = Visibility.Visible;
p.Minimum = 0;
p.Maximum = 100;
p.Value = percentage;

But it’s not elegant. You are introducing a variable “p” in the local scope of the whole function. This goes against naming conventions. Morever, you can’t limit the scope of “p” within a certain place in the function.

Update: Previously I proposed a way to do it without generic extention method which was not so clean. Andy T posted this cleaner solution in comments.

ParallelWork: Feature rich multithreaded fluent task execution library for WPF

ParallelWork is an open source free helper class that lets you run multiple work in parallel threads, get success, failure and progress update on the WPF UI thread, wait for work to complete, abort all work (in case of shutdown), queue work to run after certain time, chain parallel work one after another. It’s more convenient than using .NET’s BackgroundWorker because you don’t have to declare one component per work, nor do you need to declare event handlers to receive notification and carry additional data through private variables. You can safely pass objects produced from different thread to the success callback. Moreover, you can wait for work to complete before you do certain operation and you can abort all parallel work while they are in-flight. If you are building highly responsive WPF UI where you have to carry out multiple job in parallel yet want full control over those parallel jobs completion and cancellation, then the ParallelWork library is the right solution for you.

I am using the ParallelWork library in my PlantUmlEditor project, which is a free open source UML editor built on WPF. You can see some realistic use of the ParallelWork library there. Moreover, the test project comes with 400 lines of Behavior Driven Development flavored tests, that confirms it really does what it says it does.

The source code of the library is part of the “Utilities” project in PlantUmlEditor source code hosted at Google Code.

The library comes in two flavors, one is the ParallelWork static class, which has a collection of static methods that you can call. Another is the Start class, which is a fluent wrapper over the ParallelWork class to make it more readable and aesthetically pleasing code.

ParallelWork allows you to start work immediately on separate thread or you can queue a work to start after some duration. You can start an immediate work in a new thread using the following methods:

• void StartNow(Action doWork, Action onComplete)
• void StartNow(Action doWork, Action onComplete, Action<Exception> failed)

For example,

ParallelWork.StartNow(() =>
{
    workStartedAt = DateTime.Now;
    Thread.Sleep(howLongWorkTakes);
},
() =>
{
    workEndedAt = DateTime.Now; 
});

Or you can use the fluent way Start.Work:

Start.Work(() =>
    {
        workStartedAt = DateTime.Now;
        Thread.Sleep(howLongWorkTakes);
    })
    .OnComplete(() =>
    {
        workCompletedAt = DateTime.Now;
    })
    .Run();

Besides simple execution of work on a parallel thread, you can have the parallel thread produce some object and then pass it to the success callback by using these overloads:

• void StartNow<T>(Func<T> doWork, Action<T> onComplete)
• void StartNow<T>(Func<T> doWork, Action<T> onComplete, Action<Exception> fail)

For example,

ParallelWork.StartNow<Dictionary<string, string>>(
    () =>
    {                        
        test = new Dictionary<string,string>();
        test.Add("test", "test");

        return test;
    },
    (result) =>
    {
        Assert.True(result.ContainsKey("test"));
    });

Or, the fluent way:

Start<Dictionary<string, string>>.Work(() =>
    {
        test = new Dictionary<string, string>();
        test.Add("test", "test");

        return test;
    })
    .OnComplete((result) =>
    {
        Assert.True(result.ContainsKey("test"));
    })
    .Run();

You can also start a work to happen after some time using these methods:

• DispatcherTimer StartAfter(Action onComplete, TimeSpan duration)
• DispatcherTimer StartAfter(Action doWork,Action onComplete,TimeSpan duration)

You can use this to perform some timed operation on the UI thread, as well as perform some operation in separate thread after some time.

ParallelWork.StartAfter(
    () =>
    {
        workStartedAt = DateTime.Now;
        Thread.Sleep(howLongWorkTakes);
    },
    () =>
    {
        workCompletedAt = DateTime.Now;
    },
    waitDuration);

Or, the fluent way:

Start.Work(() =>
    {
        workStartedAt = DateTime.Now;
        Thread.Sleep(howLongWorkTakes);
    })
    .OnComplete(() =>
    {
        workCompletedAt = DateTime.Now;
    })
    .RunAfter(waitDuration);

There are several overloads of these functions to have a exception callback for handling exceptions or get progress update from background thread while work is in progress. For example, I use it in my PlantUmlEditor to perform background update of the application.

// Check if there's a newer version of the app
Start<bool>.Work(() => 
{
    return UpdateChecker.HasUpdate(Settings.Default.DownloadUrl);
})
.OnComplete((hasUpdate) =>
{
    if (hasUpdate)
    {
        if (MessageBox.Show(Window.GetWindow(me),
            "There's a newer version available. 
Do you want to download and install?",
            "New version available",
            MessageBoxButton.YesNo,
            MessageBoxImage.Information) == MessageBoxResult.Yes)
        {
            ParallelWork.StartNow(() => {
                var tempPath = System.IO.Path.Combine(
                    Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),
                    Settings.Default.SetupExeName);

                UpdateChecker.DownloadLatestUpdate(Settings.Default.DownloadUrl, tempPath);
            }, () => { },
                (x) =>
                {
                    MessageBox.Show(Window.GetWindow(me),
                        "Download failed. When you run next time, 
it will try downloading again.",
                        "Download failed",
                        MessageBoxButton.OK,
                        MessageBoxImage.Warning);
                });
        }
    }
})
.OnException((x) => 
{
    MessageBox.Show(Window.GetWindow(me),
        x.Message,
        "Download failed",
        MessageBoxButton.OK,
        MessageBoxImage.Exclamation);
});

The above code shows you how to get exception callbacks on the UI thread so that you can take necessary actions on the UI. Moreover, it shows how you can chain two parallel works to happen one after another.

Sometimes you want to do some parallel work when user does some activity on the UI. For example, you might want to save file in an editor while user is typing every 10 second. In such case, you need to make sure you don’t start another parallel work every 10 seconds while a work is already queued. You need to make sure you start a new work only when there’s no other background work going on. Here’s how you can do it:

private void ContentEditor_TextChanged(object sender, EventArgs e)
{
  if (!ParallelWork.IsAnyWorkRunning())
  {
     ParallelWork.StartAfter(SaveAndRefreshDiagram, 
                                 TimeSpan.FromSeconds(10));
  }
}

If you want to shutdown your application and want to make sure no parallel work is going on, then you can call the StopAll() method.

ParallelWork.StopAll();

If you want to wait for parallel works to complete without a timeout, then you can call the WaitForAllWork(TimeSpan timeout). It will block the current thread until the all parallel work completes or the timeout period elapses.

result = ParallelWork.WaitForAllWork(TimeSpan.FromSeconds(1));

The result is true, if all parallel work completed. If it’s false, then the timeout period elapsed and all parallel work did not complete.

For details how this library is built and how it works, please read the following codeproject article:

ParallelWork: Feature rich multithreaded fluent task execution library for WPF

http://www.codeproject.com/KB/WPF/parallelwork.aspx

If you like the article, please vote for me.

Open Source WPF UML Design tool

PlantUmlEditor is my new free open source UML designer project built using WPF and .NET 3.5. If you have used plantuml before, you know that you can quickly create sophisitcated UML diagrams without struggling with a designer. Especially those who use Visio to draw UML diagrams (God forbid!), you will be at heaven. This is a super fast way to get your diagrams up and ready for show. You can *write* UML diagrams in plain English, following a simple syntax and get diagrams generated on-the-fly.

This editor really saves time designing UML diagrams. I have to produce quick diagrams to convey ideas quickly to Architects, Designers and Developers everyday. So, I use this tool to write some quick diagrams at the speed of coding, and the diagrams get generated on the fly. Instead of writing a long mail explaining some complex operation or some business process in English, I can quickly write it in the editor in almost plain English, and get a nice looking activity/sequence diagram generated instantly. Making major changes is also as easy as doing search-replace and copy-pasting blocks here and there. You don't get such agility in any conventional mouse-based UML designers.

I have submited a full codeproject article to give you a detail walkthrough how I have built this. Please read this article and vote for me if you like it.

PlantUML Editor: A fast and simple UML editor using WPF

http://www.codeproject.com/KB/smart/plantumleditor.aspx

You can download the project from here:

http://code.google.com/p/plantumleditor/

Do Unit Test and Integration Test from same test code using Conditional Compilation

You usually write unit test and integration test code separately using different technologies. For example, for unit test, you use some mocking framework like Moq to do the mocking. For integration test, you do not use any mocking, just some test classes that hits some service or facade to do end-to-end integration test. However, sometimes you see that the integration and unit test are more or less same, they test the same class using its interface and perform the same tests against the same expectation. For example, if you think about a WCF service, you write unit test to test the ServiceContract using the interface where you use some mocking framework to mock the interface of the WCF Service. If you look at the following example, I am using Moq to test IPortalService interface which is a ServiceContract for a WCF service. I am using xUnit and SubSpec to do BDD style tests.

[Specification]
public void GetAllWidgetDefinitions_should_return_all_widget_in_widget_gallery()
{
    var portalServiceMock = new Mock<IPortalService>();
    var portalService = portalServiceMock.Object;

    "Given a already populated widget gallery".Context(() =>
    {
        portalServiceMock.Setup(p => p.GetAllWidgetDefinitions())
            .Returns(new Widget[] { new Widget { ID = 1 }, new Widget { ID = 2 }})
            .Verifiable();
    });

    Widget[] widgets = default(Widget[]);
    "When a widget is added to one of the page".Do(() =>
    {
        widgets = portalService.GetAllWidgetDefinitions();
    });

    "It should create the widget on the first row and first 
column on the same page".Assert(() =>
    {
        portalServiceMock.VerifyAll();
        Assert.NotEqual(0, widgets.Length);
        Assert.NotEqual(0, widgets[0].ID);
    });
}

Now when I want to do an end-to-end test to see if the service really works by connecting all the wires, then I write a test like this:

[Specification]
public void GetAllWidgetDefinitions_should_return_all_widget_in_widget_gallery()
{
    var portalService = new ManageCustomerPortalClient();

    "Given a already populated widget gallery".Context(() =>
    {
    });

    Widget[] widgets = default(Widget[]);
    "When a widget is added to one of the page".Do(() =>
    {
        widgets = portalService.GetAllWidgetDefinitions();
    });

    "It should create the widget on the first row and 
first column on the same page".Assert(() =>
    {
        Assert.NotEqual(0, widgets.Length);
        Assert.NotEqual(0, widgets[0].ID);
    });
}

If you look at the difference, it’s very little. The mockings are gone. The same operation is called using the same parameters. The same Asserts are done to test against the same expectation. It’s an awful duplication of code.

Conditional compilation saves the day. You could write the unit test using some conditional compilation directive so that in real environment, those mockings are gone and the real stuff gets run. For example, the following code does both unit test and integration test for me. All I do is turn on/off some conditional compilation.

[Specification]
public void GetAllWidgetDefinitions_should_return_all_widget_in_widget_gallery()
{
#if MOCK
    var portalServiceMock = new Mock<IPortalService>();
    var portalService = portalServiceMock.Object;
#else
    var portalService = new ManageCustomerPortalClient();
#endif

    "Given a already populated widget gallery".Context(() =>
    {
#if MOCK
        portalServiceMock.Setup(p => p.GetAllWidgetDefinitions())
            .Returns(new Widget[] { new Widget { ID = 1 }, new Widget { ID = 2 }})
            .Verifiable();
#endif
    });

    Widget[] widgets = default(Widget[]);
    "When a widget is added to one of the page".Do(() =>
    {
        widgets = portalService.GetAllWidgetDefinitions();
    });

    "It should create the widget on the first row and 
first column on the same page".Assert(() =>
    {
#if MOCK
        portalServiceMock.VerifyAll();
#endif
        Assert.NotEqual(0, widgets.Length);
        Assert.NotEqual(0, widgets[0].ID);
    });
}

The code is now in unit test mode. When I run this, it performs unit test using Moq. When I want to switch to integration test mode, all I do is take out the “MOCK” word from Project Properties->Build->Conditional Compilation.

Hope this gives you ideas to save unit test and integration test coding time.

Is your computer running slow, battery running out quickly?

If your computer is running hot or battery running out quickly then it is most likely due to some application or process consuming high CPU or memory. If you keep running applications for a long time, for example, Outlook, then it continues to grow in memory consumption and does not free up memory efficiently. As a result, your computer runs out of physical memory and other applications run slower. Sometimes Outlook, browser, image editing applications or some other application start taking full CPU as they get into some heavy internal processing and make your CPU hot and other applications perform slower.

My new CPUAlert is an application that monitors CPU and memory consumption of applications and alerts you if some application is consistently taking high CPU or high memory. It not only saves your CPU and Battery’s lifetime but also makes your computer run smooth and let your active applications run as fast as they can be.

While it is running, if some process is consuming more than 200 MB memory, it will show you an alert:

Here you can see my Outlook is taking 244 MB of physical RAM.

You can either postpone the alert for 5 mins (just press ESC), or ignore the process permanently so that you no longer receive alert for the process anymore, or you can close it and reclaim memory.

The handy feature is “Restart” which closes the application and starts again. This generally frees up memory that clogs up in the process.

Same alert will come if some process is consuming more than 30% CPU for over 5 mins.

You can configure all these settings like what’s the tolerable limit for CPU and memory, how frequently to show alert, how long to wait before closing application etc by right clicking on the Task bar icon and choosing Settings.

Source code of the project is available at:

http://code.google.com/p/cpualert/

The installer can also be downloaded from there.

Warning: The code is not in a good shape. I was frustrated at some process taking high CPU and memory and I wrote this app within hours to get the job done for me.

If you like the application, spread the word!

Fast Streaming Ajax Proxy with GET PUT POST DELETE

I have enhanced my streaming Ajax Proxy with POST, PUT and DELETE features. Previously it supported only GET. Now it supports all 4 popular methods for complete REST support. Using this proxy, you can call REST API on external domain directly from your website’s javascript code. You can test the proxy from this link:

labs.omaralzabir.com/ajaxstreamingproxy/GetPutDeleteTest.aspx

The latest source code for the Ajax Proxy is available here:

http://code.google.com/p/fastajaxproxy/

You can find a detail CodeProject article that explains how the streaming asynchronous aspect of this proxy works:

Fast, Scalable, Streaming AJAX Proxy - continuously deliver data from across domains

Here’s how the test UI looks like where you can test POST, PUT and DELETE:

If you want to run the sample source code on your local IIS, make sure you allow the POST, PUT, and DELETE headers on .ashx extension from IIS properties:

The sample project shows how you can use the proxy to make calls to external domains. You can directly hit any external URL and perform POST or DELETE from your javascript code:

var proxyUrl = "StreamingProxy.ashx";
function download(method, proxyUrl, contentUrl, isJson, bodyContent, completeCallback) {
    var request = new Sys.Net.WebRequest();
    
    if (method == "POST" || method == "PUT")
        request.set_httpVerb("POST");
    else
        request.set_httpVerb("GET");

    var url = proxyUrl + "?m=" + method + 
  (isJson ? "&t=" + escape("application/json") : "") + "&u=" + escape(contentUrl);

    request.set_url(url);

    if (bodyContent.length > 0) {
        request.set_body(bodyContent);
        request.get_headers()["Content-Length"] = bodyContent.length;
    }

    var startTime = new Date().getTime();

    request.add_completed(function(executor) {
        if (executor.get_responseAvailable()) {
            var content = executor.get_responseData();
            var endTime = new Date().getTime();

            var statistics =
            "Duration: " + (endTime - startTime) + "ms" + '\n' +
            "Length: " + content.length + " bytes" + '\n' +
            "Status Code: " + executor.get_statusCode() + " " + '\n' +
            "Status: [" + executor.get_statusText() + "]" + '\n';
            appendStat(statistics);

            $get('resultContent').value = content;
            completeCallback();
        }
    });

    var executor = new Sys.Net.XMLHttpExecutor();
    request.set_executor(executor);
    executor.executeRequest();
}

I am using MS AJAX here. You can use jQuery to perform the same test as well. All you need to do is hit the URL of the StreamingProxy.ashx and pass the actual URL in query string parameter “u” and pass the type of the http method in query string parameter “m”. That’s it!

Windows 7 64bit, Outlook 2010 64bit, Conferencing Addin 64bit, Macbook Pro 64bit

I am a 64bit freak. I got Windows 7, Outlook 2010, Conferencing Addin all 64 bit versions to work on a Macbook pro. Those who are thinking about moving to 64bit and hesitating whether something will break, GO AHEAD! Macbook Pro hardware and Microsoft’s software are the best combination out there. You will enjoy every moment you spend with your laptop. Moreover, I have tried these combinations on HP tablet PC, Sony VAIO, Dell Inspiron and Dell Vostro. HP works best. Others are struggling with driver issues.

I will give you positive and negative feedback with the apps I have tried so far:

Outlook 2010 64 bit:

Here are my negative feedback. Outlook Product Manager, please read this. I am a hardcore Outlook customer of you.

• All my Outlook COM addins are dead. Outlook 2010 64bit does not support them. Looks like not so good backward compatibility.
• Not so significant improvement with Exchange 2007. The startup time has improved from about 5 secs to 2 secs. But the startup time saving is not really a big saver since I start outlook and it keeps running for days until my PC is so screwed that I need a restart.
• Office Communicator 2005 does not work.
• The beta Office 2010 applications are CPU hungry. I see 30% to 40% CPU most of the time.
• It took me over 30 hours until Outlook 2010 started to perform well. All this time, it was indexing and indexing and indexing and burning CPU.
• There’s nothing so ground breaking and productivity enhancing in Outlook 2010 yet. After upgrading and using it for couple of days, I don’t see something so attractive that justifies the time spent in upgrading for busy professional. It’s not upgrade at this stage so far. You have to uninstall all Office 2007 or earlier products, addins etc and then install Outlook 2010.
• Outlook Keyboard shortcuts are changed, having hard time adjusting. My precious Alt+L for Reply to All is gone. Now it’s Ctrl+Shift+R. Come on guys, when do you just Reply and not to Reply to All? I barely remember ever using Reply only. It’s always Reply to All. Can’t you make a easier shortcut for this?
• Keyboard focus gets lost to some weird place sometimes and my navigation using cursor gets broken. I have to click using mouse to get into track.
• Quick Tasks are kind of limited. For ex, “Reply & Delete”, who would want to press CTRL+SHIFT+1 to do reply and delete? It’s more natural to press Ctrl+R to reply and then send it and hit DEL. The choices on Quick Tasks are limited as well. I was hoping I would be able to chain multiple commands like – open a new message window, select a specific account to send mail using, select a specific signature and after the mail is sent, show move dialog box to move the conversation to a specific folder. Nope, it does not work this way. First of all there are limited commands which does not even support this. Secondly, all the actions are performed instantly one after another without waiting for the first action to complete.
• Quick Steps cannot be added to Quick Access Toolbar. Go figure!

Now the good things:

• Overall Outlook experience is smooth. Opening new mail, typing address, doing search, moving messages, viewing a folder on conversation view mode are all significantly faster, even with Exchange. It’s hard to say if it’s due to fully 64 bit environment or due to the fact that none of my COM addins are working.
• Outlook exits. Finally! None of the previous Outlook would terminate the process if I exited Outlook. It remains in memory forever unless I kill it from task manager. Now the Outlook really closes, or at least kills itself when I exit. Whenever I exit Outlook and start again, I see it doing some Data Integrity check. This means it is not really closing itself properly, but killing itself. I assume that’s bad and my data in Outlook are slowly getting messed up.
• The conversation view is great!
• Inline appointment viewer is a life saver. When I get an appointment invite, the email preview shows a small view of the calendar around the meeting time. I can see if I am occupied or if there’s an available time before or after the meeting. This saves me a lot of time everyday as rescheduling meeting is a tedious job in my company and it takes around 4 to 7 reschedules attempts to get a suitable time slot in everyone's diary for every darn meeting.
• Quick Steps is more or less useful. I am getting used to using CTRL+SHIFT+1 to “reply to all and delete” and CTRL+SHIFT+2 for “reply to all and move to folder”. You just have to configure the quick steps to make it suit you. Previously I used to use QuickFile addin, which was a super useful tool, worth paying $39.95.

Onenote 2010 64 bit

The UI is certainly much slicker. It really looks and feels like a notebook now. Sketching performance is improved.

However, a big bug. I was sketching and suddenly my pointer switched to selection mode from pen. All pen options are disabled. I tried exiting and coming back. Nope. Can’t go back to pen mode at all. I am using a Genius Tablet. Looks like Onenote is Tablet PC friendly only. Hope Apple makes a Tablet Macbook Pro soon.

Word 2010 64 bit

Haven’t used it much. Ribbons are as confusing as before. The File menu is even more confusing now. No new shape styles that makes word documents stand out from the rest. No new Smart Art worth mentioning. Overall – disappointing.

The print features are much improved!

Powerpoint 2010 64 bit

I did not notice any significant new feature in Powerpoint, sadly. The ribbon has been made more useful than before. There’s a “Transition” and “Animations” ribbon bar which is very useful to use and saves time putting animations in slides. But that’s all I could see from my limited trial. This is disappointing. I was expecting there would be richer collection of shapes which are really cool to look and makes presentations look like Web 2.0 sites, a lot of new Smart Arts, but nothing.

Visio 2010 64 bit

The UML Diagram designer is as crappy as ever. Come on Microsoft, watch the other UML designers and learn from them. Currently Visio is my last choice for UML design and makes my job life unhappy because my company forces me to use it. I use PlantUML wherever I can.

I don’t see any new amazingly cool diagram either. I was hoping the Detailed Network Diagram stencil would be much improved with smooth round glossy servers, amazingly cool looking router icons etc. But no luck. The new ribbon interface is as confusing as other Office applications.

Conclusion

So far I can see significant improvement in Outlook only. Other apps do not have anything that stands out.

Unit Testing and Integration Testing in real projects

I am yet to find a proper sample on how to do realistic Test Driven Development (TDD) and how to write proper unit tests for complex business applications, that gives you enough confidence to stop doing manual tests anymore. Generally the samples show you how to test a Stack or a LinkedList, which is far simpler than testing a typical N-tier application, especially if you are using Entity Framework or Linq to SQL or some ORM in data access layer, and doing logging, validation, caching, error handling at middle tier. There are many articles, blog posts, video tutorials on how to write unit tests, which I believe are all very good starting points. But all these examples show you basic tests, not good enough to let your QA team go. So, let me try to show you some realistic unit and integration test examples which should help you write tests that gives you confidence and helps you gradually move towards TDD.  

I will show you tests done on my open source project Dropthings, which is a Web 2.0 AJAX portal built using jQuery, ASP.NET 3.5, Linq to SQL, Dependency Injection using Unity, caching using Microsoft Enterprise Library, Velocity and so on. Basically all the hot techs you can grasp in one shot. The project is a typical N-tier application where there’s a web layer, a business layer and a data access layer. Writing unit tests, integration tests and load tests for this project was challenging, and thus interesting to share so that you can see how you can implement Unit Testing and Integration Testing in a real project and gradually get into Test Driven Development.

Read this codeproject article of mine to learn how I did Integration Tests and Unit Tests using Behavior Driven Development approach:

Unit Testing and Integration Testing in business applications

http://www.codeproject.com/KB/testing/realtesting.aspx

If you like it, please vote for me.

 

Simple way to cache objects and collections for greater performance and scalability

Caching of frequently used data greatly increases the scalability of your application since you can avoid repeated queries on database, file system or to webservices. When objects are cached, it can be retrieved from the cache which is lot faster and more scalable than loading from database, file or web service. However, implementing caching is tricky and monotonous when you have to do it for many classes. Your data access layer gets a whole lot of code that deals with caching objects and collection, updating cache when objects change or get deleted, expire collections when a contained object changes or gets deleted and so on. The more code you write, the more maintenance overhead you add. Here I will show you how you can make the caching a lot easier using Linq to SQL and my library AspectF. It’s a library that helps you get rid of thousands of lines of repeated code from a medium sized project and eliminates plumbing (logging, error handling, retrying etc) type code completely.

Here’s an example how caching significantly improves the performance and scalabitlity of applications. Dropthings – my open source Web 2.0 AJAX portal, without caching can only serve about 11 request/sec with 10 concurrent users on a dual core 64 bit PC. Here data is loaded from database as well as from external sources. Avg page response time is 1.44 sec.

After implementing caching, it became significantly faster, around 32 requests/sec. Page load time decreased significantly as well to 0.41 sec only. During the load test, CPU utilization was around 60%.

It shows clearly the significant difference it can make to your application. If you are suffering from poor page load performance and high CPU or disk activity on your database and application server, then caching Top 5 most frequently used objects in your application will solve that problem right away. It’s a quick win to make your application a lot faster than doing complex re-engineering in your application.

Common approaches to caching objects and collections

Sometimes the caching can be simple, for example caching a single object which does not belong to a collection and does not have child collections that are cached separately. In such case, you write simple code like this:

• Is the object being requested already in cache?
  • Yes, then serve it from cache.
  • No, then load it from database and then cache it.

On the other hand, when you are dealing with cached collection where each item in the collection is also cached separately, then the caching logic is not so simple. For example, say you have cached a User collection. But each User object is also cached separately because you need to load individual User objects frequently. Then the caching logic gets more complicated:

• Is the collection being requested already in cache?
  • Yes. Get the collection. For each object in the collection:
    • Is that object individually available in cache?
      • Yes, get the individual object from cache. Update it in the collection.
      • No, discard the whole collection from cache. Go to next step:
  • No. Load the collection from source (eg database) and cache each item in the collection separately. Then cache the collection.

You might be thinking why do we need to read each individual item from cache and why do we need to cache each item in collection separarely when the whole collection is already in cache? There are two scenarios you need to address when you cache a collection and individual items in that collection are also cached separately:

• An individual item has been updated and the updated item is in cache. But the collection, which contains all those individual items, has not been refreshed. So, if you get the collection from cache and return as it is, you will get stale individual items inside that collection. This is why each item needs to be retrieved from cache separately.
• An item in the collection may have been force expired in cache. For ex, something changed in the object or the object has been deleted. So, you expired it in cache so that on next retrieval it comes from database. If you load the collection from cache only, then the collection will contain the stale object.

If you are doing it the conventional way, you will be writing a lot of repeated code in your data access layer. For example, say you are loading a Page collection that belongs to a user. If you want to cache the collection of Page for a user as well as cache individual Page objects so that each Page can be retrieved from Cache directly. Then you need to write code like this:

public List<Page> GetPagesOfUserOldSchool(Guid userGuid)
{
    ICache cache = Services.Get<ICache>();
    bool isCacheStale = false;
    string cacheKey = CacheSetup.CacheKeys.PagesOfUser(userGuid);
    var cachedPages = cache.Get(cacheKey) as List<Page>;
    if (cachedPages != null)
    {
        var resultantPages = new List<Page>();
        // If each item in the collection is no longer in cache, invalidate the collection
        // and load again.
        foreach (Page cachedPage in cachedPages)
        {
            var individualPageInCache = cache.Get(CacheSetup.CacheKeys.PageId(cachedPage.ID)) as Page;
            if (null == individualPageInCache)
            {
                // Some item is missing in cache. So, the collection is stale. 
                isCacheStale = true;
            }
            else
            {
                resultantPages.Add(individualPageInCache);
            }
        }

        cachedPages = resultantPages;
    }

    if (isCacheStale)
    {
        // Collection not cached. Need to load collection from database and then cache it.
        var pagesOfUser = _database.GetList<Page, Guid>(...);
        pagesOfUser.Each(page =>
        {
            page.Detach();
            cache.Add(CacheSetup.CacheKeys.PageId(page.ID), page);
        });
        cache.Add(cacheKey, pagesOfUser);
        return pagesOfUser;
    }
    else
    {
        return cachedPages;
    }
}

Imagine writing this kind of code over and over again for each and every entity that you want to cache. This becomes a maintenace nightmare as your project grows.

Here’s how you could do it using AspectF:

public List<Page> GetPagesOfUser(Guid userGuid)
{
    return AspectF.Define
        .CacheList<Page, List<Page>>(Services.Get<ICache>(), 
                     CacheSetup.CacheKeys.PagesOfUser(userGuid), 
                     page => CacheSetup.CacheKeys.PageId(page.ID))
        .Return<List<Page>>(() => 
            _database.GetList<Page, Guid>(...).Select(p => p.Detach()).ToList());
}

Instead of 42 lines of code, you can do it in 5 lines!

Read my article Simple way to cache objects and collections for greater performance and scalability on CodeProject and learn:

• Caching Linq to SQL entities
• Handling update and delete scenarios
• Expiring dependent objects and collections in cache
• Handling objects that’s cached with multiple keys
• Avoid database query optimizations when you cache sets of data

Enjoy. Don’t forget to vote for me!

  

7 tips for for loading Javascript rich Web 2.0-like sites significantly faster

Introduction

When you create rich Ajax application, you use external JavaScript frameworks and you have your own homemade code that drives your application. The problem with well known JavaScript framework is, they offer rich set of features which are not always necessary in its entirety. You may end up using only 30% of jQuery but you still download the full jQuery framework. So, you are downloading 70% unnecessary scripts. Similarly, you might have written your own javascripts which are not always used. There might be features which are not used when the site loads for the first time, resulting in unnecessary download during initial load. Initial loading time is crucial – it can make or break your website. We did some analysis and found that every 500ms we added to initial loading, we lost approx 30% traffic who never wait for the whole page to load and just close browser or go away. So, saving initial loading time, even by couple of hundred milliseconds, is crucial for survival of a startup, especially if it’s a Rich AJAX website.

You must have noticed Microsoft’s new tool Doloto which helps solve the following problem:

Modern Web 2.0 applications, such as GMail, Live Maps, Facebook and many others, use a combination of Dynamic HTML, JavaScript and other Web browser technologies commonly referred as AJAX to push page generation and content manipulation to the client web browser. This improves the responsiveness of these network-bound applications, but the shift of application execution from a back-end server to the client also often dramatically increases the amount of code that must first be downloaded to the browser. This creates an unfortunate Catch-22: to create responsive distributed Web 2.0 applications developers move code to the client, but for an application to be responsive, the code must first be transferred there, which takes time.

Microsoft Research looked at this problem and published this research paper in 2008, where they showed how much improvement can be achieved on initial loading if there was a way to split the javascripts frameworks into two parts – one primary part which is absolutely essential for initial rendering of the page and one auxiliary part which is not essential for initial load and can be downloaded later or on-demand when user does some action. They looked at my earlier startup Pageflakes and reported:

2.2.2 Dynamic Loading: Pageflakes
A contrast to Bunny Hunt is the Pageflakes application, an
industrial-strength mashup page providing portal-like functionality.
While the download size for Pageflakes is over 1 MB, its initial
execution time appears to be quite fast. Examining network activity
reveals that Pageflakes downloads only a small stub of code
with the initial page, and loads the rest of its code dynamically in
the background. As illustrated by Pageflakes, developers today can
use dynamic code loading to improve their web application’s performance.
However, designing an application architecture that is
amenable to dynamic code loading requires careful consideration
of JavaScript language issues such as function closures, scoping,
etc. Moreover, an optimal decomposition of code into dynamically
loaded components often requires developers to set aside the semantic
groupings of code and instead primarily consider the execution
order of functions. Of course, evolving code and changing
user workloads make both of these issues a software maintenance
nightmare.

Back in 2007, I was looking at ways to improve the initial load time and reduce user dropout. The number of users who would not wait for the page to load and go away was growing day by day as we introduced new and cool features. It was a surprise. We thought new features will keep more users on our site but the opposite happened. Analysis concluded it was the initial loading time that caused more dropout than it retained users. So, all our hard work was essentially going to drain and we had to come up with something ground breaking to solve the problem. Of course we had already tried all the basic stuffs – IIS compression, browser caching, on-demand loading of JavaScript, css and html when user does something, deferred JavaScript execution – but nothing helped. The frameworks and our own hand coded framework was just too large. So, the idea tricked me, what if we could load functions inside a class in two steps. First step will load the class with absolutely essential functions and second step will inject more functions to the existing classes.

I published a codeproject article which shows you 7 tricks to significantly improve page load time even if you have large amount of Javascript used on the page.

7 Tips for Loading JavaScript Rich Web 2.0-like Sites Significantly Faster

1. Use Doloto
2. Split a Class into Multiple JavaScript Files
3. Stub the Functions Which Aren't Called During Initial Load
4. JavaScript Code in Text
5. Break UI Loading into Multiple Stages
6. Always Grow Content from Top to Bottom, Never Shrink or Jump
7. Deliver Browser Specific Script from Server

If you like these tricks, please vote for me!

  

Windows 7 64bit works!

Windows 7 64bit finally works! This is the first 64bit OS I could really use in my daily acitvities. I tried Vista 64bit, it was unreliable. It would show blue screen right when I am about to make a presentation to the CEO. Until Microsoft released SP1, Vista 64 bit was not usable at all. Then came Windows 7 beta. I immediately tried the 64bit version of Windows 7 beta. It was even worse than Vista. It would crash every now and then – waking up from standby, trying to do livemeeting share, switching screens, plugging in external USB drives and what not. So, I patiently waited for the final version to come out before I get on installing it on all my laptops. Happy to say, the final version works perfectly on HP tx2000 Tablet PC, DELL Vostro 1500, DELL Inspiron 1520. Once you do a full windows update and install some drivers here and there, it all works perfectly. And let me say, Windows 7 is beautiful. I found back the joy of working on computers again!

Working on 64bit Operating System is challenging. You don’t always find the right printer driver. Your cool external USB speakers won’t work – even if it is made by Microsoft. And above all, there’s that C:\Windows\Winsxs folder which keeps increasing forever. By the time I was done with Vista 64bit (two years approx in business), my Winsxs folder was staggering 26 GB eating up every bit out of my C: partition. I had no choice but to format and start over. It seems like this folder keeps copy of every single DLL version it ever sees. The more windows update I do, the larger it gets. Now on a fresh new Windows 7 installation, after installing VS 2008, Office Applications, Windows Live applications and some handy tools, the Winsxs folder is 5.62 GB. Let’s see how it keeps growing over the year. A useful information for 64bit wannabes, make sure your C partition is at least 60 GB. I just installed Windows 7 64bit 3 days back and it has already taken 31 GB space.

Since I am doing a totally useless post, let me sprinkle some productivity tips on it before you lose interest reading my blog.

I realized I do a lot of context swiching. I get over 200 mails per day, so I pretty much switch focus from Visual Studio/Browser to Outlook once every minute, which is big cencentration killer. So, I tried the above setup on my 25” screen and it works great!

The left half of the screen is visual studio and the right half screen shows Outlook and my todolist. As you see, I can see the emails coming up on Outlook without ever switching. The Visual Studio screen width is the right size to read code without horizontally scrolling. The right bottom half of the screen shows my toodlist so that I am always doing the right task from my todolist and not wondering around heedless. If I browse, I bring up the browser on top of the Visual Studio and keep the right half same so that while browsing I am not missing important mails and I still have an eye on my next actions.

I have been using Toodledo for a year. I love it! It has a geat iPhone app which is the only reason I use Toodledo and not other alternatives. The ajax interface is slick, especially when you use Google Chrome to make an application out of it on your desktop. You can turn on keyboard shortcuts and then the toodledo inside Google Chrome’s application like view becomes the best web based todolist application out there. Whenever I file a task, I hit ‘n’, enter the task title, press tab, 1/2 for priority, hit enter and I am done. How convenient! Especially when I read mails and file actionable tasks at least 40 to 60 times per day.

  

AspectF fluent way to put Aspects into your code for separation of concern

Aspects are common features that you write every now and then in different parts of your project. it can be some specific way of handling exceptions in your code, or logging method calls, or timing execution of methods, or retrying some methods and so on. If you are not doing it using any Aspect Oriented Programming framework, then you are repeating a lot of similar code throughout the project, which is making your code hard to maintain. For ex, say you have a business layer where methods need to be logged, errors need to be handled in a certain way, execution needs to be timed, database operations need to be retried and so on. So, you write code like this:

public bool InsertCustomer(string firstName, string lastName, int age, 
    Dictionary<string, string> attributes)
{
    if (string.IsNullOrEmpty(firstName)) 
        throw new ApplicationException("first name cannot be empty");

    if (string.IsNullOrEmpty(lastName))
        throw new ApplicationException("last name cannot be empty");

    if (age < 0)
        throw new ApplicationException("Age must be non-zero");

    if (null == attributes)
        throw new ApplicationException("Attributes must not be null");

    // Log customer inserts and time the execution
    Logger.Writer.WriteLine("Inserting customer data...");
    DateTime start = DateTime.Now;

    try
    {
        CustomerData data = new CustomerData();
        bool result = data.Insert(firstName, lastName, age, attributes);
        if (result == true)
        {
            Logger.Writer.Write("Successfully inserted customer data in " 
                + (DateTime.Now-start).TotalSeconds + " seconds");
        }
        return result;
    }
    catch (Exception x)
    {
        // Try once more, may be it was a network blip or some temporary downtime
        try
        {
            CustomerData data = new CustomerData();
            if (result == true)
            {
                Logger.Writer.Write("Successfully inserted customer data in " 
                    + (DateTime.Now-start).TotalSeconds + " seconds");
            }
            return result;
        }
        catch 
        {
            // Failed on retry, safe to assume permanent failure.

            // Log the exceptions produced
            Exception current = x;
            int indent = 0;
            while (current != null)
            {
                string message = new string(Enumerable.Repeat('\t', indent).ToArray())
                    + current.Message;
                Debug.WriteLine(message);
                Logger.Writer.WriteLine(message);
                current = current.InnerException;
                indent++;
            }
            Debug.WriteLine(x.StackTrace);
            Logger.Writer.WriteLine(x.StackTrace);

            return false;
        }
    }

}

Here  you see the two lines of real code, which inserts the Customer calling a class, is hardly visible due to all the concerns (log, retry, exception handling, timing) you have to implement in business layer. There’s validation, error handling, caching, logging, timing, auditing, retring, dependency resolving and what not in business layers nowadays. The more a project matures, the more concerns get into your codebase. So, you keep copying and pasting boilerplate codes and write the tiny amount of real stuff somewhere inside that boilerplate. What’s worse, you have to do this for every business layer method. Say now you want to add a UpdateCustomer method in your business layer. you have to copy all the concerns again and put the two lines of real code somewhere inside that boilerplate.

Think of a scenario where you have to make a project wide change on how errors are handled. You have to go through all the hundreds of business layer functions you wrote and change it one by one. Say you need to change the way you time execution. You have to go through hundreds of functions again and do that.

Aspect Oriented Programming solves these challenges. When you are doing AOP, you do it the cool way:

[EnsureNonNullParameters]
[Log]
[TimeExecution]
[RetryOnceOnFailure]
public void InsertCustomerTheCoolway(string firstName, string lastName, int age,
    Dictionary<string, string> attributes)
{
    CustomerData data = new CustomerData();
    data.Insert(firstName, lastName, age, attributes);
}

Here you have separated the common stuffs like logging, timing, retrying, validation, which are formally called ‘concern’, completely out of your real code. The method is nice and clean, to the point. All the concerns are taken out of the code of the function and added to the function using Attribute. Each Attribute here represents one Aspect. For example, you can add Logging aspect to any function just by adding the Log attribute. Whatever AOP framework you use, the framework ensures the Aspects are weaved into the code either at build time or at runtime.

There are AOP frameworks which allows you to weave the Aspects at compile time by using post build events and IL manipulations eg PostSharp, some does it at runtime using DynamicProxy and some requires your classes to inherit from ContextBoundObject in order to support Aspects using C# built-in features. All of these have some barrier to entry, you have to justify using some external library, do enough performance test to make sure the libraries scale and so on. What you need is a dead simple way to achieve “separation of concern”, may not be full blown Aspect Oriented Programming. Remember, the purpose here is separation of concern and keep code nice and clean.

So, let me show you a dead simple way of separation of concern, writing standard C# code, no Attribute or IL manipulation black magics, simple calls to classes and delegates, yet achieve nice separation of concern in a reusable and maintainable way. Best of all, it’s light, just one small class.

public void InsertCustomerTheEasyWay(string firstName, string lastName, int age,
    Dictionary<string, string> attributes)
{
    AspectF.Define
        .Log(Logger.Writer, "Inserting customer the easy way")
        .HowLong(Logger.Writer, "Starting customer insert", 
                                "Inserted customer in {1} seconds")
        .Retry()
        .Do(() =>
            {
                CustomerData data = new CustomerData();
                data.Insert(firstName, lastName, age, attributes);
            });
}

If you want to read details about how it works and it can save you hundreds of hours of repeatetive coding, read on:

AspectF fluent way to add Aspects for cleaner maintainable code

If you like it, please vote for me!

  

ASP.NET AJAX testing made easy using Visual Studio 2008 Web Test

Visual Studio 2008 comes with rich Web Testing support, but it’s not rich enough to test highly dynamic AJAX websites where the page content is generated dynamically from database and the same page output changes very frequently based on some external data source e.g. RSS feed. Although you can use the Web Test Record feature to record some browser actions by running a real browser and then play it back. But if the page that you are testing changes everytime you visit the page, then your recorded tests no longer work as expected. The problem with recorded Web Test is that it stores the generated ASP.NET Control ID, Form field names inside the test. If the page is no longer producing the same ASP.NET Control ID or same Form fields, then the recorded test no longer works. A very simple example is in VS Web Test, you can say “click the button with ID ctrl00_UpdatePanel003_SubmitButton002”, but you cannot say “click the 2nd Submit button inside the third UpdatePanel”. Another key limitation is in Web Tests, you cannot address Controls using the Server side Control ID like “SubmitButton”. You have to always use the generated Client ID which is something weird like “ctrl_00_SomeControl001_SubmitButton”. Moreover, if you are making AJAX calls where certain call returns some JSON or updates some UpdatePanel and then based on the server returned response, you want to make further AJAX calls or post the refreshed UpdatePanel, then recorded tests don’t work properly. You *do* have the option to write the tests hand coded and write code to handle such scenario but it’s pretty difficult to write hand coded tests when you are using UpdatePanels because you have to keep track of the page viewstates, form hidden variables etc across async post backs. So, I have built a library that makes it significantly easier to test dynamic AJAX websites and UpdatePanel rich web pages. There are several ExtractionRule and ValidationRule available in the library which makes testing Cookies, Response Headers, JSON output, discovering all UpdatePanel in a page, finding controls in the response body, finding controls inside some UpdatePanel all very easy.

First, let me give you an example of what can be tested using this library. My open source project Dropthings produces a Web 2.0 Start Page where the page is composed of widgets.

Each widget is composed of two UpdatePanel. There’s a header area in each widget which is one UpdatePanel and the body area is another UpdatePanel. Each widget is rendered from database using the unique ID of the widget row, which is an INT IDENTITY. Every page has unique widgets, with unique ASP.NET Control ID. As a result, there’s no way you can record a test and play it back because none of the ASP.NET Control IDs are ever same for the same page on different visits. This is where my library comes to the rescue.

See the web test I did:

This test simulates an anonymous user visit. When anonymous user visits Dropthings for the first time, two pages are created with some default widgets. You can also add new widgets on the page, you can drag & drop widgets, you can delete a widget that you don’t like.

This Web Test simulates these behaviors automatically:

• Visit the homepage
• Show the widget list which is an UpdatePanel. It checks if the UpdatePanel contains the BBC World widget.
• Then it clicks on the “Edit” link of the “How to of the day” widget which brings up some options dynamically inside an UpdatePanel. Then it tries to change the Dropdown value inside the UpdatePanel to 10.
• Adds a new widget from the Widget List. Ensures that the UpdatePanel postback successfully renders the new widget.
• Deletes the newly added widget and ensures the widget is gone.
• Logs user out.

If you want to learn details about the project, read my codeproject article:

Please vote if you find this useful.

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

Dropthings – my open source Web 2.0 Ajax Portal has gone through a technology overhauling. Previously it was built using ASP.NET AJAX, a little bit of Workflow Foundation and Linq to SQL. Now Dropthings boasts full jQuery front-end combined with ASP.NET AJAX UpdatePanel, Silverlight widget, full Workflow Foundation implementation on the business layer, 100% Linq to SQL Compiled Queries on the data access layer, Dependency Injection and Inversion of Control (IoC) using Microsoft Enterprise Library 4.1 and Unity. It also has a ASP.NET AJAX Web Test framework that makes it real easy to write Web Tests that simulates real user actions on AJAX web pages. This article will walk you through the challenges in getting these new technologies to work in an ASP.NET website and how performance, scalability, extensibility and maintainability has significantly improved by the new technologies. Dropthings has been licensed for commercial use by prominent companies including BT Business, Intel, Microsoft IS, Denmark Government portal for Citizens; Startups like Limead and many more. So, this is serious stuff! There’s a very cool open source implementation of Dropthings framework available at National University of Singapore portal.

Visit: http://dropthings.omaralzabir.com

I have published a new article on this on CodeProject:

http://www.codeproject.com/KB/ajax/Web20Portal.aspx

Get the source code

Latest source code is hosted at Google code:

http://code.google.com/p/dropthings

There’s a CodePlex site for documentation and issue tracking:

http://www.codeplex.com/dropthings

You will need Visual Studio 2008 Team Suite with Service Pack 1 and Silverlight 2 SDK in order to run all the projects. If you have only Visual Studio 2008 Professional, then you will have to remove the Dropthings.Test project.

New features introduced

Dropthings new release has the following features:

• Template users – you can define a user who’s pages and widgets are used as a template for new users. Whatever you put in that template user’s pages, it will be copied for every new user. Thus this is an easier way to define the default pages and widgets for new users. Similarly you can do the same for a registered user. The template users can be defined in the web.config.
• Widget-to-Widget communication – Widgets can send message to each other. Widgets can subscribe to an Event Broker and exchange messages using a Pub-Sub pattern.
• WidgetZone – you can create any number of zones in any shape on the page. You can have widgets laid in horizontal layout, you can have zones on different places on the page and so on. With this zone model, you are no longer limited to the Page-Column model where you could only have N vertical columns.
• Role based widgets – now widgets are mapped to roles so that you can allow different users to see different widget list using ManageWidgetPersmission.aspx.
• Role based page setup – you can define page setup for different roles. For ex, Managers see different pages and widgets than Employees.
• Widget maximize – you can maximize a widget to take full screen. Handy for widgets with lots of content.
• Free form resize – you can freely resize widgets vertically.
• Silverlight Widgets – You can now make widgets in Silverlight!

Why the technology overhauling

Performance, Scalability, Maintainability and Extensibility – four key reasons for the overhauling. Each new technology solved one of more of these problems.

First, jQuery was used to replace my personal hand-coded large amount of Javascript code that offered the client side drag & drop and other UI effects. jQuery already has a rich set of library for Drag & Drop, Animations, Event handling, cross browser javascript framework and so on. So, using jQuery means opening the door to thousands of jQuery plugins to be offered on Dropthings. This made Dropthings highly extensible on the client side. Moreover, jQuery is very light. Unlike AJAX Control Toolkit jumbo sized framework and heavy control extenders, jQuery is very lean. So, total javascript size decreased significantly resulting in improved page load time. In total, the jQuery framework, AJAX basic framework, all my stuffs are total 395KB, sweet! Performance is key; it makes or breaks a product.

Secondly, Linq to SQL queries are replaced with Compiled Queries. Dropthings did not survive a load test when regular lambda expressions were used to query database. I could only reach up to 12 Req/Sec using 20 concurrent users without burning up web server CPU on a Quad Core DELL server.

Thirdly, Workflow Foundation is used to build operations that require multiple Data Access Classes to perform together in a single transaction. Instead of writing large functions with many if…else conditions, for…loops, it’s better to write them in a Workflow because you can visually see the flow of execution and you can reuse Activities among different Workflows. Best of all, architects can design workflows and developers can fill-in code inside Activities. So, I could design a complex operations in a workflow without writing the real code inside Activities and then ask someone else to implement each Activity. It is like handing over a design document to developers to implement each unit module, only that here everything is strongly typed and verified by compiler. If you strictly follow Single Responsibility Principle for your Activities, which is a smart way of saying one Activity does only one and very simple task, you end up with a highly reusable and maintainable business layer and a very clean code that’s easily extensible.

Fourthly, Unity Dependency Injection (DI) framework is used to pave the path for unit testing and dependency injection. It offers Inversion of Control (IoC), which enables testing individual classes in isolation. Moreover, it has a handy feature to control lifetime of objects. Instead of creating instance of commonly used classes several times within the same request, you can make instances thread level, which means only one instance is created per thread and subsequent calls reuse the same instance. Are these going over your head? No worries, continue reading, I will explain later on.

Fifthly, enabling API for Silverlight widgets allows more interactive widgets to be built using Silverlight. HTML and Javascripts still have limitations on smooth graphics and continuous transmission of data from web server. Silverlight solves all of these problems.

Read the article for details on how all these improvements were done and how all these hot techs play together in a very useful open source project for enterprises.

http://www.codeproject.com/KB/ajax/Web20Portal.aspx

Don’t forget to vote for me if you like it.

Memory Leak with delegates and workflow foundation

Recently after Load Testing my open source project Dropthings, I encountered a lot of memory leak. I found lots of Workflow Instances and Linq Entities were left in memory and never collected. After profiling the web application using .NET Memory Profiler, it showed the real picture:

It shows you that instances of the several types are being created but not being removed. You see the “New” column has positive value, but the “Remove” column has 0. That means new instances are being created, but not removed. Basically the way you do Memory Profiling is, you take two snapshots. Say you take one snapshot when you first visit your website. Then you do some action on the website that results in allocation of objects. Then you take another snapshot. When you compare both snapshots, you can see how many instances of classes were created between these two snapshots and how many were removed. If they are not equal, then you have leak. Generally in web application many objects are created on every page hit and the end of the request, all those objects are supposed to be released. If they are not released, then we have a problem. But that’s the scenario for desktop applications because in a desktop application, objects can remain in memory until app is closed. But you should know best from the code which objects were supposed to go out of scope and get released.

For beginners, leak means objects are being allocated but not being freed because someone is holding reference to the objects. When objects leak, they remain in memory forever, until the process (or app domain) is closed. So, if you have a leaky website, your website is continuously taking up memory until it runs out of memory on the web server and thus crash. So, memory leak is a bad – it prevents you from running your product for long duration and requires frequent restart of app pool.

So, the above screenshot shows Workflow and Linq related classes are not being removed, and thus leaking. This means somewhere workflow instances are not being released and thus all workflow related objects are remaining. You can see the number is same 48 for all workflow related objects. This is a good indication that, almost every instance of workflow is leaked because there were total 48 workflows created and ran. Moreover it indicates we have a leak from a top Workflow instance level, not in some specific Activity or somewhere deep in the code.

As the workflows use Linq stuff, they held reference to the Linq stuffs and thus the Linq stuffs leaked as well. Sometimes you might be looking for why A is leaking. But you actually end up finding that since B was holding reference to A and B was leaking and thus A was leaking as well. This is sometimes tricky to figure out and you spend a lot of time looking at the wrong direction.

Now let me show you the buggy code:

ManualWorkflowSchedulerService manualScheduler = 
  workflowRuntime.GetService<ManualWorkflowSchedulerService>();

WorkflowInstance instance = workflowRuntime.CreateWorkflow(workflowType, properties);
instance.Start();

EventHandler<WorkflowCompletedEventArgs> completedHandler = null;
completedHandler = delegate(object o, WorkflowCompletedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instance.InstanceId) // 1. instance
    {
        workflowRuntime.WorkflowCompleted -= completedHandler; // 2. terminatedhandler

        // copy the output parameters in the specified properties dictionary
        Dictionary<string,object>.Enumerator enumerator = 
            e.OutputParameters.GetEnumerator();
        while( enumerator.MoveNext() )
        {
            KeyValuePair<string,object> pair = enumerator.Current;
            if( properties.ContainsKey(pair.Key) )
            {
                properties[pair.Key] = pair.Value;
            }
        }
    }
};

Exception x  = null;
EventHandler<WorkflowTerminatedEventArgs> terminatedHandler = null;
terminatedHandler = delegate(object o, WorkflowTerminatedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instance.InstanceId) // 3. instance
    {
        workflowRuntime.WorkflowTerminated -= terminatedHandler; // 4. completeHandler
        Debug.WriteLine( e.Exception );

        x = e.Exception;
    }
};

workflowRuntime.WorkflowCompleted += completedHandler;
workflowRuntime.WorkflowTerminated += terminatedHandler;

manualScheduler.RunWorkflow(instance.InstanceId);

Can you spot the code where it leaked?

I have numbered the lines in comment where the leak is happening. Here the delegate is acting like a closure and those who are from Javascript background know closure is evil. They leak memory unless very carefully written. Here the delegate keeps a reference to the instance object. So, if somehow delegate is not released, the instance will remain in memory forever and thus leak. Now can you find a situation when the delegate will not be released?

Say the workflow completed. It will fire the completeHandler. But the completeHandler will not release the terminateHandler. Thus the terminateHandler remains in memory and it also holds reference to the instance. So, we have a leaky delegate leaking whatever it is holding onto outside it’s scope. Here the only thing outside the scope if the instance, which it is tried to access from the parent function.

Since the workflow instance is not released, all the properties the workflow and all the activities inside it are holding onto remains in memory. Most of the workflows and activities expose public properties which are Linq Entities. Thus the Linq Entities remain in memory. Now Linq Entities keep a reference to the DataContext from where it is produced. Thus we have DataContext remaining in memory. Moreover, DataContext keeps reference to many internal objects and metadata cacahe, so they remain in memory as well.

So, the correct code is:

ManualWorkflowSchedulerService manualScheduler = 
    workflowRuntime.GetService<ManualWorkflowSchedulerService>();

WorkflowInstance instance = workflowRuntime.CreateWorkflow(workflowType, properties);
instance.Start();
var instanceId = instance.InstanceId;

EventHandler<WorkflowCompletedEventArgs> completedHandler = null;
completedHandler = delegate(object o, WorkflowCompletedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instanceId) // 1. instanceId is a Guid
    {
        // copy the output parameters in the specified properties dictionary
        Dictionary<string,object>.Enumerator enumerator = 
                e.OutputParameters.GetEnumerator();
        while( enumerator.MoveNext() )
        {
            KeyValuePair<string,object> pair = enumerator.Current;
            if( properties.ContainsKey(pair.Key) )
            {
                properties[pair.Key] = pair.Value;
            }
        }
    }
};

Exception x  = null;
EventHandler<WorkflowTerminatedEventArgs> terminatedHandler = null;
terminatedHandler = delegate(object o, WorkflowTerminatedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instanceId) // 2. instanceId is a Guid
    {
        x = e.Exception;
        Debug.WriteLine(e.Exception);
    }
};

workflowRuntime.WorkflowCompleted += completedHandler;
workflowRuntime.WorkflowTerminated += terminatedHandler;

manualScheduler.RunWorkflow(instance.InstanceId);
// 3. Both delegates are now released
workflowRuntime.WorkflowTerminated -= terminatedHandler;
workflowRuntime.WorkflowCompleted -= completedHandler;

There are two changes – in both delegates, the instanceId variable is passed, instead of the instance. Since instanceId is a Guid, which is a struct type data type, not a class, there’s no issue of referencing. Structs are copied, not referenced. So, they don’t leak memory. Secondly, both delegates are released at the end of the workflow execution, thus releasing both references.

In Dropthings, I am using the famous CallWorkflow Activity by John Flanders, which is widely used to execute one Workflow from another synchronously. There’s a CallWorkflowService class which is responsible for synchronously executing another workflow and that has similar memory leak problem. The original code of the service is as following:

public class CallWorkflowService : WorkflowRuntimeService
{
    #region Methods

    public void StartWorkflow(Type workflowType,Dictionary<string,object> inparms, 
           Guid caller,IComparable qn)
    {
        WorkflowRuntime wr = this.Runtime;
        WorkflowInstance wi = wr.CreateWorkflow(workflowType,inparms);
        wi.Start();
        ManualWorkflowSchedulerService ss = 
             wr.GetService<ManualWorkflowSchedulerService>();
        if (ss != null)
            ss.RunWorkflow(wi.InstanceId);
        EventHandler<WorkflowCompletedEventArgs> d  = null;
        d = delegate(object o, WorkflowCompletedEventArgs e)
        {
            if (e.WorkflowInstance.InstanceId ==wi.InstanceId)
            {
                wr.WorkflowCompleted -= d;
                WorkflowInstance c = wr.GetWorkflow(caller);
                c.EnqueueItem(qn, e.OutputParameters, null, null);
            }
        };
        EventHandler<WorkflowTerminatedEventArgs> te = null;
        te = delegate(object o, WorkflowTerminatedEventArgs e)
        {
            if (e.WorkflowInstance.InstanceId == wi.InstanceId)
            {
                wr.WorkflowTerminated -= te;
                WorkflowInstance c = wr.GetWorkflow(caller);
                c.EnqueueItem(qn, new Exception("Called Workflow Terminated", 
                  e.Exception), null, null);
            }
        };
        wr.WorkflowCompleted += d;
        wr.WorkflowTerminated += te;
    }

    #endregion Methods
}

As you see, it has that same delegate holding reference to instance object problem. Moreover, there’s some queue stuff there, which requires the caller and qn parameter passed to the StartWorkflow function. So, not a straight forward fix.

I tried to rewrite the whole CallWorkflowService so that it does not require two delegates to be created per Workflow. Then I took the delegates out. Thus there’s no chance of closure holding reference to unwanted objects. The result looks like this:

public class CallWorkflowService : WorkflowRuntimeService
{
    #region Fields

    private EventHandler<WorkflowCompletedEventArgs> _CompletedHandler = null;
    private EventHandler<WorkflowTerminatedEventArgs> _TerminatedHandler = null;
    private Dictionary<Guid, WorkflowInfo> _WorkflowQueue = 
       new Dictionary<Guid, WorkflowInfo>();

    #endregion Fields

    #region Methods

    public void StartWorkflow(Type workflowType,Dictionary<string,object> inparms,
        Guid caller,IComparable qn)
    {
        WorkflowRuntime wr = this.Runtime;
        WorkflowInstance wi = wr.CreateWorkflow(workflowType,inparms);
        wi.Start();

        var instanceId = wi.InstanceId;
        _WorkflowQueue[instanceId] = new WorkflowInfo { Caller = caller, qn = qn };

        ManualWorkflowSchedulerService ss = 
           wr.GetService<ManualWorkflowSchedulerService>();
        if (ss != null)
            ss.RunWorkflow(wi.InstanceId);
    }

    protected override void OnStarted()
    {
        base.OnStarted();

        if (null == _CompletedHandler)
        {
            _CompletedHandler = delegate(object o, WorkflowCompletedEventArgs e)
            {
                var instanceId = e.WorkflowInstance.InstanceId;
                if (_WorkflowQueue.ContainsKey(instanceId))
                {
                    WorkflowInfo wf = _WorkflowQueue[instanceId];
                    WorkflowInstance c = this.Runtime.GetWorkflow(wf.Caller);
                    c.EnqueueItem(wf.qn, e.OutputParameters, null, null);
                    _WorkflowQueue.Remove(instanceId);
                }
            };
            this.Runtime.WorkflowCompleted += _CompletedHandler;
        }

        if (null == _TerminatedHandler)
        {
            _TerminatedHandler = delegate(object o, WorkflowTerminatedEventArgs e)
            {
                var instanceId = e.WorkflowInstance.InstanceId;
                if (_WorkflowQueue.ContainsKey(instanceId))
                {
                    WorkflowInfo wf = _WorkflowQueue[instanceId];
                    WorkflowInstance c = this.Runtime.GetWorkflow(wf.Caller);
                    c.EnqueueItem(wf.qn, 
                      new Exception("Called Workflow Terminated", e.Exception), 
                      null, null);
                    _WorkflowQueue.Remove(instanceId);
                }
            };

            this.Runtime.WorkflowTerminated += _TerminatedHandler;
        }
    }

    protected override void OnStopped()
    {
        _WorkflowQueue.Clear();

        base.OnStopped();
    }

    #endregion Methods

    #region Nested Types

    private struct WorkflowInfo
    {
        #region Fields

        public Guid Caller;
        public IComparable qn;

        #endregion Fields
    }

    #endregion Nested Types
}

After fixing the problem, another Memory Profile result showed the leak is gone:

As you see, the numbers vary, which means there’s no consistent leak. Moreover, looking at the types that remains in memory, they look more like metadata than instances of classes.  So, they are basically cached instances of metadata, not instances allocated during workflow execution which are supposed to be freed. So, we solved the memory leak!

Now you know how to write anonymous delegates without leaking memory and how to run workflow without leaking them. Basically, the principle theory is – if you are referencing some outside object from an anonymous delegate, make sure that object is not holding reference to the delegate in some way, may be directly or may be via some child objects of its own. Because then you have a circular reference. If possible, do not try to access objects e.g. instance inside an anonymous delegate that is declared outside the delegate. Try accessing instrinsic data types like int, string, DateTime, Guid etc which are not reference type variables. So, instead of referencing to an object, you should declare local variables e.g. instanceId that gets the value of properties (e.g. instance.InstanceId) from the object and then use those local variables inside the anonymous delegate.

Optimize ASP.NET Membership Stored Procedures for greater speed and scalability

Last year at Pageflakes, when we were getting millions of hits per day, we were having query timeout due to lock timeout and Transaction Deadlock errors. These locks were produced from aspnet_Users and aspnet_Membership tables. Since both of these tables are very high read (almost every request causes a read on these tables) and high write (every anonymous visit creates a row on aspnet_Users), there were just way too many locks created on these tables per second. SQL Counters showed thousands of locks per second being created. Moreover, we had queries that would select thousands of rows from these tables frequently and thus produced more locks for longer period, forcing other queries to timeout and thus throw errors on the website.

If you have read my last blog post, you know why such locks happen. Basically every table when it grows up to hold millions of records and becomes popular goes through this trouble. It’s just a part of scalability problem that is common to database. But we rarely take prevention about it in our early design.

The solution is simple, you should either have WITH (NOLOCK) or SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED before SELECT queries. Either of this will do. They tell SQL Server not to hold any lock on the table while it is reading the table. If some row is locked while the read is happening, it will just ignore that row. When you are reading a table thousand times per second, without these options, you are issuing lock on many places around the table thousand times per second. It not only makes read from table slower, but also so many lock prevents insert, update, delete from happening timely and thus queries timeout. If you have queries like “show the currently online users from last one hour based on LastActivityDate field”, that is going to issue such a wide lock that even other harmless select queries will timeout. And did I tell you that there’s no index on LastActivityDate on aspnet_Users table?

Now don’t blame yourself for not putting either of these options on your every stored proc and every dynamically generated SQL from the very first day. ASP.NET developers made the same mistake. You won’t see either of these used in any of the stored procs used by ASP.NET Membership. For example, the following stored proc gets called whenever you access Profile object:

ALTER PROCEDURE [dbo].[aspnet_Profile_GetProperties]
    @ApplicationName      nvarchar(256),
    @UserName             nvarchar(256),
    @CurrentTimeUtc       datetime
AS
BEGIN

    DECLARE @ApplicationId uniqueidentifier
    SELECT  @ApplicationId = NULL
    SELECT  @ApplicationId = ApplicationId FROM 
      dbo.aspnet_Applications WHERE LOWER(@ApplicationName) = LoweredApplicationName
    IF (@ApplicationId IS NULL)
        RETURN

    DECLARE @UserId uniqueidentifier
    DECLARE @LastActivityDate datetime
    SELECT  @UserId = NULL

    SELECT @UserId = UserId, @LastActivityDate = LastActivityDate
    FROM   dbo.aspnet_Users 
    WHERE  ApplicationId = @ApplicationId AND LoweredUserName = LOWER(@UserName)

    IF (@UserId IS NULL)
        RETURN
    SELECT TOP 1 PropertyNames, PropertyValuesString, PropertyValuesBinary
    FROM         dbo.aspnet_Profile
    WHERE        UserId = @UserId

    IF (@@ROWCOUNT > 0)
    BEGIN
        UPDATE dbo.aspnet_Users
        SET    LastActivityDate=@CurrentTimeUtc
        WHERE  UserId = @UserId
    END
END

There are two SELECT operations that hold lock on two very high read tables – aspnet_Users and aspnet_Profile. Moreover, there’s a nasty UPDATE statement. It tries to update the LastActivityDate of a user whenever you access Profile object for the first time within a http request.

This stored proc alone is enough to bring your site down. It did to us because we are using Profile Provider everywhere. This stored proc was called around 300 times/sec. We were having nightmarish slow performance on the website and many lock timeouts and transaction deadlocks. So, we added the transaction isolation level and we also modified the UPDATE statement to only perform an update when the LastActivityDate is over an hour. So, this means, the same user’s LastActivityDate won’t be updated if the user hits the site within the same hour.

So, after the modifications, the stored proc looked like this:

ALTER PROCEDURE [dbo].[aspnet_Profile_GetProperties]
    @ApplicationName      nvarchar(256),
    @UserName             nvarchar(256),
    @CurrentTimeUtc       datetime
AS
BEGIN
    -- 1. Please no more locks during reads
    SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;

    DECLARE @ApplicationId uniqueidentifier
    --SELECT  @ApplicationId = NULL
    --SELECT  @ApplicationId = ApplicationId FROM dbo.aspnet_Applications 
    WHERE LOWER(@ApplicationName) = LoweredApplicationName
    --IF (@ApplicationId IS NULL)
    --    RETURN
    
    -- 2. No more call to Application table. We have only one app dude!
    SET @ApplicationId = dbo.udfGetAppId()

    DECLARE @UserId uniqueidentifier
    DECLARE @LastActivityDate datetime
    SELECT  @UserId = NULL

    SELECT @UserId = UserId, @LastActivityDate = LastActivityDate
    FROM   dbo.aspnet_Users 
    WHERE  ApplicationId = @ApplicationId AND LoweredUserName = LOWER(@UserName)

    IF (@UserId IS NULL)
        RETURN
    SELECT TOP 1 PropertyNames, PropertyValuesString, PropertyValuesBinary
    FROM         dbo.aspnet_Profile
    WHERE        UserId = @UserId

    IF (@@ROWCOUNT > 0)
    BEGIN
        -- 3. Do not update the same user within an hour
        IF DateDiff(n, @LastActivityDate, @CurrentTimeUtc) > 60
        BEGIN
            -- 4. Use ROWLOCK to lock only a row since we know this query
            -- is highly selective
            UPDATE dbo.aspnet_Users WITH(ROWLOCK)
            SET    LastActivityDate=@CurrentTimeUtc
            WHERE  UserId = @UserId
        END
    END
END

The changes I made are numbered and commented. No need for further explanation. The only tricky thing here is, I have eliminate call to Application table just to get the ApplicationID from ApplicationName. Since there’s only one application in a database (ever heard of multiple applications storing their user separately on the same database and the same table?), we don’t need to look up the ApplicationID on every call to every Membership stored proc. We can just get the ID and hard code it in a function.

CREATE FUNCTION dbo.udfGetAppId()
RETURNS uniqueidentifier
WITH EXECUTE AS CALLER
AS
BEGIN
    RETURN CONVERT(uniqueidentifier, 'fd639154-299a-4a9d-b273-69dc28eb6388')
END;

This UDF returns the ApplicationID that I have hardcoded copying from the Application table. Thus it eliminates the need for quering on the Application table.

Similarly you should do the changes in all other stored procedures that belong to Membership Provider. All the stroc procs are missing proper locking, issues aggressive lock during update and too frequent updates than practical need. Most of them also try to resolve ApplicationID from ApplicationName, which is unnecessary when you have only one web application per database. Make these changes and enjoy lock contention free super performance from Membership Provider!

Linq to SQL solve Transaction deadlock and Query timeout problem using uncommitted reads

When your database tables start accumulating thousands of rows and many users start working on the same table concurrently, SELECT queries on the tables start producing lock contentions and transaction deadlocks. This is a common problem in any high volume website. As soon as you start getting several concurrent users hitting your website that results in SELECT queries on some large table like aspnet_users table that are also being updated very frequently, you end up having one of these errors:

Transaction (Process ID ##) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

Or,

Timeout Expired. The Timeout Period Elapsed Prior To Completion Of The Operation Or The Server Is Not Responding.

The solution to these problems are – use proper index on the table and use transaction isolation level Read Uncommitted or WITH (NOLOCK) in your SELECT queries. So, if you had a query like this:

SELECT * FORM aspnet_users 
where ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

You should end up having any of the above errors under high load. There are two ways to solve this:

SET TRANSACTION LEVEL READ UNCOMMITTED;
SELECT * FROM aspnet_Users 
WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

Or use the WITH (NOLOCK):

SELECT * FROM aspnet_Users WITH (NOLOCK) 
WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

The reason for the errors are that since aspnet_users is a high read and high write table, during read, the table is partially locked and during write, it is also locked. So, when the locks overlap on each other from several queries and especially when there’s a query that’s trying to read a large number of rows and thus locking large number of rows, some of the queries either timeout or produce deadlocks.

Linq to Sql does not produce queries with the WITH (NOLOCK) option nor does it use READ UNCOMMITTED. So, if you are using Linq to SQL queries, you are going to end up with any of these problems on production pretty soon when your site becomes highly popular.

For example, here’s a very simple query:

using (var db = new DropthingsDataContext())
{
    var user = db.aspnet_Users.First();
    var pages = user.Pages.ToList();
}

DropthingsDataContext is a DataContext built from Dropthings database.

When you attach SQL Profiler, you get this:

You see none of the queries have READ UNCOMMITTED or WITH (NOLOCK).

The fix is to do this:

using (var db = new DropthingsDataContext2())
{
    db.Connection.Open();
    db.ExecuteCommand("SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;");

    var user = db.aspnet_Users.First();
    var pages = user.Pages.ToList();
}

This will result in the following profiler output

As you see, both queries execute within the same connection and the isolation level is set before the queries execute. So, both queries enjoy the isolation level.

Now there’s a catch, the connection does not close. This seems to be a bug in the DataContext that when it is disposed, it does not dispose the connection it is holding onto.

In order to solve this, I have made a child class of the DropthingsDataContext named DropthingsDataContext2 which overrides the Dispose method and closes the connection.

   class DropthingsDataContext2 : DropthingsDataContext, IDisposable
    {
        public new void Dispose()
        {
            if (base.Connection != null)
                if (base.Connection.State != System.Data.ConnectionState.Closed)
                {
                    base.Connection.Close();
                    base.Connection.Dispose();
                }

            base.Dispose();            
        }
    }

This solved the connection problem.

There you have it, no more transaction deadlock or lock contention from Linq to SQL queries. But remember, this is only to eliminate such problems when your database already has the right indexes. If you do not have the proper index, then you will end up having lock contention and query timeouts anyway.

There’s one more catch, READ UNCOMMITTED will return rows from transactions that have not completed yet. So, you might be reading rows from transactions that will rollback. Since that’s generally an exceptional scenario, you are more or less safe with uncommitted read, but not for financial applications where transaction rollback is a common scenario. In such case, go for committed read or repeatable read.

There’s another way you can achieve the same, which seems to work, that is using .NET Transactions. Here’s the code snippet:

using (var transaction = new TransactionScope(
    TransactionScopeOption.RequiresNew,
    new TransactionOptions()
    {
        IsolationLevel = IsolationLevel.ReadUncommitted,
        Timeout = TimeSpan.FromSeconds(30)
    }))
{
    using (var db = new DropthingsDataContext())
    {
        var user = db.aspnet_Users.First();
        var pages = user.Pages.ToList();

        transaction.Complete();
    }
}

Profiler shows a transaction begins and ends:

The downside is it wraps your calls in a transaction. So, you are unnecessarily creating transactions even for SELECT operations. When you do this hundred times per second on a web application, it’s a significant over head.

Some really good examples of deadlocks are given in this article:

http://www.code-magazine.com/article.aspx?quickid=0309101&page=2

I highly recommend it.

Strongly typed workflow input and output arguments

When you run a Workflow using Workflow Foundation, you pass arguments to the workflow in a Dictionary form where the type of Dictionary is Dictionary<string, object>. This means you miss the strong typing features of .NET languages. You have to know what arguments the workflow expects by looking at the Workflow public properties. Moreover, there’s no way to make arguments required. You pass parameter, expect it to run, if it throws exception, you pass more arguments, hope it works now. Similarly, if you are running workflow synchronously using ManualWorkflowSchedulerService, you expect return arguments from the Workflow immediately, but there again, you have to rely on the Dictionary key and value pair. No strong typing there as well.

In order to solve this, so that you could pass Workflow arguments as strongly typed classes, you can establish a format that every Workflow has only two arguments named "Request” and “Response” and none other. Whatever needs to be passed to the Workflow and expected out of it, must be passed via Request and must be expected via Response properties. Now the type of these arguments can be workflow specific, it can be any class with one or more parameters. This way, you could write code like this:

The advantages of these strongly typed approach are:

• Compile time validation of input parameters passed to workflow. No risk of passing unexpected object in Dictionary’s object type value.
• Enforce required values by creating Request objects with non-default constructor.
• Establish a fixed contract for Workflow input and output via the strongly typed Request and Response classes or interfaces.
• Validate input arguments for the Workflow directly from the Request class, without going through the overhead of running a workflow.

If we follow this approach, we create workflows with only two DependencyProperty, one for Request and one for Response. Showing you an example from my open source project Dropthings, which uses Workflow for the entire Business Layer. Below you see the Workflow that executes when a new user visits Dropthings.com, creates a new user and setups all the pages and widgets for the user. It has only two Dependency property – Request and Response.

The Request parameters is of type IUserVisitWorkflowRequest. So, you can pass any class as Request argument that implements the interface.

Here I have used fancy inheritance to create Request object hierarchy. You don’t need to do that. Just remember, you can pass any class. You don’t even need to use interface for Request parameter. It can be a class directly. I use all these interfaces in order to facilitate Dependency Inversion.

Similarly, the Response object is also a class.

The Response returns quite some properties. So, it’s kinda handy to wrap them all in one property.

So, there you have it, strongly typed Workflow arguments. You can attach properties of the Request object to any activity directly form the designer:

There’s really no compromise to make in this approach. Everything works as before.

In order to make workflow execution simpler, I use a helper method like the following, that takes the Request and Response object and creates the Dictionary for me. This Dictionary always contains one “Request” and one “Response” entry.

This way, I can run Workflow in strongly typed fashion:

Here I can specify the Request, Response and Workflow type using strong typing. This way I get strongly typed return object as well as pass strongly type Request object. There’s no dictionary building, no risky string key and object type value passing.  You can ignore the ObjectContainer.Resolve() stuff, because that’s just returning me an existing reference of WorkflowRuntime.

Hope you like this approach.

99.99% available ASP.NET and SQL Server SaaS Production Architecture

You have a hot ASP.NET+SQL Server product, growing at thousand users per day and you have hit the limit of your own garage hosting capability. Now that you have enough VC money in your pocket, you are planning to go out and host on some real hosting facility, maybe a colocation or managed hosting. So, you are thinking, how to design a physical architecture that will ensure performance, scalability, security and availability of your product? How can you achieve four-nine (99.99%) availability? How do you securely let your development team connect to production servers? How do you choose the right hardware for web and database server? Should you use Storage Area Network (SAN) or just local disks on RAID? How do you securely connect your office computers to production environment?

Here I will answer all these queries. Let me first show you a diagram that I made for Pageflakes where we ensured we get four-nine availability. Since Pageflakes is a Level 3 SaaS, it’s absolutely important that we build a high performance, highly available product that can be used from anywhere in the world 24/7 and end-user gets quick access to their content with complete personalization and customization of content and can share it with others and to the world. So, you can take this production architecture as a very good candidate for Level 3 SaaS:

Here’s a CodeProject article that explains all the ideas:

99.99% available ASP.NET and SQL Server SaaS Production Architecture

Hope you like it. Appreciate your vote.