Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 : .net

Simple way to cache objects and collections for greater performance and scalability

omar — Sun, 01 Nov 2009 16:41:00 GMT

Caching of frequently used data greatly increases the scalability of your application since you can avoid repeated queries on database, file system or to webservices. When objects are cached, it can be retrieved from the cache which is lot faster and more scalable than loading from database, file or web service. However, implementing caching is tricky and monotonous when you have to do it for many classes. Your data access layer gets a whole lot of code that deals with caching objects and collection, updating cache when objects change or get deleted, expire collections when a contained object changes or gets deleted and so on. The more code you write, the more maintenance overhead you add. Here I will show you how you can make the caching a lot easier using Linq to SQL and my library AspectF. It’s a library that helps you get rid of thousands of lines of repeated code from a medium sized project and eliminates plumbing (logging, error handling, retrying etc) type code completely.

Here’s an example how caching significantly improves the performance and scalabitlity of applications. Dropthings – my open source Web 2.0 AJAX portal, without caching can only serve about 11 request/sec with 10 concurrent users on a dual core 64 bit PC. Here data is loaded from database as well as from external sources. Avg page response time is 1.44 sec.

After implementing caching, it became significantly faster, around 32 requests/sec. Page load time decreased significantly as well to 0.41 sec only. During the load test, CPU utilization was around 60%.

It shows clearly the significant difference it can make to your application. If you are suffering from poor page load performance and high CPU or disk activity on your database and application server, then caching Top 5 most frequently used objects in your application will solve that problem right away. It’s a quick win to make your application a lot faster than doing complex re-engineering in your application.

Common approaches to caching objects and collections

Sometimes the caching can be simple, for example caching a single object which does not belong to a collection and does not have child collections that are cached separately. In such case, you write simple code like this:

Is the object being requested already in cache?
- Yes, then serve it from cache.
- No, then load it from database and then cache it.

On the other hand, when you are dealing with cached collection where each item in the collection is also cached separately, then the caching logic is not so simple. For example, say you have cached a User collection. But each User object is also cached separately because you need to load individual User objects frequently. Then the caching logic gets more complicated:

Is the collection being requested already in cache?
- Yes. Get the collection. For each object in the collection:
  - Is that object individually available in cache?
    - Yes, get the individual object from cache. Update it in the collection.
    - No, discard the whole collection from cache. Go to next step:
- No. Load the collection from source (eg database) and cache each item in the collection separately. Then cache the collection.

You might be thinking why do we need to read each individual item from cache and why do we need to cache each item in collection separarely when the whole collection is already in cache? There are two scenarios you need to address when you cache a collection and individual items in that collection are also cached separately:

An individual item has been updated and the updated item is in cache. But the collection, which contains all those individual items, has not been refreshed. So, if you get the collection from cache and return as it is, you will get stale individual items inside that collection. This is why each item needs to be retrieved from cache separately.
An item in the collection may have been force expired in cache. For ex, something changed in the object or the object has been deleted. So, you expired it in cache so that on next retrieval it comes from database. If you load the collection from cache only, then the collection will contain the stale object.

If you are doing it the conventional way, you will be writing a lot of repeated code in your data access layer. For example, say you are loading a Page collection that belongs to a user. If you want to cache the collection of Page for a user as well as cache individual Page objects so that each Page can be retrieved from Cache directly. Then you need to write code like this:

public List<Page> GetPagesOfUserOldSchool(Guid userGuid)
{
    ICache cache = Services.Get<ICache>();
    bool isCacheStale = false;
    string cacheKey = CacheSetup.CacheKeys.PagesOfUser(userGuid);
    var cachedPages = cache.Get(cacheKey) as List<Page>;
    if (cachedPages != null)
    {
        var resultantPages = new List<Page>();
        // If each item in the collection is no longer in cache, invalidate the collection
        // and load again.
        foreach (Page cachedPage in cachedPages)
        {
            var individualPageInCache = cache.Get(CacheSetup.CacheKeys.PageId(cachedPage.ID)) as Page;
            if (null == individualPageInCache)
            {
                // Some item is missing in cache. So, the collection is stale. 
                isCacheStale = true;
            }
            else
            {
                resultantPages.Add(individualPageInCache);
            }
        }

        cachedPages = resultantPages;
    }

    if (isCacheStale)
    {
        // Collection not cached. Need to load collection from database and then cache it.
        var pagesOfUser = _database.GetList<Page, Guid>(...);
        pagesOfUser.Each(page =>
        {
            page.Detach();
            cache.Add(CacheSetup.CacheKeys.PageId(page.ID), page);
        });
        cache.Add(cacheKey, pagesOfUser);
        return pagesOfUser;
    }
    else
    {
        return cachedPages;
    }
}

Imagine writing this kind of code over and over again for each and every entity that you want to cache. This becomes a maintenace nightmare as your project grows.

Here’s how you could do it using AspectF:

public List<Page> GetPagesOfUser(Guid userGuid)
{
    return AspectF.Define
        .CacheList<Page, List<Page>>(Services.Get<ICache>(), 
                     CacheSetup.CacheKeys.PagesOfUser(userGuid), 
                     page => CacheSetup.CacheKeys.PageId(page.ID))
        .Return<List<Page>>(() => 
            _database.GetList<Page, Guid>(...).Select(p => p.Detach()).ToList());
}

Instead of 42 lines of code, you can do it in 5 lines!

Read my article Simple way to cache objects and collections for greater performance and scalability on CodeProject and learn:

Caching Linq to SQL entities
Handling update and delete scenarios
Expiring dependent objects and collections in cache
Handling objects that’s cached with multiple keys
Avoid database query optimizations when you cache sets of data

Enjoy. Don’t forget to vote for me!

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

omar — Wed, 08 Apr 2009 18:16:22 GMT

Dropthings – my open source Web 2.0 Ajax Portal has gone through a technology overhauling. Previously it was built using ASP.NET AJAX, a little bit of Workflow Foundation and Linq to SQL. Now Dropthings boasts full jQuery front-end combined with ASP.NET AJAX UpdatePanel, Silverlight widget, full Workflow Foundation implementation on the business layer, 100% Linq to SQL Compiled Queries on the data access layer, Dependency Injection and Inversion of Control (IoC) using Microsoft Enterprise Library 4.1 and Unity. It also has a ASP.NET AJAX Web Test framework that makes it real easy to write Web Tests that simulates real user actions on AJAX web pages. This article will walk you through the challenges in getting these new technologies to work in an ASP.NET website and how performance, scalability, extensibility and maintainability has significantly improved by the new technologies. Dropthings has been licensed for commercial use by prominent companies including BT Business, Intel, Microsoft IS, Denmark Government portal for Citizens; Startups like Limead and many more. So, this is serious stuff! There’s a very cool open source implementation of Dropthings framework available at National University of Singapore portal.

Visit: http://dropthings.omaralzabir.com

I have published a new article on this on CodeProject:

http://www.codeproject.com/KB/ajax/Web20Portal.aspx

Get the source code

Latest source code is hosted at Google code:

http://code.google.com/p/dropthings

There’s a CodePlex site for documentation and issue tracking:

http://www.codeplex.com/dropthings

You will need Visual Studio 2008 Team Suite with Service Pack 1 and Silverlight 2 SDK in order to run all the projects. If you have only Visual Studio 2008 Professional, then you will have to remove the Dropthings.Test project.

New features introduced

Dropthings new release has the following features:

Template users – you can define a user who’s pages and widgets are used as a template for new users. Whatever you put in that template user’s pages, it will be copied for every new user. Thus this is an easier way to define the default pages and widgets for new users. Similarly you can do the same for a registered user. The template users can be defined in the web.config.
Widget-to-Widget communication – Widgets can send message to each other. Widgets can subscribe to an Event Broker and exchange messages using a Pub-Sub pattern.
WidgetZone – you can create any number of zones in any shape on the page. You can have widgets laid in horizontal layout, you can have zones on different places on the page and so on. With this zone model, you are no longer limited to the Page-Column model where you could only have N vertical columns.
Role based widgets – now widgets are mapped to roles so that you can allow different users to see different widget list using ManageWidgetPersmission.aspx.
Role based page setup – you can define page setup for different roles. For ex, Managers see different pages and widgets than Employees.
Widget maximize – you can maximize a widget to take full screen. Handy for widgets with lots of content.
Free form resize – you can freely resize widgets vertically.
Silverlight Widgets – You can now make widgets in Silverlight!

Why the technology overhauling

Performance, Scalability, Maintainability and Extensibility – four key reasons for the overhauling. Each new technology solved one of more of these problems.

First, jQuery was used to replace my personal hand-coded large amount of Javascript code that offered the client side drag & drop and other UI effects. jQuery already has a rich set of library for Drag & Drop, Animations, Event handling, cross browser javascript framework and so on. So, using jQuery means opening the door to thousands of jQuery plugins to be offered on Dropthings. This made Dropthings highly extensible on the client side. Moreover, jQuery is very light. Unlike AJAX Control Toolkit jumbo sized framework and heavy control extenders, jQuery is very lean. So, total javascript size decreased significantly resulting in improved page load time. In total, the jQuery framework, AJAX basic framework, all my stuffs are total 395KB, sweet! Performance is key; it makes or breaks a product.

Secondly, Linq to SQL queries are replaced with Compiled Queries. Dropthings did not survive a load test when regular lambda expressions were used to query database. I could only reach up to 12 Req/Sec using 20 concurrent users without burning up web server CPU on a Quad Core DELL server.

Thirdly, Workflow Foundation is used to build operations that require multiple Data Access Classes to perform together in a single transaction. Instead of writing large functions with many if…else conditions, for…loops, it’s better to write them in a Workflow because you can visually see the flow of execution and you can reuse Activities among different Workflows. Best of all, architects can design workflows and developers can fill-in code inside Activities. So, I could design a complex operations in a workflow without writing the real code inside Activities and then ask someone else to implement each Activity. It is like handing over a design document to developers to implement each unit module, only that here everything is strongly typed and verified by compiler. If you strictly follow Single Responsibility Principle for your Activities, which is a smart way of saying one Activity does only one and very simple task, you end up with a highly reusable and maintainable business layer and a very clean code that’s easily extensible.

Fourthly, Unity Dependency Injection (DI) framework is used to pave the path for unit testing and dependency injection. It offers Inversion of Control (IoC), which enables testing individual classes in isolation. Moreover, it has a handy feature to control lifetime of objects. Instead of creating instance of commonly used classes several times within the same request, you can make instances thread level, which means only one instance is created per thread and subsequent calls reuse the same instance. Are these going over your head? No worries, continue reading, I will explain later on.

Fifthly, enabling API for Silverlight widgets allows more interactive widgets to be built using Silverlight. HTML and Javascripts still have limitations on smooth graphics and continuous transmission of data from web server. Silverlight solves all of these problems.

Read the article for details on how all these improvements were done and how all these hot techs play together in a very useful open source project for enterprises.

http://www.codeproject.com/KB/ajax/Web20Portal.aspx

Don’t forget to vote for me if you like it.

Memory Leak with delegates and workflow foundation

omar — Sat, 14 Mar 2009 08:35:53 GMT

Recently after Load Testing my open source project Dropthings, I encountered a lot of memory leak. I found lots of Workflow Instances and Linq Entities were left in memory and never collected. After profiling the web application using .NET Memory Profiler, it showed the real picture:

It shows you that instances of the several types are being created but not being removed. You see the “New” column has positive value, but the “Remove” column has 0. That means new instances are being created, but not removed. Basically the way you do Memory Profiling is, you take two snapshots. Say you take one snapshot when you first visit your website. Then you do some action on the website that results in allocation of objects. Then you take another snapshot. When you compare both snapshots, you can see how many instances of classes were created between these two snapshots and how many were removed. If they are not equal, then you have leak. Generally in web application many objects are created on every page hit and the end of the request, all those objects are supposed to be released. If they are not released, then we have a problem. But that’s the scenario for desktop applications because in a desktop application, objects can remain in memory until app is closed. But you should know best from the code which objects were supposed to go out of scope and get released.

For beginners, leak means objects are being allocated but not being freed because someone is holding reference to the objects. When objects leak, they remain in memory forever, until the process (or app domain) is closed. So, if you have a leaky website, your website is continuously taking up memory until it runs out of memory on the web server and thus crash. So, memory leak is a bad – it prevents you from running your product for long duration and requires frequent restart of app pool.

So, the above screenshot shows Workflow and Linq related classes are not being removed, and thus leaking. This means somewhere workflow instances are not being released and thus all workflow related objects are remaining. You can see the number is same 48 for all workflow related objects. This is a good indication that, almost every instance of workflow is leaked because there were total 48 workflows created and ran. Moreover it indicates we have a leak from a top Workflow instance level, not in some specific Activity or somewhere deep in the code.

As the workflows use Linq stuff, they held reference to the Linq stuffs and thus the Linq stuffs leaked as well. Sometimes you might be looking for why A is leaking. But you actually end up finding that since B was holding reference to A and B was leaking and thus A was leaking as well. This is sometimes tricky to figure out and you spend a lot of time looking at the wrong direction.

Now let me show you the buggy code:

ManualWorkflowSchedulerService manualScheduler = 
  workflowRuntime.GetService<ManualWorkflowSchedulerService>();

WorkflowInstance instance = workflowRuntime.CreateWorkflow(workflowType, properties);
instance.Start();

EventHandler<WorkflowCompletedEventArgs> completedHandler = null;
completedHandler = delegate(object o, WorkflowCompletedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instance.InstanceId) // 1. instance
    {
        workflowRuntime.WorkflowCompleted -= completedHandler; // 2. terminatedhandler

        // copy the output parameters in the specified properties dictionary
        Dictionary<string,object>.Enumerator enumerator = 
            e.OutputParameters.GetEnumerator();
        while( enumerator.MoveNext() )
        {
            KeyValuePair<string,object> pair = enumerator.Current;
            if( properties.ContainsKey(pair.Key) )
            {
                properties[pair.Key] = pair.Value;
            }
        }
    }
};

Exception x  = null;
EventHandler<WorkflowTerminatedEventArgs> terminatedHandler = null;
terminatedHandler = delegate(object o, WorkflowTerminatedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instance.InstanceId) // 3. instance
    {
        workflowRuntime.WorkflowTerminated -= terminatedHandler; // 4. completeHandler
        Debug.WriteLine( e.Exception );

        x = e.Exception;
    }
};

workflowRuntime.WorkflowCompleted += completedHandler;
workflowRuntime.WorkflowTerminated += terminatedHandler;

manualScheduler.RunWorkflow(instance.InstanceId);

Can you spot the code where it leaked?

I have numbered the lines in comment where the leak is happening. Here the delegate is acting like a closure and those who are from Javascript background know closure is evil. They leak memory unless very carefully written. Here the delegate keeps a reference to the instance object. So, if somehow delegate is not released, the instance will remain in memory forever and thus leak. Now can you find a situation when the delegate will not be released?

Say the workflow completed. It will fire the completeHandler. But the completeHandler will not release the terminateHandler. Thus the terminateHandler remains in memory and it also holds reference to the instance. So, we have a leaky delegate leaking whatever it is holding onto outside it’s scope. Here the only thing outside the scope if the instance, which it is tried to access from the parent function.

Since the workflow instance is not released, all the properties the workflow and all the activities inside it are holding onto remains in memory. Most of the workflows and activities expose public properties which are Linq Entities. Thus the Linq Entities remain in memory. Now Linq Entities keep a reference to the DataContext from where it is produced. Thus we have DataContext remaining in memory. Moreover, DataContext keeps reference to many internal objects and metadata cacahe, so they remain in memory as well.

So, the correct code is:

ManualWorkflowSchedulerService manualScheduler = 
    workflowRuntime.GetService<ManualWorkflowSchedulerService>();

WorkflowInstance instance = workflowRuntime.CreateWorkflow(workflowType, properties);
instance.Start();
var instanceId = instance.InstanceId;

EventHandler<WorkflowCompletedEventArgs> completedHandler = null;
completedHandler = delegate(object o, WorkflowCompletedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instanceId) // 1. instanceId is a Guid
    {
        // copy the output parameters in the specified properties dictionary
        Dictionary<string,object>.Enumerator enumerator = 
                e.OutputParameters.GetEnumerator();
        while( enumerator.MoveNext() )
        {
            KeyValuePair<string,object> pair = enumerator.Current;
            if( properties.ContainsKey(pair.Key) )
            {
                properties[pair.Key] = pair.Value;
            }
        }
    }
};

Exception x  = null;
EventHandler<WorkflowTerminatedEventArgs> terminatedHandler = null;
terminatedHandler = delegate(object o, WorkflowTerminatedEventArgs e)
{
    if (e.WorkflowInstance.InstanceId == instanceId) // 2. instanceId is a Guid
    {
        x = e.Exception;
        Debug.WriteLine(e.Exception);
    }
};

workflowRuntime.WorkflowCompleted += completedHandler;
workflowRuntime.WorkflowTerminated += terminatedHandler;

manualScheduler.RunWorkflow(instance.InstanceId);
// 3. Both delegates are now released
workflowRuntime.WorkflowTerminated -= terminatedHandler;
workflowRuntime.WorkflowCompleted -= completedHandler;

There are two changes – in both delegates, the instanceId variable is passed, instead of the instance. Since instanceId is a Guid, which is a struct type data type, not a class, there’s no issue of referencing. Structs are copied, not referenced. So, they don’t leak memory. Secondly, both delegates are released at the end of the workflow execution, thus releasing both references.

In Dropthings, I am using the famous CallWorkflow Activity by John Flanders, which is widely used to execute one Workflow from another synchronously. There’s a CallWorkflowService class which is responsible for synchronously executing another workflow and that has similar memory leak problem. The original code of the service is as following:

public class CallWorkflowService : WorkflowRuntimeService
{
    #region Methods

    public void StartWorkflow(Type workflowType,Dictionary<string,object> inparms, 
           Guid caller,IComparable qn)
    {
        WorkflowRuntime wr = this.Runtime;
        WorkflowInstance wi = wr.CreateWorkflow(workflowType,inparms);
        wi.Start();
        ManualWorkflowSchedulerService ss = 
             wr.GetService<ManualWorkflowSchedulerService>();
        if (ss != null)
            ss.RunWorkflow(wi.InstanceId);
        EventHandler<WorkflowCompletedEventArgs> d  = null;
        d = delegate(object o, WorkflowCompletedEventArgs e)
        {
            if (e.WorkflowInstance.InstanceId ==wi.InstanceId)
            {
                wr.WorkflowCompleted -= d;
                WorkflowInstance c = wr.GetWorkflow(caller);
                c.EnqueueItem(qn, e.OutputParameters, null, null);
            }
        };
        EventHandler<WorkflowTerminatedEventArgs> te = null;
        te = delegate(object o, WorkflowTerminatedEventArgs e)
        {
            if (e.WorkflowInstance.InstanceId == wi.InstanceId)
            {
                wr.WorkflowTerminated -= te;
                WorkflowInstance c = wr.GetWorkflow(caller);
                c.EnqueueItem(qn, new Exception("Called Workflow Terminated", 
                  e.Exception), null, null);
            }
        };
        wr.WorkflowCompleted += d;
        wr.WorkflowTerminated += te;
    }

    #endregion Methods
}

As you see, it has that same delegate holding reference to instance object problem. Moreover, there’s some queue stuff there, which requires the caller and qn parameter passed to the StartWorkflow function. So, not a straight forward fix.

I tried to rewrite the whole CallWorkflowService so that it does not require two delegates to be created per Workflow. Then I took the delegates out. Thus there’s no chance of closure holding reference to unwanted objects. The result looks like this:

public class CallWorkflowService : WorkflowRuntimeService
{
    #region Fields

    private EventHandler<WorkflowCompletedEventArgs> _CompletedHandler = null;
    private EventHandler<WorkflowTerminatedEventArgs> _TerminatedHandler = null;
    private Dictionary<Guid, WorkflowInfo> _WorkflowQueue = 
       new Dictionary<Guid, WorkflowInfo>();

    #endregion Fields

    #region Methods

    public void StartWorkflow(Type workflowType,Dictionary<string,object> inparms,
        Guid caller,IComparable qn)
    {
        WorkflowRuntime wr = this.Runtime;
        WorkflowInstance wi = wr.CreateWorkflow(workflowType,inparms);
        wi.Start();

        var instanceId = wi.InstanceId;
        _WorkflowQueue[instanceId] = new WorkflowInfo { Caller = caller, qn = qn };

        ManualWorkflowSchedulerService ss = 
           wr.GetService<ManualWorkflowSchedulerService>();
        if (ss != null)
            ss.RunWorkflow(wi.InstanceId);
    }

    protected override void OnStarted()
    {
        base.OnStarted();

        if (null == _CompletedHandler)
        {
            _CompletedHandler = delegate(object o, WorkflowCompletedEventArgs e)
            {
                var instanceId = e.WorkflowInstance.InstanceId;
                if (_WorkflowQueue.ContainsKey(instanceId))
                {
                    WorkflowInfo wf = _WorkflowQueue[instanceId];
                    WorkflowInstance c = this.Runtime.GetWorkflow(wf.Caller);
                    c.EnqueueItem(wf.qn, e.OutputParameters, null, null);
                    _WorkflowQueue.Remove(instanceId);
                }
            };
            this.Runtime.WorkflowCompleted += _CompletedHandler;
        }

        if (null == _TerminatedHandler)
        {
            _TerminatedHandler = delegate(object o, WorkflowTerminatedEventArgs e)
            {
                var instanceId = e.WorkflowInstance.InstanceId;
                if (_WorkflowQueue.ContainsKey(instanceId))
                {
                    WorkflowInfo wf = _WorkflowQueue[instanceId];
                    WorkflowInstance c = this.Runtime.GetWorkflow(wf.Caller);
                    c.EnqueueItem(wf.qn, 
                      new Exception("Called Workflow Terminated", e.Exception), 
                      null, null);
                    _WorkflowQueue.Remove(instanceId);
                }
            };

            this.Runtime.WorkflowTerminated += _TerminatedHandler;
        }
    }

    protected override void OnStopped()
    {
        _WorkflowQueue.Clear();

        base.OnStopped();
    }

    #endregion Methods

    #region Nested Types

    private struct WorkflowInfo
    {
        #region Fields

        public Guid Caller;
        public IComparable qn;

        #endregion Fields
    }

    #endregion Nested Types
}

After fixing the problem, another Memory Profile result showed the leak is gone:

As you see, the numbers vary, which means there’s no consistent leak. Moreover, looking at the types that remains in memory, they look more like metadata than instances of classes. So, they are basically cached instances of metadata, not instances allocated during workflow execution which are supposed to be freed. So, we solved the memory leak!

Now you know how to write anonymous delegates without leaking memory and how to run workflow without leaking them. Basically, the principle theory is – if you are referencing some outside object from an anonymous delegate, make sure that object is not holding reference to the delegate in some way, may be directly or may be via some child objects of its own. Because then you have a circular reference. If possible, do not try to access objects e.g. instance inside an anonymous delegate that is declared outside the delegate. Try accessing instrinsic data types like int, string, DateTime, Guid etc which are not reference type variables. So, instead of referencing to an object, you should declare local variables e.g. instanceId that gets the value of properties (e.g. instance.InstanceId) from the object and then use those local variables inside the anonymous delegate.

Optimize ASP.NET Membership Stored Procedures for greater speed and scalability

omar — Fri, 13 Mar 2009 13:23:00 GMT

Last year at Pageflakes, when we were getting millions of hits per day, we were having query timeout due to lock timeout and Transaction Deadlock errors. These locks were produced from aspnet_Users and aspnet_Membership tables. Since both of these tables are very high read (almost every request causes a read on these tables) and high write (every anonymous visit creates a row on aspnet_Users), there were just way too many locks created on these tables per second. SQL Counters showed thousands of locks per second being created. Moreover, we had queries that would select thousands of rows from these tables frequently and thus produced more locks for longer period, forcing other queries to timeout and thus throw errors on the website.

If you have read my last blog post, you know why such locks happen. Basically every table when it grows up to hold millions of records and becomes popular goes through this trouble. It’s just a part of scalability problem that is common to database. But we rarely take prevention about it in our early design.

The solution is simple, you should either have WITH (NOLOCK) or SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED before SELECT queries. Either of this will do. They tell SQL Server not to hold any lock on the table while it is reading the table. If some row is locked while the read is happening, it will just ignore that row. When you are reading a table thousand times per second, without these options, you are issuing lock on many places around the table thousand times per second. It not only makes read from table slower, but also so many lock prevents insert, update, delete from happening timely and thus queries timeout. If you have queries like “show the currently online users from last one hour based on LastActivityDate field”, that is going to issue such a wide lock that even other harmless select queries will timeout. And did I tell you that there’s no index on LastActivityDate on aspnet_Users table?

Now don’t blame yourself for not putting either of these options on your every stored proc and every dynamically generated SQL from the very first day. ASP.NET developers made the same mistake. You won’t see either of these used in any of the stored procs used by ASP.NET Membership. For example, the following stored proc gets called whenever you access Profile object:

ALTER PROCEDURE [dbo].[aspnet_Profile_GetProperties]
    @ApplicationName      nvarchar(256),
    @UserName             nvarchar(256),
    @CurrentTimeUtc       datetime
AS
BEGIN

    DECLARE @ApplicationId uniqueidentifier
    SELECT  @ApplicationId = NULL
    SELECT  @ApplicationId = ApplicationId FROM 
      dbo.aspnet_Applications WHERE LOWER(@ApplicationName) = LoweredApplicationName
    IF (@ApplicationId IS NULL)
        RETURN

    DECLARE @UserId uniqueidentifier
    DECLARE @LastActivityDate datetime
    SELECT  @UserId = NULL

    SELECT @UserId = UserId, @LastActivityDate = LastActivityDate
    FROM   dbo.aspnet_Users 
    WHERE  ApplicationId = @ApplicationId AND LoweredUserName = LOWER(@UserName)

    IF (@UserId IS NULL)
        RETURN
    SELECT TOP 1 PropertyNames, PropertyValuesString, PropertyValuesBinary
    FROM         dbo.aspnet_Profile
    WHERE        UserId = @UserId

    IF (@@ROWCOUNT > 0)
    BEGIN
        UPDATE dbo.aspnet_Users
        SET    LastActivityDate=@CurrentTimeUtc
        WHERE  UserId = @UserId
    END
END

There are two SELECT operations that hold lock on two very high read tables – aspnet_Users and aspnet_Profile. Moreover, there’s a nasty UPDATE statement. It tries to update the LastActivityDate of a user whenever you access Profile object for the first time within a http request.

This stored proc alone is enough to bring your site down. It did to us because we are using Profile Provider everywhere. This stored proc was called around 300 times/sec. We were having nightmarish slow performance on the website and many lock timeouts and transaction deadlocks. So, we added the transaction isolation level and we also modified the UPDATE statement to only perform an update when the LastActivityDate is over an hour. So, this means, the same user’s LastActivityDate won’t be updated if the user hits the site within the same hour.

So, after the modifications, the stored proc looked like this:

ALTER PROCEDURE [dbo].[aspnet_Profile_GetProperties]
    @ApplicationName      nvarchar(256),
    @UserName             nvarchar(256),
    @CurrentTimeUtc       datetime
AS
BEGIN
    -- 1. Please no more locks during reads
    SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;

    DECLARE @ApplicationId uniqueidentifier
    --SELECT  @ApplicationId = NULL
    --SELECT  @ApplicationId = ApplicationId FROM dbo.aspnet_Applications 
    WHERE LOWER(@ApplicationName) = LoweredApplicationName
    --IF (@ApplicationId IS NULL)
    --    RETURN
    
    -- 2. No more call to Application table. We have only one app dude!
    SET @ApplicationId = dbo.udfGetAppId()

    DECLARE @UserId uniqueidentifier
    DECLARE @LastActivityDate datetime
    SELECT  @UserId = NULL

    SELECT @UserId = UserId, @LastActivityDate = LastActivityDate
    FROM   dbo.aspnet_Users 
    WHERE  ApplicationId = @ApplicationId AND LoweredUserName = LOWER(@UserName)

    IF (@UserId IS NULL)
        RETURN
    SELECT TOP 1 PropertyNames, PropertyValuesString, PropertyValuesBinary
    FROM         dbo.aspnet_Profile
    WHERE        UserId = @UserId

    IF (@@ROWCOUNT > 0)
    BEGIN
        -- 3. Do not update the same user within an hour
        IF DateDiff(n, @LastActivityDate, @CurrentTimeUtc) > 60
        BEGIN
            -- 4. Use ROWLOCK to lock only a row since we know this query
            -- is highly selective
            UPDATE dbo.aspnet_Users WITH(ROWLOCK)
            SET    LastActivityDate=@CurrentTimeUtc
            WHERE  UserId = @UserId
        END
    END
END

The changes I made are numbered and commented. No need for further explanation. The only tricky thing here is, I have eliminate call to Application table just to get the ApplicationID from ApplicationName. Since there’s only one application in a database (ever heard of multiple applications storing their user separately on the same database and the same table?), we don’t need to look up the ApplicationID on every call to every Membership stored proc. We can just get the ID and hard code it in a function.

CREATE FUNCTION dbo.udfGetAppId()
RETURNS uniqueidentifier
WITH EXECUTE AS CALLER
AS
BEGIN
    RETURN CONVERT(uniqueidentifier, 'fd639154-299a-4a9d-b273-69dc28eb6388')
END;

This UDF returns the ApplicationID that I have hardcoded copying from the Application table. Thus it eliminates the need for quering on the Application table.

Similarly you should do the changes in all other stored procedures that belong to Membership Provider. All the stroc procs are missing proper locking, issues aggressive lock during update and too frequent updates than practical need. Most of them also try to resolve ApplicationID from ApplicationName, which is unnecessary when you have only one web application per database. Make these changes and enjoy lock contention free super performance from Membership Provider!

Linq to SQL solve Transaction deadlock and Query timeout problem using uncommitted reads

omar — Sat, 07 Mar 2009 07:30:52 GMT

When your database tables start accumulating thousands of rows and many users start working on the same table concurrently, SELECT queries on the tables start producing lock contentions and transaction deadlocks. This is a common problem in any high volume website. As soon as you start getting several concurrent users hitting your website that results in SELECT queries on some large table like aspnet_users table that are also being updated very frequently, you end up having one of these errors:

Transaction (Process ID ##) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

Or,

Timeout Expired. The Timeout Period Elapsed Prior To Completion Of The Operation Or The Server Is Not Responding.

The solution to these problems are – use proper index on the table and use transaction isolation level Read Uncommitted or WITH (NOLOCK) in your SELECT queries. So, if you had a query like this:

SELECT * FORM aspnet_users 
where ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

You should end up having any of the above errors under high load. There are two ways to solve this:

SET TRANSACTION LEVEL READ UNCOMMITTED;
SELECT * FROM aspnet_Users 
WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

Or use the WITH (NOLOCK):

SELECT * FROM aspnet_Users WITH (NOLOCK) 
WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

The reason for the errors are that since aspnet_users is a high read and high write table, during read, the table is partially locked and during write, it is also locked. So, when the locks overlap on each other from several queries and especially when there’s a query that’s trying to read a large number of rows and thus locking large number of rows, some of the queries either timeout or produce deadlocks.

Linq to Sql does not produce queries with the WITH (NOLOCK) option nor does it use READ UNCOMMITTED. So, if you are using Linq to SQL queries, you are going to end up with any of these problems on production pretty soon when your site becomes highly popular.

For example, here’s a very simple query:

using (var db = new DropthingsDataContext())
{
    var user = db.aspnet_Users.First();
    var pages = user.Pages.ToList();
}

DropthingsDataContext is a DataContext built from Dropthings database.

When you attach SQL Profiler, you get this:

You see none of the queries have READ UNCOMMITTED or WITH (NOLOCK).

The fix is to do this:

using (var db = new DropthingsDataContext2())
{
    db.Connection.Open();
    db.ExecuteCommand("SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;");

    var user = db.aspnet_Users.First();
    var pages = user.Pages.ToList();
}

This will result in the following profiler output

As you see, both queries execute within the same connection and the isolation level is set before the queries execute. So, both queries enjoy the isolation level.

Now there’s a catch, the connection does not close. This seems to be a bug in the DataContext that when it is disposed, it does not dispose the connection it is holding onto.

In order to solve this, I have made a child class of the DropthingsDataContext named DropthingsDataContext2 which overrides the Dispose method and closes the connection.

   class DropthingsDataContext2 : DropthingsDataContext, IDisposable
    {
        public new void Dispose()
        {
            if (base.Connection != null)
                if (base.Connection.State != System.Data.ConnectionState.Closed)
                {
                    base.Connection.Close();
                    base.Connection.Dispose();
                }

            base.Dispose();            
        }
    }

This solved the connection problem.

There you have it, no more transaction deadlock or lock contention from Linq to SQL queries. But remember, this is only to eliminate such problems when your database already has the right indexes. If you do not have the proper index, then you will end up having lock contention and query timeouts anyway.

There’s one more catch, READ UNCOMMITTED will return rows from transactions that have not completed yet. So, you might be reading rows from transactions that will rollback. Since that’s generally an exceptional scenario, you are more or less safe with uncommitted read, but not for financial applications where transaction rollback is a common scenario. In such case, go for committed read or repeatable read.

There’s another way you can achieve the same, which seems to work, that is using .NET Transactions. Here’s the code snippet:

using (var transaction = new TransactionScope(
    TransactionScopeOption.RequiresNew,
    new TransactionOptions()
    {
        IsolationLevel = IsolationLevel.ReadUncommitted,
        Timeout = TimeSpan.FromSeconds(30)
    }))
{
    using (var db = new DropthingsDataContext())
    {
        var user = db.aspnet_Users.First();
        var pages = user.Pages.ToList();

        transaction.Complete();
    }
}

Profiler shows a transaction begins and ends:

The downside is it wraps your calls in a transaction. So, you are unnecessarily creating transactions even for SELECT operations. When you do this hundred times per second on a web application, it’s a significant over head.

Some really good examples of deadlocks are given in this article:

http://www.code-magazine.com/article.aspx?quickid=0309101&page=2

I highly recommend it.

Strongly typed workflow input and output arguments

omar — Sat, 27 Dec 2008 15:05:45 GMT

When you run a Workflow using Workflow Foundation, you pass arguments to the workflow in a Dictionary form where the type of Dictionary is Dictionary<string, object>. This means you miss the strong typing features of .NET languages. You have to know what arguments the workflow expects by looking at the Workflow public properties. Moreover, there’s no way to make arguments required. You pass parameter, expect it to run, if it throws exception, you pass more arguments, hope it works now. Similarly, if you are running workflow synchronously using ManualWorkflowSchedulerService, you expect return arguments from the Workflow immediately, but there again, you have to rely on the Dictionary key and value pair. No strong typing there as well.

In order to solve this, so that you could pass Workflow arguments as strongly typed classes, you can establish a format that every Workflow has only two arguments named "Request” and “Response” and none other. Whatever needs to be passed to the Workflow and expected out of it, must be passed via Request and must be expected via Response properties. Now the type of these arguments can be workflow specific, it can be any class with one or more parameters. This way, you could write code like this:

The advantages of these strongly typed approach are:

Compile time validation of input parameters passed to workflow. No risk of passing unexpected object in Dictionary’s object type value.
Enforce required values by creating Request objects with non-default constructor.
Establish a fixed contract for Workflow input and output via the strongly typed Request and Response classes or interfaces.
Validate input arguments for the Workflow directly from the Request class, without going through the overhead of running a workflow.

If we follow this approach, we create workflows with only two DependencyProperty, one for Request and one for Response. Showing you an example from my open source project Dropthings, which uses Workflow for the entire Business Layer. Below you see the Workflow that executes when a new user visits Dropthings.com, creates a new user and setups all the pages and widgets for the user. It has only two Dependency property – Request and Response.

The Request parameters is of type IUserVisitWorkflowRequest. So, you can pass any class as Request argument that implements the interface.

Here I have used fancy inheritance to create Request object hierarchy. You don’t need to do that. Just remember, you can pass any class. You don’t even need to use interface for Request parameter. It can be a class directly. I use all these interfaces in order to facilitate Dependency Inversion.

Similarly, the Response object is also a class.

The Response returns quite some properties. So, it’s kinda handy to wrap them all in one property.

So, there you have it, strongly typed Workflow arguments. You can attach properties of the Request object to any activity directly form the designer:

There’s really no compromise to make in this approach. Everything works as before.

In order to make workflow execution simpler, I use a helper method like the following, that takes the Request and Response object and creates the Dictionary for me. This Dictionary always contains one “Request” and one “Response” entry.

This way, I can run Workflow in strongly typed fashion:

Here I can specify the Request, Response and Workflow type using strong typing. This way I get strongly typed return object as well as pass strongly type Request object. There’s no dictionary building, no risky string key and object type value passing. You can ignore the ObjectContainer.Resolve() stuff, because that’s just returning me an existing reference of WorkflowRuntime.

Hope you like this approach.

Linq to SQL: Delete an entity using Primary Key only

omar — Thu, 30 Oct 2008 00:27:00 GMT

Linq to Sql does not come with a function like .Delete(ID) which allows you to delete an entity using it’s primary key. You have to first get the object that you want to delete and then call .DeleteOnSubmit(obj) to queue it for delete. Then you have to call DataContext.SubmitChanges() to play the delete queries on database. So, how to delete object without getting them from database and avoid database roundtrip?

You can call this function using DeleteByPK<Employee, int>(10, dataContext);

First type is the entity type and second one is the type of the primary key. If your object’s primary key is a Guid field, specify Guid instead of int.

How it works:

It figures out the table name and the primary key field name from the entity
Then it uses the table name and primary key field name to build a DELETE query

Figuring out the table name and primary key field name is a bit hard. There’s some reflection involved. The GetTableDef<TSource>() returns the table name and primary key field name for an entity.

Every Linq Entity class is decorated with a Table attribute that has the table name:

Then the primary key field is decorated with a Column attribute with IsPrimaryKey = true.

So, using reflection we can figure out the table name and the primary key property and the field name.

Here’s the code that does it:

Before you scream “Reflection is SLOW!!!!” the definition is cached. So, reflection is used only once per appDomain per entity. Subsequent call is just a dictionary lookup away, which is as fast as it can get.

You can also delete a collection of object without ever getting any one of them. The the following function to delete a whole bunch of objects:

The code is available here:

http://code.msdn.microsoft.com/DeleteEntitiesLinq

Solving common problems with Compiled Queries in Linq to Sql for high demand ASP.NET websites

omar — Mon, 27 Oct 2008 13:31:00 GMT

If you are using Linq to SQL, instead of writing regular Linq Queries, you should be using Compiled Queries. if you are building an ASP.NET web application that’s going to get thousands of hits per hour, the execution overhead of Linq queries is going to consume too much CPU and make your site slow. There’s a runtime cost associated with each and every Linq Query you write. The queries are parsed and converted to a nice SQL Statement on *every* hit. It’s not done at compile time because there’s no way to figure out what you might be sending as the parameters in the queries during runtime. So, if you have common Linq to Sql statements like the following one throughout your growing web application, you are soon going to have scalability nightmares:

var query = from widget in dc.Widgets
                where widget.ID == id && widget.PageID == pageId
                select widget;

var widget = query.SingleOrDefault();

There’s a nice blog post by JD Conley that shows how evil Linq to Sql queries are:

You see how many times SqlVisitor.Visit is called to convert a Linq Query to its SQL representation? The runtime cost to convert a Linq query to its SQL Command representation is just way too high.

Rico Mariani has a very informative performance comparison of regular Linq queries vs Compiled Linq queries performance:

Compiled Query wins on every case.

So, now you know about the benefits of compiled queries. If you are building ASP.NET web application that is going to get high traffic and you have a lot of Linq to Sql queries throughout your project, you have to go for compiled queries. Compiled Queries are built for this specific scenario.

In this article, I will show you some steps to convert regular Linq to Sql queries to their Compiled representation and how to avoid the dreaded exception “Compiled queries across DataContexts with different LoadOptions not supported.”

Here are some step by step instruction on converting a Linq to Sql query to its compiled form:

First we need to find out all the external decision factors in a query. It mostly means parameters in the WHERE clause. Say, we are trying to get a user from aspnet_users table using Username and Application ID:

Here, we have two external decision factor – one is the Username and another is the Application ID. So, first think this way, if you were to wrap this query in a function that will just return this query as it is, what would you do? You would create a function that takes the DataContext (dc named here), then two parameters named userName and applicationID, right?

So, be it. We create one function that returns just this query:

Next step is to replace this function with a Func<> representation that returns the query. This is the hard part. If you haven’t dealt with Func<> and Lambda expression before, then I suggest you read this and this and then continue.

So, here’s the delegate representation of the above function:

Couple of things to note here. I have declared the delegate as static readonly because a compiled query is declared only once and reused by all threads. If you don’t declare Compiled Queries as static, then you don’t get the performance gain because compiling queries everytime when needed is even worse than regular Linq queries.

Then there’s the complex Func<DropthingsDataContext, string, Guid, IQueryable<aspnet_User>> thing. Basically the generic Func<> is declared to have three parameters from the GetQuery function and a return type of IQueryable<aspnet_User>. Here the parameter types are specified so that the delegate is created strongly typed. Func<> allows up to 4 parameters and 1 return type.

Next comes the real business, compiling the query. Now that we have the query in delegate form, we can pass this to CompiledQuery.Compile function which compiles the delegate and returns a handle to us. Instead of directly assigning the lambda expression to the func, we will pass the expression through the CompiledQuery.Compile function.

Here’s where head starts to spin. This is so hard to read and maintain. Bear with me. I just wrapped the lambda expression on the right side inside the CompiledQuery.Compile function. Basically that’s the only change. Also, when you call CompiledQuery.Compile<>, the generic types must match and be in exactly the same order as the Func<> declaration.

Fortunately, calling a compiled query is as simple as calling a function:

There you have it, a lot faster Linq Query execution. The hard work of converting all your queries into Compiled Query pays off when you see the performance difference.

Now, there are some challenges to Compiled Queries. Most common one is, what do you do when you have more than 4 parameters to supply to a Compiled Query? You can’t declare a Func<> with more than 4 types. Solution is to use a struct to encapsulate all the parameters. Here’s an example:

Calling the query is quite simple:

Now to the dreaded challenge of using LoadOptions with Compiled Query. You will notice that the following code results in an exception:

The above DataLoadOption runs perfectly when you use regular Linq Queries. But it does not work with compiled queries. When you run this code and the query hits the second time, it produces an exception:

Compiled queries across DataContexts with different LoadOptions not supported

A compiled query remembers the DataLoadOption once its called. It does not allow executing the same compiled query with a different DataLoadOption again. Although you are creating the same DataLoadOption with the same LoadWith<> calls, it still produces exception because it remembers the exact instance that was used when the compiled query was called for the first time. Since next call creates a new instance of DataLoadOptions, it does not match and the exception is thrown. You can read details about the problem in this forum post.

The solution is to use a static DataLoadOption. You cannot create a local DataLoadOption instance and use in compiled queries. It needs to be static. Here’s how you can do it:

Basically the idea is to construct a static instance of DataLoadOptions using a static function. As writing function for every single DataLoadOptions combination is painful, I created a static delegate here and executed it right on the declaration line. This is in interesting way to declare a variable that requires more than one statement to prepare it.

Using this option is very simple:

Now you can use DataLoadOptions with compiled queries.

Deploy ASP.NET MVC on IIS 6, solve 404, compression and performance problems

omar — Mon, 30 Jun 2008 08:12:22 GMT

There are several problems with ASP.NET MVC application when deployed on IIS 6.0:

Extensionless URLs give 404 unless some URL Rewrite module is used or wildcard mapping is enabled
IIS 6.0 built-in compression does not work for dynamic requests. As a result, ASP.NET pages are served uncompressed resulting in poor site load speed.
Mapping wildcard extension to ASP.NET introduces the following problems:
- Slow performance as all static files get handled by ASP.NET and ASP.NET reads the file from file system on every call
- Expires headers doesn't work for static content as IIS does not serve them anymore. Learn about benefits of expires header from here. ASP.NET serves a fixed expires header that makes content expire in a day.
- Cache-Control header does not produce max-age properly and thus caching does not work as expected. Learn about caching best practices from here.
After deploying on a domain as the root site, the homepage produces HTTP 404.

Problem 1: Visiting your website's homepage gives 404 when hosted on a domain

You have done the wildcard mapping, mapped .mvc extention to ASP.NET ISAPI handler, written the route mapping for Default.aspx or default.aspx (lowercase), but still when you visit your homepage after deployment, you get:

You will find people banging their heads on the wall here:

Solution is to capture hits going to "/" and then rewrite it to Default.aspx:

You can apply this approach to any URL that ASP.NET MVC is not handling for you and it should handle. Just see the URL reported on the 404 error page and then rewrite it to a proper URL.

Problem 2: IIS 6 compression is no longer working after wildcard mapping

When you enable wildcard mapping, IIS 6 compression no longer works for extensionless URL because IIS 6 does not see any extension which is defined in IIS Metabase. You can learn about IIS 6 compression feature and how to configure it properly from my earlier post.

Solution is to use an HttpModule to do the compression for dynamic requests.

Problem 3: ASP.NET ISAPI does not cache Static Files

When ASP.NET's DefaultHttpHandler serves static files, it does not cache the files in-memory or in ASP.NET cache. As a result, every hit to static file results in a File read. Below is the decompiled code in DefaultHttpHandler when it handles a static file. As you see here, it makes a file read on every hit and it only set the expiration to one day in future. Moreover, it generates ETag for each file based on file's modified date. For best caching efficiency, we need to get rid of that ETag, produce an expiry date on far future (like 30 days), and produce Cache-Control header which offers better control over caching.

So, we need to write a custom static file handler that will cache small files like images, Javascripts, CSS, HTML and so on in ASP.NET cache and serve the files directly from cache instead of hitting the disk. Here are the steps:

Install an HttpModule that installs a Compression Stream on Response.Filter so that anything written on Response gets compressed. This serves dynamic requests.
Replace ASP.NET's DefaultHttpHandler that listens on *.* for static files.
Write our own Http Handler that will deliver compressed response for static resources like Javascript, CSS, and HTML.

Here's the mapping in ASP.NET's web.config for the DefaultHttpHandler. You will have to replace this with your own handler in order to serve static files compressed and cached.

Solution 1: An Http Module to compress dynamic requests

First, you need to serve compressed responses that are served by the MvcHandler or ASP.NET's default Page Handler. The following HttpCompressionModule hooks on the Response.Filter and installs a GZipStream or DeflateStream on it so that whatever is written on the Response stream, it gets compressed.

These are formalities for a regular HttpModule. The real hook is installed as below:

Here you see we ignore requests that are handled by ASP.NET's DefaultHttpHandler and our own StaticFileHandler that you will see in next section. After that, it checks whether the request allows content to be compressed. Accept-Encoding header contains "gzip" or "deflate" or both when browser supports compressed content. So, when browser supports compressed content, a Response Filter is installed to compress the output.

Solution 2: An Http Module to compress and cache static file requests

Here's how the handler works:

Hooks on *.* so that all unhandled requests get served by the handler
Handles some specific files like js, css, html, graphics files. Anything else, it lets ASP.NET transmit it
The extensions it handles itself, it caches the file content so that subsequent requests are served from cache
It allows compression of some specific extensions like js, css, html. It does not compress graphics files or any other extension.

Let's start with the handler code:

Here you will find the extensions the handler handles and the extensions it compresses. You should only put files that are text files in the COMPRESS_FILE_TYPES.

Now start handling each request from BeginProcessRequest.

Here you decide the compression mode based on Accept-Encoding header. If browser does not support compression, do not perform any compression. Then check if the file being requested falls in one of the extensions that we support. If not, let ASP.NET handle it. You will see soon how.

Calculate the cache key based on the compression mode and the physical path of the file. This ensures that no matter what the URL requested, we have one cache entry for one physical file. Physical file path won't be different for the same file. Compression mode is used in the cache key because we need to store different copy of the file's content in ASP.NET cache based on Compression Mode. So, there will be one uncompressed version, a gzip compressed version and a deflate compressed version.

Next check if the file exits. If not, throw HTTP 404. Then create a memory stream that will hold the bytes for the file or the compressed content. Then read the file and write in the memory stream either directly or via a GZip or Deflate stream. Then cache the bytes in the memory stream and deliver to response. You will see the ReadFileData and CacheAndDeliver functions soon.

This function delivers content directly from ASP.NET cache. The code is simple, read from cache and write to the response.

When the content is not available in cache, read the file bytes and store in a memory stream either as it is or compressed based on what compression mode you decided before:

Here bytes are read in chunk in order to avoid large amount of memory allocation. You could read the whole file in one shot and store in a byte array same as the size of the file length. But I wanted to save memory allocation. Do a performance test to figure out if reading in 8K chunk is not the best approach for you.

Now you have the bytes to write to the response. Next step is to cache it and then deliver it.

Now the two functions that you have seen several times and have been wondering what they do. Here they are:

WriteResponse has no tricks, but ProduceResponseHeader has much wisdom in it. First it turns off response buffering so that ASP.NET does not store the written bytes in any internal buffer. This saves some memory allocation. Then it produces proper cache headers to cache the file in browser and proxy for 30 days, ensures proxy revalidate the file after the expiry date and also produces the Last-Modified date from the file's last write time in UTC.

How to use it

Get the HttpCompressionModule and StaticFileHandler from:

http://code.msdn.microsoft.com/fastmvc

Then install them in web.config. First you install the StaticFileHandler by removing the existing mapping for path="*" and then you install the HttpCompressionModule.

That's it! Enjoy a faster and more responsive ASP.NET MVC website deployed on IIS 6.0.

Share this post :

del.icio.us

digg

dotnetkicks

furl

live

spurl

technorati

yahoo

Fast, Streaming AJAX proxy - continuously download from cross domain

omar — Mon, 14 Apr 2008 07:29:00 GMT

Due to browser's prohibition on cross domain XMLHTTP call, all AJAX websites must have server side proxy to fetch content from external domain like Flickr or Digg. From client side javascript code, an XMLHTTP call goes to the server side proxy hosted on the same domain and then the proxy downloads the content from the external server and sends back to the browser. In general, all AJAX websites on the Internet that are showing content from external domains are following this proxy approach except some rare ones who are using JSONP. Such a proxy gets a very large number of hits when a lot of component on the website are downloading content from external domains. So, it becomes a scalability issue when the proxy starts getting millions of hits. Moreover, web page's overall load performance largely depends on the performance of the proxy as it delivers content to the page. In this article, we will take a look how we can take a conventional AJAX Proxy and make it faster, asynchronous, continuously stream content and thus make it more scalable.

You can see such a proxy in action when you go to Pageflakes.com. You will see flakes (widgets) loading many different content like weather feed, flickr photo, youtube videos, RSS from many different external domains. All these are done via a Content Proxy. Content Proxy served about 42.3 million URLs last month which is quite an engineering challenge for us to make it both fast and scalable. Sometimes Content Proxy serves megabytes of data, which poses even greater engineering challenge. As such proxy gets large number of hits, if we can save on an average 100ms from each call, we can save 4.23 million seconds of download/upload/processing time every month. That's about 1175 man hours wasted throughout the world by millions of people staring at browser waiting for content to download.

Such a content proxy takes an external server's URL as a query parameter. It downloads the content from the URL and then writes the content as response back to browser.

Figure: Content Proxy working as a middleman between browser and external domain

The above timeline shows how request goes to the server and then server makes a request to external server, downloads the response and then transmits back to the browser. The response arrow from proxy to browser is larger than the response arrow from external server to proxy because generally proxy server's hosting environment has better download speed than the user's Internet connectivity.

Such a content proxy is also available in my open source Ajax Web Portal Dropthings.com. You can see from its code how such a proxy is implemented.

The following is a very simple synchronous, non-streaming, blocking Proxy:

[WebMethod]
[ScriptMethod(UseHttpGet=true)]
public string GetString(string url)
{
        using (WebClient client = new WebClient())
        {
            string response = client.DownloadString(url);
            return response;
        }
    }
}

Although it shows the general principle, but it's no where close to a real proxy because:

It's a synchronous proxy and thus not scalable. Every call to this web method causes the ASP.NET thread to wait until the call to the external URL completes.
It's non streaming. It first downloads the entire content on the server, storing it in a string and then uploading that entire content to the browser. If you pass MSDN feed URL, it will download that gigantic 220 KB RSS XML on the server and store it on a 220 KB long string (actually double the size as .NET strings are all Unicode string) and then write 220 KB to ASP.NET Response buffer, consuming another 220 KB UTF8 byte array in memory. Then that 220 KB will be passed to IIS in chunks so that it can transmit it to the browser.
It does not produce proper response header to cache the response on the server. Nor does it deliver important headers like Content-Type from the source.
If external URL is providing gzipped content, it decompresses the content into a string representation and thus wastes server memory.
It does not cache the content on the server. So, repeated call to the same external URL within the same second or minute will download content from the external URL and thus waste bandwidth on your server.

So, we need an asynchronous streaming proxy that transmits the content to the browser while it downloads from the external domain server. So, it will download bytes from external URL in small chunks and immediately transmit that to the browser. As a result, browser will see a continuous transmission of bytes right after calling the web service. There will be no delay while the content is fully downloaded on the server.

Before I show you the complex streaming proxy code, let's take an evolutionary approach. Let's build a better Content Proxy that the one shown above, which is synchronous, non-streaming but does not have the other problems mentioned above. We will build a HTTP Handler named RegularProxy.ashx which will take url as a query parameter. It will also take cache as a query parameter which it will use to produce proper response headers in order to cache the content on the browser. Thus it will save browser from downloading the same content again and again.

<%@ WebHandler Language="C#" Class="RegularProxy" %>

using System;
using System.Web;
using System.Web.Caching;
using System.Net;
using ProxyHelpers;
public class RegularProxy : IHttpHandler {
        
    public void ProcessRequest (HttpContext context) {
        string url = context.Request["url"];
        int cacheDuration = Convert.ToInt32(context.Request["cache"]?? "0");
        string contentType = context.Request["type"];

        // We don't want to buffer because we want to save memory
        context.Response.Buffer = false;
            
        // Serve from cache if available
        if (context.Cache[url] != null)
        {
            context.Response.BinaryWrite(context.Cache[url] as byte[]);
            context.Response.Flush();
            return;
        }
        using (WebClient client = new WebClient())
        {
            if (!string.IsNullOrEmpty(contentType))
                client.Headers["Content-Type"] = contentType;
            
            client.Headers["Accept-Encoding"] = "gzip";
            client.Headers["Accept"] = "*/*";
            client.Headers["Accept-Language"] = "en-US";
            client.Headers["User-Agent"] = 
"Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6";
            
            byte[] data = client.DownloadData(url);

            context.Cache.Insert(url, data, null,
                        Cache.NoAbsoluteExpiration,
                        TimeSpan.FromMinutes(cacheDuration),
                        CacheItemPriority.Normal, null); 
            
            if (!context.Response.IsClientConnected) return;

            
            // Deliver content type, encoding and length as it is received from the external URL
            context.Response.ContentType = client.ResponseHeaders["Content-Type"];
            string contentEncoding = client.ResponseHeaders["Content-Encoding"];
            string contentLength = client.ResponseHeaders["Content-Length"];

            if (!string.IsNullOrEmpty(contentEncoding))
                context.Response.AppendHeader("Content-Encoding", contentEncoding);
            if (!string.IsNullOrEmpty(contentLength))
                context.Response.AppendHeader("Content-Length", contentLength);

            if (cacheDuration > 0)
                HttpHelper.CacheResponse(context, cacheDuration);
            
            // Transmit the exact bytes downloaded
            context.Response.BinaryWrite(data);
        }
    }
 
    public bool IsReusable {
        get {
            return false;
        }
    }

}

There are two enhancements in this proxy:

It allows server side caching of content. Same URL requested by a different browser within a time period will not be downloaded on server again, instead it will be served from cache.
It generates proper response cache header so that the content can be cached on browser.
It does not decompress the downloaded content in memory. It keeps the original byte stream intact. This saves memory allocation.
It transmits the data in non-buffered fashion, which means ASP.NET Response object does not buffer the response and thus saves memory

However, this is a blocking proxy. We need to make a streaming asynchronous proxy for better performance. Here's why:

Figure: Continuous streaming proxy

As you see, when data is transmitted from server to browser while server downloads the content, the delay for server side download is eliminated. So, if server takes 300ms to download something from external source, and then 700ms to send it back to browser, you can save up to 300ms Network Latency between server and browser. The situation gets even better when the external server that serves the content is slow and takes quite some time to deliver the content. The slower external site is, the more saving you get in this continuous streaming approach. This is significantly faster than blocking approach when the external server is in Asia or Australia and your server is in USA.

The approach for continuous proxy is:

Read bytes from external server in chunks of 8KB from a separate thread (Reader thread) so that it's not blocked
Store the chunks in an in-memory Queue
Write the chunks to ASP.NET Response from that same queue
If the queue is finished, wait until more bytes are downloaded by the reader thread

The Pipe Stream needs to be thread safe and it needs to support blocking Read. By blocking read it means, if a thread tries to read a chunk from it and the stream is empty, it will suspend that thread until another thread writes something on the stream. Once a write happens, it will resume the reader thread and allow it to read. I have taken the code of PipeStream from CodeProject article by James Kolpack and extended it to make sure it's high performance, supports chunks of bytes to be stored instead of single bytes, support timeout on waits and so on.

A did some comparison between Regular proxy (blocking, synchronous, download all then deliver) and Streaming Proxy (continuous transmission from external server to browser). Both proxy downloads the MSDN feed and delivers it to the browser. The time taken here shows the total duration of browser making the request to the proxy and then getting the entire response.

Figure: Time taken by Streaming Proxy vs Regular Proxy while downloading MSDN feed

Not a very scientific graph and response time varies on the link speed between the browser and the proxy server and then from proxy server to the external server. But it shows that most of the time, Streaming Proxy outperformed Regular proxy.

Figure: Test client to compare between Regular Proxy and Streaming Proxy

You can also test both proxy's response time by going to http://labs.dropthings.com/AjaxStreamingProxy. Put your URL and hit Regular/Stream button and see the "Statistics" text box for the total duration. You can turn on "Cache response" and hit a URL from one browser. Then go to another browser and hit the URL to see the response coming from server cache directly. Also if you hit the URL again on the same browser, you will see response comes instantly without ever making call to the server. That's browser cache at work.

Learn more about Http Response caching from my blog post:
Making best use of cache for high performance website

A Visual Studio Web Test run inside a Load Test shows a better picture:

Figure: Regular Proxy load test result shows Average Requests/Sec 0.79 and Avg Response Time 2.5 sec

Figure: Streaming Proxy load test result shows Avg Req/Sec is 1.08 and Avg Response Time 1.8 sec.

From the above load test results, Streaming Proxy is 26% better Request/Sec and Average Response Time is 29% better. The numbers may sound small, but at Pageflakes, 29% better response time means 1.29 million seconds saved per month for all the users on the website. So, we are effectively saving 353 man hours per month which was wasted staring at browser screen while it downloads content.

Building the Streaming Proxy

The details how the Streaming Proxy is built is quite long and not suitable for a blog post. So, I have written a CodeProject article:

Fast, Scalable, Streaming AJAX Proxy - continuously deliver data from cross domain

Please read the article and please vote for me if your find it useful.

Desktop RSS Feed Aggregator client with Outlook Integration

omar — Tue, 05 Feb 2008 13:10:00 GMT

Back in 2005, I built an open source RSS Feed Aggregator Desktop client which has been quite popular since its release. It has 48,672 downloads so far. From the activity I see on sourceforge statistics, it's being used about 20,000 times per day.

http://rssfeederdotnet.sourceforge.net/

What is RSS Feeder.NET

RSS Feeder.NET is a free open source desktop RSS feed aggregator which downloads feeds from web sources and stores locally for offline viewing, searching and processing. It is also a rich blogging tool which you can use to blog to variety of blog engines including WordPress, B2Evolution, .Text, Community Server etc. You can be fully MS Outlook® dependent or can run fully standalone. You can also use both at the same time whichever you find comfortable to work with. It does not increase Outlook load time, nor does it make Outlook slow frequently or prevent from closing properly. It is a Smart Client that makes best use of both Local Resource and Distributed Web Information sources.

Download Installer ver 3.3 (Nov 10, 2005)

Download Source Code ver 3.3 (Nov 10, 2005)

Here’s an Article which explains how it is developed:
A RSS Feed Aggregator & Blog Smart Client

Features:

Newspaper mode. You can read feeds in a more readable newspaper mode called “Blogpaper”
Auto Discovery. Drag any hyper link and I will find out whether there is any RSS Feed in that page
Outlook integration. You can store feeds in Outlook folders
Blogging. It provides you a Outlook 2003 style convenient workspace to manage your blog accounts and write rich posts
Blog from Outlook. You can specify an Outlook folder for a weblog account. All the posts from that folder is automatically posted to the weblog during synchronization. You can write posts as HTML using Word editor. Post content (HTML markup) is cleaned rigorously before posting to the weblog.
Outlook View. It uses a customized view to present a more readable list of feeds in Outlook Folders. The standard Post view is not easy to browse through quickly. The view puts the subject first in bold and an excerpt of the post under the subject.
Optimized Startup. You can safely put RSS Feeder at startup and it won’t make your Windows® start slower. A clever lazy loading process puts no effort on Windows® during startup instead starts the app when Windows® has finally regained its strength after the long boot up struggle.
Newsgator Import. Newsgator users can use RSS Feeder to import all the subscriptions and seamlessly replace Newsgator without any modification to Outlook folder locations.

Browse Mode

Blogpaper mode

Outlook 2003 View

Tray Application

Blog Module

10 ASP.NET Performance and Scalability Secrets

omar — Wed, 30 Jan 2008 15:47:00 GMT

ASP.NET 2.0 has many secrets, when revealed, can give you big performance and scalability boost. For instance, there are secret bottlenecks in Membership and Profile provider which can be solved easily to make authentication and authorization faster. Furthermore, ASP.NET Http pipeline can be tweaked to avoid executing unnecessary code that gets hit on each and every request. Not only that, ASP.NET Worker Process can be pushed to its limit to squeeze out every drop of performance out of it. Page fragment output caching on the browser (not on the server) can save significant amount of download time on repeated visits. On demand UI loading can give your site a fast and smooth feeling. Finally, Content Delivery Networks (CDN) and proper use of HTTP Cache headers can make your website screaming fast when implemented properly. In this article, you will learn these techniques that can give your ASP.NET application a big performance and scalability boost and prepare it to perform well under 10 times to 100 times more traffic.

http://www.codeproject.com/KB/aspnet/10ASPNetPerformance.aspx

In this article I have shown the following techniques:

ASP.NET Pipeline optimization
ASP.NET Process configuration optimization
Things you must do for ASP.NET before going live
Content Delivery Network
Caching AJAX calls on browser
Making best use of Browser Cache
On demand progressive UI loading for fast smooth experience
Optimize ASP.NET 2.0 Profile provider
How to query ASP.NET 2.0 Membership tables without bringing down the site
Prevent Denial of Service (DOS) attack

The above techniques can be implemented on any ASP.NET website especially those who use ASP.NET 2.0's Membership and Profile provider.

You can learn a lot more about performance and scalability improvement of ASP.NET and ASP.NET AJAX websites from my book - Building a Web 2.0 portal using ASP.NET 3.5.

My first book - Building a Web 2.0 Portal with ASP.NET 3.5

omar — Sun, 13 Jan 2008 20:23:00 GMT

My first book "Building a Web 2.0 Portal with ASP.NET 3.5" from O'Reilly is published and available in the stores. This book explains in detail the architecture design, development, test, deployment, performance and scalability challenges of my open source web portal Dropthings.com. Dropthings is a prototype of a web portal similar to iGoogle or Pageflakes. But this portal is developed using recently released brand new technologies like ASP.NET 3.5, C# 3.0, Linq to Sql, Linq to XML, and Windows Workflow foundation. It makes heavy use of ASP.NET AJAX 1.0. Throughout my career I have built several state-of-the-art personal, educational, enterprise and mass consumer web portals. This book collects my experience in building all of those portals.

O'Reilly Website:
http://www.oreilly.com/catalog/9780596510503/

Amazon:
http://www.amazon.com/Building-Web-2-0-Portal-ASP-NET/dp/0596510500

Disclaimer: This book does not show you how to build Pageflakes. Dropthings is entirely different in terms of architecture, implementation and the technologies involved.

You learn how to:

Implement a highly decoupled architecture following the popular n-tier, widget-based application model
Provide drag-and-drop functionality, and use ASP.NET 3.5 to build the server-side part of the web layer
Use LINQ to build the data access layer, and Windows Workflow Foundation to build the business layer as a collection of workflows
Build client-side widgets using JavaScript for faster performance and better caching
Get maximum performance out of the ASP.NET AJAX Framework for faster, more dynamic, and scalable sites
Build a custom web service call handler to overcome shortcomings in ASP.NET AJAX 1.0 for asynchronous, transactional, cache-friendly web services
Overcome JavaScript performance problems, and help the user interface load faster and be more responsive
Solve various scalability and security problems as your site grows from hundreds to millions of users
Deploy and run a high-volume production site while solving software, hardware, hosting, and Internet infrastructure problems

If you're ready to build state-of-the art, high-volume web applications that can withstand millions of hits per day, this book has exactly what you need.

Linq to SQL: How to Attach object to a different data context

omar — Sat, 08 Dec 2007 12:59:00 GMT

After upgrading to Visual Studio 2008 RTM, you will have trouble updating Linq to SQL Classes which are read from one data context and then updated into another data context. You will get this exception during update:

System.NotSupportedException: An attempt has been made to Attach or Add an entity that is not new, perhaps having been loaded from another DataContext. This is not supported.

Here's a typical example taken from a Forum post:

   1: public static void
UpdateEmployee(Employee employee)

   2: {

   3: using (HRDataContext dataContext =
new HRDataContext())

   4: {

   5: //Get original employee

   6: Employee originalEmployee =
dataContext.Employees.Single(e=
>e.EmployeeId==employee.EmployeeId);

7:

   8: //attach to datacontext

   9:
dataContext.Employees.Attach(employee, originalEmployee);

10:

  11: //save changes

  12: dataContext.SubmitChanges();

13:

  14: }

  15: }

When you call the Attach function, you will get the exception mentioned above.

Here's a way to do this. First, create a partial class that adds a Detach method to Employee class. This method will detach the object from it's data context and detach associated objects.

   1: public partial class Employee

   2: {

   3:   public void Detach()

   4:   {

   5:     this.PropertyChanged = null;
this.PropertyChanging = null;

6:

   7:     // Assuming there's a foreign
key from Employee to Boss

   8:     this.Boss = default(EntityRef
<
Boss
>);

   9:     // Similarly set child objects
to default as well

10:

  11:     this.Subordinates =
default(EntitySet
<
Subordinate
>);

  12:   }

  13: }

14:

15:

Now during update, call Detach before attaching the object to a DataContext.

   1: public static void
UpdateEmployee(Employee employee)

   2: {

   3:     using (HRDataContext
dataContext = new HRDataContext())

   4:     {

   5:         //attach to datacontext

   6:         employee.Detach();

7:

   8:        
dataContext.Employees.Attach(employee);

   9:         //save changes

10:

  11:        
dataContext.SubmitChanges();

  12:     }

  13: }

14:

This'll work. It assumes the employee object already has its primary key populated.

You might see during update, it's generating a bloated UPDATE statement where each and every property is appearing on the WHERE clause. In that case, set UpdateCheck to Never for all properties of Employee class from the Object Relational Designer.

On-demand UI loading on AJAX websites

omar — Fri, 09 Nov 2007 09:01:00 GMT

AJAX websites are all about loading as many features as possible into the browser without having any postback. If you look at the Start Pages like Pageflakes, it's only one single page that gives you all the features of the whole application with zero postback. A quick and dirty approach for doing this is to deliver every possible html snippets inside hidden divs during page load and then make those divs visible when needed. But this makes first time loading way too long and browser gives sluggish performance as too much stuff is on the DOM. So, a better approach is to load html snippet and necessary javascript on-demand. In my dropthings project, I have shown an example how this is done.

When you click on the "help" link, it loads the content of the help dynamically. This html is not delivered as part of the default.aspx that renders the first page. Thus the giant html and graphics related to the help section has no effect on site load performance. It is only loaded when user clicks the "help" link. Moreover, it gets cached on the browser and thus loads only once. When user clicks the "help" link again, it's served directly from the browser cache, instead of fetching from the origin server again.

The principle is making an XMLHTTP call to an .aspx page, get the response html, put that response html inside a container DIV, make that DIV visible.

AJAX framework has a Sys.Net.WebRequest class which you can use to make regular HTTP calls. You can define http method, URI, headers and the body of the call. It’s kind of a low level function for making direct calls via XMLHTTP. Once you construct a web request, you can execute it using Sys.Net.XMLHttpExecutor.

   1: function showHelp()

   2: {

   3:     var request = 
new Sys.Net.WebRequest();

   4:     request.set_httpVerb(
"GET");

   5:     request.set_url(
'help.aspx');

   6:     request.add_completed(
function( executor )

   7:     {

   8:         
if (executor.get_responseAvailable())

   9:         {

  10:             var helpDiv = $get(
'HelpDiv');

  11:             var helpLink = $get(
'HelpLink');

12:

  13:             var helpLinkBounds =
Sys.UI.DomElement.getBounds(helpLink);

14:

  15:             helpDiv.style.top =
(helpLinkBounds.y + helpLinkBounds.height) + 
"px";

16:

  17:             var content =
executor.get_responseData();

  18:             helpDiv.innerHTML =
content;

  19:             helpDiv.style.display =

"block";

  20:         }

  21:     });

22:

  23:     var executor = 
new Sys.Net.XMLHttpExecutor();

  24:     request.set_executor(executor);

  25:     executor.executeRequest();

  26: }

The example shows how the help section is loaded by hitting help.aspx and injecting its response inside the HelpDiv. The response can be cached by the output cache directive set on help.aspx. So, next time when user clicks on the link, the UI pops up immediately. The help.aspx file has no <html> block, only the content that is set inside the DIV.

   1: 
<%@ Page Language="C#" AutoEventWireup="true"
CodeFile="Help.aspx.cs" Inherits="Help" %>

   2: 
<%@ OutputCache Location="ServerAndClient"
Duration="604800" VaryByParam="none" %>

   3: 
<
div 
class
="helpContent"
>

   4: 
<
div 
id
="lipsum"
>

   5: 
<
p
>

   6: Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Duis lorem

   7: eros, volutpat sit amet, venenatis
vitae, condimentum at, dolor. Nunc

   8: porttitor eleifend tellus. Praesent
vitae neque ut mi rutrum cursus.

Using this approach, you can break the UI into smaller .aspx files. Although these .aspx files cannot have JavaScript or stylesheet blocks, but they can contain large amount of html that you need to show on the UI on-demand. Thus you can keep initial download to absolute minimum just for loading the basic stuffs. When user explores new features on the site, load those areas incrementally.

Safe COM: Managed Disposable Strongly Typed safe wrapper to late bound COM

omar — Sat, 27 Oct 2007 03:35:00 GMT

There are several problems using COM from .NET:

You cannot implement the Dispose pattern by utilizing the "using" block in order to safely dispose COM references.
You cannot guaranty COM references are finalized. There's no way to implement "~Destructor()" for COM references.
COM reference is not released when a call to Marshal.ReleaseComObject is skipped due to an Exception.
When you "Add Reference..." to a COM library, the reference is version specific. So, if you add a reference to the Office 2003 COM library, it does not work properly when deployed to Office 2000.
The only solution to version independent COM is to use Late Bound operations but you miss all the features of a strongly typed language.

Let's solve all these problems. We want to use a Managed strongly typed approach to Late Bound COM operations and also utilize the Dispose pattern on COM objects. The solution proposed here works for any COM object which can be Microsoft Office COM libraries, IE & DHTML objects and even your own COM objects. You should use this approach whenever you are dealing with any type of COM library.

If you like this solution, please vote for me.
http://www.codeproject.com/csharp/safecomwrapper.asp

10 cool web development related articles in 2007

omar — Tue, 02 Oct 2007 17:26:00 GMT

Here's a list of 10 cool ASP.NET, AJAX and web development related articles and blog posts that I have written this year that you might want to take a look:

13 disasters for production website and their solutions
Talks about 13 production disasters that can happen to any website any time and bring down your business.

Build Google IG like Ajax Start Page in 7 days using ASP.NET Ajax and .NET 3.0
This block buster article shows how ASP.NET AJAX, Linq to XML, Linq to SQL and Workflow Foundation can be used to create a Google IG like start page in just 7 nights. Learn how to put together such hot technologies into one project and make a production site out of it.

Serve extensionless URL from ASP.NET without using ISAPI module or IIS 6 Wildcard mapping
Currently there are only two ways to service extentionless URL like www.pageflakes.com/omar that hits something besides the default document - use a custom ISAPI module or use IIS 6 wildcard mapping. Both has performance and scalability problems because both intercepts each and every hit. Learn how you can solve it by using a custom 404 handler.

Request format is unrecognized for URL unexpectedly ending in /SomeWebServiceMethod
Since ASP.NET AJAX 1.0 release, Microsoft prevented JSON hijacking by adding a special content type header. But this caused us some trouble.

Cleanup inactive anonymous users from ASP.NET Membership Tables
When you store anonymous user profile using ASP.NET Membership provider and Anonymous Identification provider, you soon end up with lots of idle anonymous user data where those users never come back. We (Pageflakes) went through a lot of difficulty keeping our database size down as we allow anonymous users to do almost everything that a registered user can do. This introduces scalability challenge. See how we solved this problem.

Prevent Denial of Service (DOS) attacks in your web application
Web applications can be brought down to its knees by hitting the site repeatedly or by calling expensive webservices randomly. Anyone can write a simple loop that hits a webserver very frequently from a high bandwidth connectivity and bring your production server down. See how to prevent such application level DOS attacks.

ASP.NET Ajax Extender for multi-column widget drag & drop
It's an ASP.NET AJAX extender that allows Pageflakes style drag & drop functionality between columns and rows.

ASP.NET Ajax in-depth performance analysis
While building an open source start page using ASP.NET AJAX, I have done a lot of performance analysis on AJAX framework in order to improve first time load and perceived speed of javascript rich pages. Check out my analysis.

Think you know how to write UPDATE statement? Think again.
Learn how to optimize common UPDATE statements

Make a surveillance application which captures desktop and then emails you as attachment
Some time back I needed to capture a certain computers desktop in order to find out what that user is doing every day. So, I made a .NET 2.0 Winforms Application which stays on system tray (optional) and capture the desktop in given time interval (say every 60 secs) and emailed the captured images to me as message attachment (say every 30 mins).

Today I received MVP award for the 3rd time on Visual C#. Thanks to Microsoft for the award and setting up my new blog. I will continue both my MVPS Blog and this blog from now on.