Cleanup inactive anonymous users from ASP.NET Membership Tables

omar — Sun, 25 Mar 2007 18:39:00 GMT

ASP.NET 2.0 Websites that allow anonymous visit and anonymous user profile have a unique challenge to cleanup unused data which is generated by anonymous users who never come back. Every first visit is creating one anonymous user, page setup, and other user specific content. If the user never comes back, it still remains in the database permanently. It is possible user might come back within a day, or a week or a month. But there’s no guaranty if user will ever come back or not. Generally sticky users are max 30% of the total users who come to most websites. So, you end up with 70% unused data which are never needed. All these requires cleanup, otherwise the database keeps growing uncontrollably and gets slower and slower. This cleanup operation is humongous for busy websites. Think about deleting millions of rows from several tables, one after another while maintaining foreign key constraints. Also the cleanup operation needs to run while the site is running, without hampering site's overall performance. The whole operation results in heavily fragmented index and space in the MDF file. The log file also becomes enormous in order to keep track of the transactions. Hard drives get really hot and start sweating furiously. While the CPU keeps smiling having nothing to do with it, it’s really painful to watch SQL Server go through this every day. Unless you clean up the database and maintain its size under control; you can't keep up with SQL Server’s RAM and Disk IO requirement.

When a user visits the site, Asp.net Membership Provider updates the LastActivityDate of aspnet_users table. From this field, I can find out how long the user has been idle. The IsAnonymous bit field tells me whether the user account is anonymous or registered. If it is registered, no need to worry. But if it is anonymous and more than 30 days old, I can be sure that the user will never come back because the cookie has already expired. If you repeatedly logout from your start page, all cookie related to the site gets cleared. That means you are producing one new anonymous user record during each log out. That anonymous record is never used because you will soon log in to have your customized pages back and then you will log out again. This will result in another anonymous user account which again becomes useless as soon as you log in.

Here’s how the whole cleanup process works:

Find out the users which are old enough to be discarded
Find out the pages user has
Delete all the widget instances on those pages
Then delete those pages
Remove rows from child tables related to aspnet_users like aspnet_profile, aspnet_UsersInRoles, aspnet_PersonalizationPerUser. Remove rows for users to be deleted
Remove the users from aspnet_users
Pray that you did not accidentally remove any good user

Here’s the giant DB script which does it all. I have put enough inline comment so that you can understand what the script is doing:

   1: -- Number of days after which we give users 'bye bye'

   2: DECLARE @Days int

   3: SET @Days = 14

4:

   5: -- No of users to delete per run. If it's too high, database will get stuck

   6: -- for a long time. If it's too low, you will end up having more trash than

   7: -- you can cleanup

   8: DECLARE @NoOfUsersToDelete int

   9: SET @NoOfUsersToDelete = 1000

10:

  11: -- Create temporary tables which holds the users and pages to delete

  12: -- As the user and the page is used to find out other tables, instead

  13: -- of running SELECT ID FORM ... repeatedly, it's better to have them

  14: -- in a temp table

  15: IF  EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[PagesToDelete]') AND type in (N'U'))

  16: DROP TABLE [dbo].[PagesToDelete]

  17: IF  EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[aspnetUsersToDelete]') AND type in (N'U'))

  18: DROP TABLE [dbo].[AspnetUsersToDelete]

19:

  20: create table PagesToDelete (PageID int NOT NULL PRIMARY KEY)

  21: create table AspnetUsersToDelete (UserID uniqueidentifier NOT NULL PRIMARY KEY)

22:

  23: -- Find out inactive anonymous users and store the UserID in the temporary

  24: -- table

  25: insert into AspnetUsersToDelete

  26: select top(@NoOfUsersToDelete) UserID from aspnet_Users where

  27: (isAnonymous = 1) and (LastActivityDate < (getDate()-@Days))

  28: order by UserID -- Saves SQL Server from sorting in clustered index again

29:

  30: print 'Users to delete: ' + convert(varchar(255),@@ROWCOUNT)

  31: GO

32:

  33: -- Get the pages of the users which will be deleted

  34: insert into PagesToDelete

  35: select ID from Page where UserID in

  36: (

  37: select UserID from AspnetUsersToDelete

  38: )

39:

  40: print 'Pages to delete: ' + convert(varchar(255),@@ROWCOUNT)

  41: GO

42:

  43: -- Delete all Widget instances on the pages to be deleted

  44: delete from WidgetInstance where PageID IN

  45: ( SELECT PageID FROM PagesToDelete )

46:

  47: print 'Widget Instances deleted: ' + convert(varchar(255), @@ROWCOUNT)

  48: GO

49:

  50: -- Delete the pages

  51: delete from Page where ID IN

  52: ( SELECT PageID FROM PagesToDelete )

  53: GO

54:

  55: -- Delete User Setting

  56: delete from UserSetting WHERE UserID IN

  57: ( SELECT UserID FROm AspnetUsersToDelete )

  58: GO

59:

  60: -- Delete profile of users

  61: delete from aspnet_Profile WHERE UserID IN

  62: ( SELECT UserID FROm AspnetUsersToDelete )

  63: GO

64:

  65: -- Delete from aspnet_UsersInRoles

  66: delete from aspnet_UsersInRoles WHERE UserID IN

  67: ( SELECT UserID FROm AspnetUsersToDelete )

  68: GO

69:

  70: -- Delete from aspnet_PersonalizationPerUser

  71: delete from aspnet_PersonalizationPerUser WHERE UserID IN

  72: ( SELECT UserID FROm AspnetUsersToDelete )

  73: GO

74:

  75: -- Delete the users

  76: delete from aspnet_users where userID IN

  77: ( SELECT UserID FROm AspnetUsersToDelete )

78:

  79: PRINT 'Users deleted: ' + convert(varchar(255), @@ROWCOUNT)

  80: GO

81:

82:

  83: drop table PagesToDelete

  84: drop table AspnetUsersToDelete

  85: GO

Now the question comes, when can I run this script? It depends on several factors:

The lowest traffic period. For example, USA midnight time when everyone in USA is sleeping if your majority users are from USA
The period when there’s no other maintenance tasks running like Index Defrag or Database Bakup. If by any chance any other maintenance task conflicts with this enormous delete operation, SQL Server is dead.
The operation will take from 10 mins to hours depending on the volume of trash to cleanup. So, consider the duration of running this script and plan other maintenance jobs accordingly.
It’s best to run 30 mins before INDEX DEFRAG jobs run. After the script completes, the tables will be heavily fragmented. So, you need to defrag the indexes.

Before running this script, there are some preparations to take:

Make sure you have turned of AutoShrink from Database Property. Database size will reduce after the cleanup and if SQL Server tried to shrink the database, there will be a big IO activity. Turn off auto shrink because the database will grow again.
Make sure the LOG file’s initial size is big enough to hold such enormous transactions. You can specify 1/3rd of the MDF size as LDF’s Initial Size. Also make sure the log file is not shrunk. Let it occupy HD space. It saves SQL Server from expanding the file and shrinking the file. Both of these require high Disk IO.

Once the cleanup job runs and the INDEX DEFRAG runs, the database performance will improve significantly. The tables are now smaller. That means the indexes are now smaller. SQL Server need not to run through large indexes anymore. Future INDEX DEFRAGs take shorter time because there’s not much data left to optimize. SQL Server also takes less RAM because it has to work with much less amount of data. Database backup size also reduces because the MDF size does not keep increasing indefinitely. As a result, the significant overhead of this cleanup operation is quite acceptable when compared to all the benefits.

Note: I will be posting some stuffs from my old blog to new blog. Please ignore if you have read them before.

Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 : asp.net sqlserver

Cleanup inactive anonymous users from ASP.NET Membership Tables