Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 : production

99.99% available ASP.NET and SQL Server SaaS Production Architecture

omar — Wed, 10 Dec 2008 08:39:00 GMT

You have a hot ASP.NET+SQL Server product, growing at thousand users per day and you have hit the limit of your own garage hosting capability. Now that you have enough VC money in your pocket, you are planning to go out and host on some real hosting facility, maybe a colocation or managed hosting. So, you are thinking, how to design a physical architecture that will ensure performance, scalability, security and availability of your product? How can you achieve four-nine (99.99%) availability? How do you securely let your development team connect to production servers? How do you choose the right hardware for web and database server? Should you use Storage Area Network (SAN) or just local disks on RAID? How do you securely connect your office computers to production environment?

Here I will answer all these queries. Let me first show you a diagram that I made for Pageflakes where we ensured we get four-nine availability. Since Pageflakes is a Level 3 SaaS, it’s absolutely important that we build a high performance, highly available product that can be used from anywhere in the world 24/7 and end-user gets quick access to their content with complete personalization and customization of content and can share it with others and to the world. So, you can take this production architecture as a very good candidate for Level 3 SaaS:

Here’s a CodeProject article that explains all the ideas:

99.99% available ASP.NET and SQL Server SaaS Production Architecture

Hope you like it. Appreciate your vote.

ASP.NET website Continuous Integration+Deployment using CruiseControl.NET, Subversion, MSBuild and Robocopy

omar — Mon, 06 Oct 2008 06:59:00 GMT

You can setup continuous integration and automated deployment for your web application using CruiseControl.NET, Subversion, MSBuild and Robocopy. I will show you how you can automatically build the entire solution, email build report to developers and QA, deploy latest code in IIS all using CruiseControl.NET every N minutes.

First get the following:

CruiseControl.NET
Subversion (install the command line tools and add the Subversion bin path to PATH environment variable)
Robocopy (Windows Vista/2008 has it built-in, here's the link for Windows 2003)
Install .NET Framework. You need it for MSBuild.

You will learn how I have configured Continuous Integration and Deployment for my open source AJAX Portal project www.Dropthings.com. The code is hosted at CodePlex. When some developer makes a commit, CruiseControl downloads the latest code, builds the entire solution, emails build report and then deploys the latest web site to IIS 6.0.

After installing CruiseControl.NET, go to Programs -> Cruise Control -> CruiseControl.NET Config.

Now keep copying and pasting the following XML blocks and make sure you understand each block and make necessary changes:

<cruisecontrol>
    <project name="Dropthings" queue="DropthingsQueue" queuePriority="1">
        <!-- 
        Path to the trunk folder where the full solution starts from. This is where
        subversion checkout and incremental update is performed 
        -->
        <workingDirectory>d:\cc\dropthings\code\trunk\</workingDirectory>
        <!-- Some path where CCNet writes its logs and stuffs. It can be outside the log folder -->
        <artifactDirectory>d:\cc\dropthings\artifact\</artifactDirectory>
        <category>Dropthings</category>
        <!-- CCNet installs a web dashboard. Enter the URL of that dashboard here -->
        <webURL>http://localhost/ccnet/</webURL>
        <modificationDelaySeconds>60</modificationDelaySeconds>
        <labeller type="defaultlabeller">
            <prefix>0.1.</prefix>
            <incrementOnFailure>true</incrementOnFailure>
            <labelFormat>000</labelFormat>
        </labeller>
        <state type="state" directory="State" />

First change the working directory. It needs to be the path of the folder where you will have the solution downloaded. I generally create folder structure like this:

D:\CC - Root for all CC.NET enabled projects
- \ProjectName - Root project folder
  - \Code - Code folder where code is downloaded from subversion
  - \Artifact - CC.NET generates a lot of stuff. All goes here.

Next comes the Subversion integration block:

<sourcecontrol type="svn">
    <!-- Subversion trunk repository to keep checking for latest code -->
    <trunkUrl>http://localhost:8081/tfs02.codeplex.com/dropthings/trunk</trunkUrl>
    <workingDirectory></workingDirectory>
    <username>***** SUBVERSION USER NAME *****</username>
    <password>***** SUBVERSION PATH *****</password>
</sourcecontrol>

Here specify the subversion location where you want to download code to the working folder. You should download the entire solution because you will be building the entire solution using MSBuild soon.

I left <workingDirectory> empty. This means whatever is specified earlier in the <workingDirectory> is used. Otherwise you can put some relative folder path here or any absolute folder.

Now we start building the tasks that CC.NET executes - Build, Email, and Deploy.

<tasks>
    <artifactcleanup   cleanUpMethod="KeepLastXBuilds"   cleanUpValue="5" />
    <modificationWriter>
        <filename>mods.xml</filename>
        <path></path>
    </modificationWriter>
  
    <!-- MSBuild task to build a .msbuild file that basically builds a .sln file -->
    <msbuild>
        <executable>C:\windows\Microsoft.NET\Framework64\v3.5\MSBuild.exe</executable>
        <workingDirectory></workingDirectory>
        <projectFile>Dropthings.msbuild</projectFile>
        <targets>Build</targets>
        <timeout>300</timeout>
        <logger>C:\Program Files (x86)\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
    </msbuild>

This block first says, keep artifacts for last 5 build and remove olders. Artifacts are like build reports, logs etc. You can increase the value for longer history.

Then the most important <msbuild> task. The executable path is to the MSBuild.exe. I am using .NET 3.5 Framework 64bit edition. You might have .NET 2.0 and 32bit version. So, set the right path here for the MSbuild.exe.

<projectFile> maps to a MSBuild file. It's a skeleton MSBuild file which basically says build this Visual Studio solution file. Here's how the msbuild file looks like:

<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Build"> 
    <!-- Rebuild entire solution -->
    <MSBuild Projects="Dropthings.sln" Targets="Rebuild" />
  </Target>
</Project>

The Dropthings.msbuild and Dropthings.sln file exists in the same trunk folder. This file says - build Dropthings.sln and do a rebuild.

Now you got the build done. Next is to deploy it. You will be using robocopy to copy files from the code folder to a destination folder which is mapped in IIS to a website. Robocopy will do a synchronization of the directories. It will add new files, overwrite old files and removes files from destination folder which no longer exists in the source folder.

Before you can deploy, you need to stop the website or restart IIS. Otherwise some files may be in use and you will not be able to delete or overwrite the files. Here's how to stop IIS using the iisreset command line tool:

<!-- 
Stop IIS before copying over the latest web project files so that there's no write lock and IIS does not
start restarting the site half way through
-->
<exec>
    <executable>iisreset</executable>
    <buildArgs>/stop</buildArgs>
</exec>

If you do not want to stop the entire IIS, instead just stop a website and recycle an application pool, you can use the iisweb.vbs script for stopping a website and iisapp.vbs script for recycling application pool. Here's an example:

<exec>
    <executable>iisweb</executable>
    <buildArgs>/stop "Dropthings"</buildArgs>
</exec>
 
<exec>
    <executable>iisapp</executable>
    <buildArgs> /a "Dropthings" /r</buildArgs>
</exec>

You need to first register cscript as the default script runtime. In order to do this, go to command line and enter iisweb. It will tell you that it cannot use wscript to run this script and it needs to make cscript default. Let it make cscript as default.

Now the time to do the deployment of latest web site files. The following task launches robocopy to do the deployment:

<!--
Sync the web project folder with the deployment folder. The deployment folder is where IIS
is mapped to serve the site. The deployment folder is at the buildArgs node. The robocopy 
utility does an exact sync, adding new files, updating old files, deleting files that no longer
exist.
-->
<exec>
    <!--<executable>C:\Program Files (x86)\Windows Resource Kits\Tools\robocopy.exe</executable>-->
    <executable>robocopy.exe</executable>
    <baseDirectory>Dropthings\</baseDirectory>
    <buildArgs>.\ d:\cc\Dropthings\Deploy *.* /E /XA:H /PURGE /XO /XD ".svn" /NDL /NC /NS /NP</buildArgs>
    <buildTimeoutSeconds>60</buildTimeoutSeconds>
    <successExitCodes>1,0</successExitCodes>
</exec>

First you need to correct the robocopy.exe path. For Windows Vista/Windows 2008, keep it as it is. For Windows 2003, you need to specify the full path. You also need to remove the (x86) from the path if you have 32bit OS.

Next is the <baseDirectory>. This is relative to the working directory. It's the path of the website folder. Dropthings website folder is located under the Dropthings folder under trunk. So, I have specified Dropthings\ as the subfolder where the website files are located. You need to specify your project's website folder's relative path here form the <workingDirectory>.

Next change the path in the <buildArgs> node. First one is the source ".\" which you keep as it is. It means copy files from the baseDirectory. Next is the absolute path to the deployment folder where the web site is mapped in IIS. You can use both relative or absolute path here. While using relative path, just keep in mind the robocopy is running from the <workingDirectory>\<baseDirectory> folder.

After the path keep the *.* and the remaining flags intact. The flags mean:

Copy all subdirectories /E
Copy hidded files /XA:H
Do not copy old files /XO
Exclude .svn directory while copying files /XD ".svn"
Do not show list of files and directories being copie /NDL, /NC, /NP

After the deployment, you need to turn IIS back on or start the website that you stopped:

<!-- Turn IIS back on -->
<exec>
    <executable>iisreset</executable>
    <buildArgs>/start</buildArgs>
</exec>
 
<!--
<exec>
    <executable>iisweb</executable>
    <buildArgs>/start "Dropthings"</buildArgs>
</exec>            
-->

Now we got the build and deployment done. Next is to email a nice report to developers and QA. If build succeeds, email both developers and QA so that they can check out the latest build. But if build fails, email only developers.

<publishers>
    <rss/>
    <xmllogger />
    <statistics />
 
    <!-- Email build report to development and QA team -->
    <email from="admin@yourcompany.com" mailhost="localhost" mailport="25" includeDetails="TRUE"
           mailhostUsername="" mailhostPassword="" useSSL="FALSE">
 
        <users>
            <user name="Developer1" group="devs" address="dev1@yourcompany.com"/>
            <user name="Developer2" group="devs" address="dev2@yourcompany.com"/>
            <user name="Developer3" group="devs" address="dev3@yourcompany.com"/>
            
            <user name="QA1" group="qa" address="qa1@yourcompany.com"/>
            <user name="QA2" group="qa" address="qa2@yourcompany.com"/>
            <user name="QA3" group="qa" address="qa3@yourcompany.com"/>                    
            
        </users>
 
        <groups>
            <group name="devs" notification="Always"/>
            <group name="qa" notification="Success"/>
        </groups>
 
        <converters>
            <!--<regexConverter find="$" replace="@dropthings.com" />-->
        </converters>
 
        <modifierNotificationTypes>
            <NotificationType>Always</NotificationType>
        </modifierNotificationTypes>
 
    </email>
    <modificationHistory  onlyLogWhenChangesFound="true" />
</publishers>

First you need to change the <email> tab where you specify the from address, mail server name, and optionally a user account for the email address that you need to use to send out emails.

Then edit the <users> node and put your developers and QA.

That's it! You got the configuration file done. Next step is to launch the CruiseControl.NET from Programs -> CruiseControl.NET -> CruiseControl.NET. It will launch a process that will execute the tasks according to the configuration. On Windows Vista, you will have to run it with Administrative privilege.

There's also a Windows Service that gets installed. It's named CruiseControl.NET. You can start the service as well on a server and go to sleep. It will do continuous integration and automated deployment for you.

There's also a web based Dashboard that you can use to force a build or stop a build or see detail build reports.

You can create multiple projects. You can have one project to build trunk code only, but do no deployment. Then you can create another project to build, deploy some branch that's ready for production. You can create another project to build and deploy on QA server and so on.

Here's the full configuration file that you can use as your baseline.

Best practices for creating websites in IIS 6.0

omar — Sat, 04 Oct 2008 10:05:22 GMT

Every time I create an IIS website, I do some steps, which I consider as best practice for creating any IIS website for better performance, maintainability, and scalability. Here' re the things I do:

Create a separate application pool for each web application

I always create separate app pool for each web app because I can select different schedule for app pool recycle. Some heavy traffic websites have long recycle schedule where low traffic websites have short recycle schedule to save memory. Moreover, I can choose different number of processes served by the app pool. Applications that are made for web garden mode can benefit from multiple process where applications that use in-process session, in memory cache needs to have single process serving the app pool. Hosting all my application under the DefaultAppPool does not give me the flexibility to control these per site.

The more app pool you create, the more ASP.NET threads you make available to your application. Each w3wp.exe has it's own thread pool. So, if some application is congesting particular w3wp.exe process, other applications can run happily on their separate w3wp.exe instance, running under separate app pool. Each app pool hosts its own w3wp.exe instance.

So, my rule of thumb: Always create new app pool for new web applications and name the app pool based on the site's domain name or some internal name that makes sense. For example, if you are creating a new website alzabir.com, name the app pool alzabir.com to easily identify it.

Another best practice: Disable the DefaultAppPool so that you don't mistakenly keep adding sites to DefaultAppPool.

First you create a new application pool. Then you create a new Website or Virtual Directory, go to Properties -> Home Directory tab -> Select the new app pool.

Customize Website properties for performance, scalability and maintainability

First you map the right host headers to your website. In order to do this, go to WebSite tab and click on "Advanced" button. Add mapping for both domain.com and www.domain.com. Most of the time, people forget to map the domain.com. Thus many visitors skip typing the www prefix and get no page served.

Next turn on some log entries:

These are very handy for analysis. If you want to measure your bandwidth consumption for specific sites, you need the Bytes Sent. If you want to measure the execution time of different pages and find out the slow running pages, you need Time Taken. If you want to measure unique and returning visitors, you need the Cookie. If you need to know who is sending you most traffic - search engines or some websites, you need the Referer. Once these entries are turned on, you can use variety of Log Analysis tools to do the analysis. For example, open source AWStats.

But if you are using Google Analytics or something else, you should have these turned off, especially the Cookie and Referer because they take quite some space on the log. If you are using ASP.NET Forms Authentication, the gigantic cookie coming with every request will produce gigabytes of logs per week if you have a medium traffic website.

This is kinda no brainer. I add Default.aspx as the default content page so that, when visitors hit the site without any .aspx page name, e.g. alzabir.com, they get the default.aspx served.

Things I do here:

Turn on Content Expiration. This makes static files remain in browser cache for 30 days and browser serves the files from its own cache instead of hitting the server. As a result, when your users revisit, they don't download all the static files like images, javascripts, css files again and again. This one setting significantly improves your site's performance.
Remove the X-Powered-By: ASP.NET header. You really don't need it unless you want to attach Visual Studio Remote Debugger to your IIS. Otherwise, it's just sending 21 bytes on every response.
Add "From" header and set the server name. I do this on each webserver and specify different names on each box. It's handy to see from which servers requests are being served. When you are trying to troubleshoot load balancing issues, it comes handy to see if a particular server is sending requests.

I set the 404 handler to some ASPX so that I can show some custom error message. There's a 404.aspx which shows some nice friendly message and suggests some other pages that user can visit. However, another reason to use this custom mapping is to serve extensionless URL from IIS. Read this blog post for details.

Make sure to set ASP.NET 2.0 for your ASP.NET 2.0, 3.0 and 3.5 websites.

Finally, you must, I repeat you "MUST" turn on IIS 6.0 gzip compression. This turns on the Volkswagen V8 engine that is built into IIS to make your site screaming fast.

Deploy ASP.NET MVC on IIS 6, solve 404, compression and performance problems

omar — Mon, 30 Jun 2008 08:12:22 GMT

There are several problems with ASP.NET MVC application when deployed on IIS 6.0:

Extensionless URLs give 404 unless some URL Rewrite module is used or wildcard mapping is enabled
IIS 6.0 built-in compression does not work for dynamic requests. As a result, ASP.NET pages are served uncompressed resulting in poor site load speed.
Mapping wildcard extension to ASP.NET introduces the following problems:
- Slow performance as all static files get handled by ASP.NET and ASP.NET reads the file from file system on every call
- Expires headers doesn't work for static content as IIS does not serve them anymore. Learn about benefits of expires header from here. ASP.NET serves a fixed expires header that makes content expire in a day.
- Cache-Control header does not produce max-age properly and thus caching does not work as expected. Learn about caching best practices from here.
After deploying on a domain as the root site, the homepage produces HTTP 404.

Problem 1: Visiting your website's homepage gives 404 when hosted on a domain

You have done the wildcard mapping, mapped .mvc extention to ASP.NET ISAPI handler, written the route mapping for Default.aspx or default.aspx (lowercase), but still when you visit your homepage after deployment, you get:

You will find people banging their heads on the wall here:

Solution is to capture hits going to "/" and then rewrite it to Default.aspx:

You can apply this approach to any URL that ASP.NET MVC is not handling for you and it should handle. Just see the URL reported on the 404 error page and then rewrite it to a proper URL.

Problem 2: IIS 6 compression is no longer working after wildcard mapping

When you enable wildcard mapping, IIS 6 compression no longer works for extensionless URL because IIS 6 does not see any extension which is defined in IIS Metabase. You can learn about IIS 6 compression feature and how to configure it properly from my earlier post.

Solution is to use an HttpModule to do the compression for dynamic requests.

Problem 3: ASP.NET ISAPI does not cache Static Files

When ASP.NET's DefaultHttpHandler serves static files, it does not cache the files in-memory or in ASP.NET cache. As a result, every hit to static file results in a File read. Below is the decompiled code in DefaultHttpHandler when it handles a static file. As you see here, it makes a file read on every hit and it only set the expiration to one day in future. Moreover, it generates ETag for each file based on file's modified date. For best caching efficiency, we need to get rid of that ETag, produce an expiry date on far future (like 30 days), and produce Cache-Control header which offers better control over caching.

So, we need to write a custom static file handler that will cache small files like images, Javascripts, CSS, HTML and so on in ASP.NET cache and serve the files directly from cache instead of hitting the disk. Here are the steps:

Install an HttpModule that installs a Compression Stream on Response.Filter so that anything written on Response gets compressed. This serves dynamic requests.
Replace ASP.NET's DefaultHttpHandler that listens on *.* for static files.
Write our own Http Handler that will deliver compressed response for static resources like Javascript, CSS, and HTML.

Here's the mapping in ASP.NET's web.config for the DefaultHttpHandler. You will have to replace this with your own handler in order to serve static files compressed and cached.

Solution 1: An Http Module to compress dynamic requests

First, you need to serve compressed responses that are served by the MvcHandler or ASP.NET's default Page Handler. The following HttpCompressionModule hooks on the Response.Filter and installs a GZipStream or DeflateStream on it so that whatever is written on the Response stream, it gets compressed.

These are formalities for a regular HttpModule. The real hook is installed as below:

Here you see we ignore requests that are handled by ASP.NET's DefaultHttpHandler and our own StaticFileHandler that you will see in next section. After that, it checks whether the request allows content to be compressed. Accept-Encoding header contains "gzip" or "deflate" or both when browser supports compressed content. So, when browser supports compressed content, a Response Filter is installed to compress the output.

Solution 2: An Http Module to compress and cache static file requests

Here's how the handler works:

Hooks on *.* so that all unhandled requests get served by the handler
Handles some specific files like js, css, html, graphics files. Anything else, it lets ASP.NET transmit it
The extensions it handles itself, it caches the file content so that subsequent requests are served from cache
It allows compression of some specific extensions like js, css, html. It does not compress graphics files or any other extension.

Let's start with the handler code:

Here you will find the extensions the handler handles and the extensions it compresses. You should only put files that are text files in the COMPRESS_FILE_TYPES.

Now start handling each request from BeginProcessRequest.

Here you decide the compression mode based on Accept-Encoding header. If browser does not support compression, do not perform any compression. Then check if the file being requested falls in one of the extensions that we support. If not, let ASP.NET handle it. You will see soon how.

Calculate the cache key based on the compression mode and the physical path of the file. This ensures that no matter what the URL requested, we have one cache entry for one physical file. Physical file path won't be different for the same file. Compression mode is used in the cache key because we need to store different copy of the file's content in ASP.NET cache based on Compression Mode. So, there will be one uncompressed version, a gzip compressed version and a deflate compressed version.

Next check if the file exits. If not, throw HTTP 404. Then create a memory stream that will hold the bytes for the file or the compressed content. Then read the file and write in the memory stream either directly or via a GZip or Deflate stream. Then cache the bytes in the memory stream and deliver to response. You will see the ReadFileData and CacheAndDeliver functions soon.

This function delivers content directly from ASP.NET cache. The code is simple, read from cache and write to the response.

When the content is not available in cache, read the file bytes and store in a memory stream either as it is or compressed based on what compression mode you decided before:

Here bytes are read in chunk in order to avoid large amount of memory allocation. You could read the whole file in one shot and store in a byte array same as the size of the file length. But I wanted to save memory allocation. Do a performance test to figure out if reading in 8K chunk is not the best approach for you.

Now you have the bytes to write to the response. Next step is to cache it and then deliver it.

Now the two functions that you have seen several times and have been wondering what they do. Here they are:

WriteResponse has no tricks, but ProduceResponseHeader has much wisdom in it. First it turns off response buffering so that ASP.NET does not store the written bytes in any internal buffer. This saves some memory allocation. Then it produces proper cache headers to cache the file in browser and proxy for 30 days, ensures proxy revalidate the file after the expiry date and also produces the Last-Modified date from the file's last write time in UTC.

How to use it

Get the HttpCompressionModule and StaticFileHandler from:

http://code.msdn.microsoft.com/fastmvc

Then install them in web.config. First you install the StaticFileHandler by removing the existing mapping for path="*" and then you install the HttpCompressionModule.

That's it! Enjoy a faster and more responsive ASP.NET MVC website deployed on IIS 6.0.

Share this post :

del.icio.us

digg

dotnetkicks

furl

live

spurl

technorati

yahoo

Reduce website download time by heavily compressing PNG and JPEG

omar — Mon, 07 Apr 2008 13:17:47 GMT

PNG and JPEG are two most popular formats for web graphics. JPEG is used for photographs, screenshots and backgrounds where PNG is used for all other graphics need including cliparts, buttons, headers, footers, borders and so on. As a result, these two types of graphics file usually take up 80% of the total graphics used in a website. Of course, there's the GIF, which is very popular. But as it supports only 256 colors, it is losing its popularity day by day. PNG seems to be a all rounder winner for all kinds of graphics need. As all browsers support PNG well enough and PNG supports alpha transparency, it's surely the best format so far on the web for all purpose graphics need for websites. So, if you can optimize all PNG and JPEG on your website and compress them rigorously, you can easily shed off several seconds of load time from your website without doing any coding. Especially if your website is graphics rich like Pageflakes, 30% reduction in total size of graphics throughout the website is a big performance win.

Optimize all PNG on your website

PNG has a lot of scope for optimization. Generally regular graphics tools like Photoshop, Paintshop pro, Paint.NET all generate PNG using a moderate compression. So, PNG can be compressed further by using advanced compression tools. OptiPNG is such a tool that can compress PNG and sometimes produce 50% smaller output. At Pageflakes, we have around 380 PNG which when compressed using OptiPNG, gives us 40% reduction in total size. This is a big win for us.

Here's what wikipedia says about OptiPNG:

OptiPNG is an open source command line computer program that reduces the size of PNG files. The compression is lossless, meaning that the resulting image will have exactly the same appearance as the source image.
The main purpose of OptiPNG is to reduce the size of the PNG IDAT data stream by trying various filtering and compression methods. It also performs automatic bit depth, color type and color palette reduction where possible, and can correct some data integrity errors in input files.

Here's a poweshell script that you can run from the root folder of your website. It will scan through all the PNG files in the webtree and run OptiPNG on each file. This takes quite some time if you have hundreds of files. So, you should make it a part of your build script.

gci -include *.png -recurse | foreach
 { $fileName = $_.FullName; cmd /c "C:\soft\png\optipng.exe -o7 `"$fileName`"" }

Here I have stored the optipng.exe on the c:\soft\png folder.

OptiPNG gives very good compression. But there's even more scope for compression. AdvanceCOMP is the ultimate in compression technology for PNG as it uses the mighty 7zip compression algorithm. It can squeeze down PNG even further after being compressed by OptiPNG using its maximum compression mode. PNG files are compressed using DEFLATE algorithm. DEFLATE has 0 to 9 compression level, where 9 is the highest. AdvanceCOMP uses 7zip DEFLATE encoder, that extends the compression factor even more. During 7zip compression, a much more detailed search of compression possibilities is performed, at the expense of significant further processor time spent on searching. Effectively, the 10-point scale used in gzip is extended to include extra settings above 9, the previous maximum search level. There will be no difference in decompression speed, regardless of the level of compressed size achieved or time taken to encode the data.

Here's a poweshell script that you can run from the root folder of your website. It will scan through all the PNG files in the webtree and run AdvanceCOMP on each file. You need to run AdvanceCOMP after running OptiPNG.

gci -include *.png -recurse | foreach
 { $fileName = $_.FullName; cmd /c "C:\soft\png\advpng.exe 
 --shrink-insane -z `"$fileName`"" }

I have collected both optipng and advpng in this zip file.

Optimize all JPEG on your website

Unfortunately, there's not much powerful tool like OptiPNG for jpeg that you can run on all your jpeg files and compress them rigorously. JPEG file is compressed when it is saved. Generally all graphics applications provide you an option to select what's the quality ratio of the jpeg being saved. So, you have to consciously make the best compression vs quality choice while saving the jpeg file. However, libjpeg project has a jpeg optimizer tool that does some optimization on jpeg files. It has a jpegtran utility which does the optimization according to wikipedia:

jpegtran - a utility for lossless transcoding between different JPEG formats. The jpegtran command-line program is useful to optimize the compression of a JPEG file, convert between progressive and non-progressive JPEG formats, eliminate non-standard application-specific data inserted by some image programs, or to perform certain transformations on a file — such as grayscaling, or rotating and flipping (within certain limits) — all done "losslessly" (i.e. without decompressing and recompressing the data, and so causing a reduction of image quality due to generation loss).

However, when we ran jpegtran on all the jpeg files in Pageflakes, we are able to reduce about 20% total size of all jpeg. So, that was not too bad.

Here's how you run jpegtran to get all the jpeg files within your website directory optimized:

gci -include *.jpg -recurse | foreach
 { $fileName = $_.FullName; $newFileName = $fileName + ".tmp"; 
cmd /c "C:\soft\jpeg\jpegtran.exe -optimize -outfile `"$newFileName`" `"$fileName`""; 
copy $newFileName $fileName; del $newFileName; }

The libjpeg binaries are uploaded here for your convenience.

Warning: You have to run all these powershell commands in a single line. I have broken the commands in multiple line for better readability.

Let's save global bandwidth, go green.

10 ASP.NET Performance and Scalability Secrets

omar — Wed, 30 Jan 2008 15:47:00 GMT

ASP.NET 2.0 has many secrets, when revealed, can give you big performance and scalability boost. For instance, there are secret bottlenecks in Membership and Profile provider which can be solved easily to make authentication and authorization faster. Furthermore, ASP.NET Http pipeline can be tweaked to avoid executing unnecessary code that gets hit on each and every request. Not only that, ASP.NET Worker Process can be pushed to its limit to squeeze out every drop of performance out of it. Page fragment output caching on the browser (not on the server) can save significant amount of download time on repeated visits. On demand UI loading can give your site a fast and smooth feeling. Finally, Content Delivery Networks (CDN) and proper use of HTTP Cache headers can make your website screaming fast when implemented properly. In this article, you will learn these techniques that can give your ASP.NET application a big performance and scalability boost and prepare it to perform well under 10 times to 100 times more traffic.

http://www.codeproject.com/KB/aspnet/10ASPNetPerformance.aspx

In this article I have shown the following techniques:

ASP.NET Pipeline optimization
ASP.NET Process configuration optimization
Things you must do for ASP.NET before going live
Content Delivery Network
Caching AJAX calls on browser
Making best use of Browser Cache
On demand progressive UI loading for fast smooth experience
Optimize ASP.NET 2.0 Profile provider
How to query ASP.NET 2.0 Membership tables without bringing down the site
Prevent Denial of Service (DOS) attack

The above techniques can be implemented on any ASP.NET website especially those who use ASP.NET 2.0's Membership and Profile provider.

You can learn a lot more about performance and scalability improvement of ASP.NET and ASP.NET AJAX websites from my book - Building a Web 2.0 portal using ASP.NET 3.5.

My first book - Building a Web 2.0 Portal with ASP.NET 3.5

omar — Sun, 13 Jan 2008 20:23:00 GMT

My first book "Building a Web 2.0 Portal with ASP.NET 3.5" from O'Reilly is published and available in the stores. This book explains in detail the architecture design, development, test, deployment, performance and scalability challenges of my open source web portal Dropthings.com. Dropthings is a prototype of a web portal similar to iGoogle or Pageflakes. But this portal is developed using recently released brand new technologies like ASP.NET 3.5, C# 3.0, Linq to Sql, Linq to XML, and Windows Workflow foundation. It makes heavy use of ASP.NET AJAX 1.0. Throughout my career I have built several state-of-the-art personal, educational, enterprise and mass consumer web portals. This book collects my experience in building all of those portals.

O'Reilly Website:
http://www.oreilly.com/catalog/9780596510503/

Amazon:
http://www.amazon.com/Building-Web-2-0-Portal-ASP-NET/dp/0596510500

Disclaimer: This book does not show you how to build Pageflakes. Dropthings is entirely different in terms of architecture, implementation and the technologies involved.

You learn how to:

Implement a highly decoupled architecture following the popular n-tier, widget-based application model
Provide drag-and-drop functionality, and use ASP.NET 3.5 to build the server-side part of the web layer
Use LINQ to build the data access layer, and Windows Workflow Foundation to build the business layer as a collection of workflows
Build client-side widgets using JavaScript for faster performance and better caching
Get maximum performance out of the ASP.NET AJAX Framework for faster, more dynamic, and scalable sites
Build a custom web service call handler to overcome shortcomings in ASP.NET AJAX 1.0 for asynchronous, transactional, cache-friendly web services
Overcome JavaScript performance problems, and help the user interface load faster and be more responsive
Solve various scalability and security problems as your site grows from hundreds to millions of users
Deploy and run a high-volume production site while solving software, hardware, hosting, and Internet infrastructure problems

If you're ready to build state-of-the art, high-volume web applications that can withstand millions of hits per day, this book has exactly what you need.

Making best use of cache for high performance website

omar — Thu, 29 Nov 2007 16:21:36 GMT

Use URLs consistently

Browsers cache content based on the URL. When URL changes, browser fetches a new version from origin server. URL can be changed by changing the query string parameters. For example, “/default.aspx” is cached on the browser. If you request “/default.aspx?123” it will fetch new content from server. Response from the new URL can also be cached in browser if you return proper caching headers. In that case, changing the query parameter to something else like “/default.aspx?456” will return new content from server. So, you need to make sure you use URL consistently everywhere when you want to get cached response. From homepage, if you have requested a file with URL “/welcome.gif”, make sure from another page you request the same file using the same URL. One common mistake is to sometimes omit the “www” subdomain from the url. www.pageflakes.com/default.aspx is not same as pageflakes.com/default.aspx. Both will be cached separately.

Cache static content for longer period

Static files can be cached for longer period like one month. If you are thinking that you should cache for couple of days so that when you change the file, users will pick it up sooner, you’re mistaken. If you update a file which was cached by Expires header, new users will immediately get the new file while old users will see the old content until it expires on their browser. So, as long as you are using Expires header to cache static files, you should use as high value as possible.

For example, if you have set expires header to cache a file for three days, one user will get the file today and store it in cache for next three days. Another user will get the file tomorrow and cache it for three days after tomorrow. If you change the file on the day after tomorrow, the first user will see it on fourth day and the second user will see it on fifth day. So, different users will see different versions of the file. As a result, it does not help setting a lower value assuming all users will pick up the latest file soon. You will have to change the url of the file in order to ensure everyone gets the exact same file immediately.

You can setup Expires header from static files from IIS Manager. You’ll learn how to do it in later section.

Use cache friendly folder structure

Store cached content under a common folder. For example, store all images of your site under the “/static” folder instead of storing images separately under different subfolders. This will help you use consistent URL throughout the site because from anywhere you can use “/static/images/somefile.gif”. Later on, we will learn it’s easier to move to a Content Delivery Network when you have static cacheable files under a common root folder.

Reuse common graphics files

Sometimes we put common graphics files under several virtual directories so that we can write smaller paths. For example, say you have indicator.gif in root folder, some subfolders and under CSS folder. You did it because you need not worry about paths from different places and you could just use the file name as relative URL. This does not help in caching. Each copy of the file is cached in browser separately. So, you should collect all graphics files in the whole solution and put them under the same root “static” folder after eliminating duplicates and use the same URL from all the pages and CSS files.

Change file name when you want to expire cache

When you want a static file to be changed, don’t just update the file because it’s already cached in user’s browser. You need to change the file name and update all references everywhere so that browser downloads the new file. You can also store the file names in database or configuration files and use data binding to generate the URL dynamically. This way you can change the URL from one place and have the whole site receive the change immediately.

Use a version number while accessing static files

If you do not want to clutter your static folder with multiple copies of the same file, you can use query string to differentiate versions of same file. For example, a GIF can be accessed with a dummy query string like “/static/images/indicator.gif?v=1”. When you change the indicator.gif, you can overwrite the same file and then update all references to the file to “/static/images/indicator.gif?v=2”. This way you can keep changing the same file again and again and just update the references to access the graphics using the new version number.

Store cacheable files in a different domain

It’s always a good idea to put static contents in a different domain. First of all, browser can open another two concurrent connections to download the static files. Another benefit is that you don’t need to send the cookies to the static files. When you put the static files on the same domain as your web application, browser sends all the ASP.NET cookies and all other cookies that your web application is producing. This makes the request headers be unnecessarily large and waste bandwidth. You don’t need to send these cookies to access the static files. So, if you put the static files in a different domain, those cookies will not be sent. For example, put your static files in www.staticcontent.com domain while your website is running on www.dropthings.com. The other domain does not need to be a completely different web site. It can just be an alias and share the same web application path.

SSL is not cached, so minimize SSL use

Any content that is served over SSL is not cached. So, you need to put static content outside SSL. Moreover, you should try limiting SSL to only secure pages like Login page or Payment page. Rest of the site should be outside SSL over regular HTTP. SSL encrypts request and response and thus puts extra load on server. Encrypted content is also larger than the original content and thus takes more bandwidth.

HTTP POST requests are never cached

Cache only happens for HTTP GET requests. HTTP POST requests are never cached. So, any kind of AJAX call you want to make cacheable, it needs to be HTTP GET enabled.

Generate Content-Length response header

When you are dynamically serving content via web service calls or HTTP handlers, make sure you emit Content-Length header. Browsers have several optimizations for downloading contents faster when it knows how many bytes to download from the response by looking at the Content-Length header. Browsers can use persisted connections more effectively when this header is present. This saves browser from opening a new connection for each request. When there’s no Content-Length header, browser doesn’t know how many bytes it’s going to receive from the server and thus keeps the connection open as long as it gets bytes delivered from the server until the connection closes. So, you miss the benefit of Persisted Connections that can greatly reduce download time of several small files like css, javascripts, and images.

How to configure static content caching in IIS

In IIS Manager, Web site properties dialog box has “HTTP Headers” tab where you can define Expires header for all requests that IIS handles. There you can define whether to expire content immediately or expire after certain number of days or on a specific date. The second option (Expire after) uses sliding expiration, not absolute expiration. This is very useful because it works per request. Whenever someone requests a static file, IIS will calculate the expiration date based on the number of days/months from the Expire after.

For dynamic pages that are served by ASP.NET, a handler can modify the expires header and override IIS default setting.

10 cool web development related articles in 2007

omar — Tue, 02 Oct 2007 17:26:00 GMT

Here's a list of 10 cool ASP.NET, AJAX and web development related articles and blog posts that I have written this year that you might want to take a look:

13 disasters for production website and their solutions
Talks about 13 production disasters that can happen to any website any time and bring down your business.

Build Google IG like Ajax Start Page in 7 days using ASP.NET Ajax and .NET 3.0
This block buster article shows how ASP.NET AJAX, Linq to XML, Linq to SQL and Workflow Foundation can be used to create a Google IG like start page in just 7 nights. Learn how to put together such hot technologies into one project and make a production site out of it.

Serve extensionless URL from ASP.NET without using ISAPI module or IIS 6 Wildcard mapping
Currently there are only two ways to service extentionless URL like www.pageflakes.com/omar that hits something besides the default document - use a custom ISAPI module or use IIS 6 wildcard mapping. Both has performance and scalability problems because both intercepts each and every hit. Learn how you can solve it by using a custom 404 handler.

Request format is unrecognized for URL unexpectedly ending in /SomeWebServiceMethod
Since ASP.NET AJAX 1.0 release, Microsoft prevented JSON hijacking by adding a special content type header. But this caused us some trouble.

Cleanup inactive anonymous users from ASP.NET Membership Tables
When you store anonymous user profile using ASP.NET Membership provider and Anonymous Identification provider, you soon end up with lots of idle anonymous user data where those users never come back. We (Pageflakes) went through a lot of difficulty keeping our database size down as we allow anonymous users to do almost everything that a registered user can do. This introduces scalability challenge. See how we solved this problem.

Prevent Denial of Service (DOS) attacks in your web application
Web applications can be brought down to its knees by hitting the site repeatedly or by calling expensive webservices randomly. Anyone can write a simple loop that hits a webserver very frequently from a high bandwidth connectivity and bring your production server down. See how to prevent such application level DOS attacks.

ASP.NET Ajax Extender for multi-column widget drag & drop
It's an ASP.NET AJAX extender that allows Pageflakes style drag & drop functionality between columns and rows.

ASP.NET Ajax in-depth performance analysis
While building an open source start page using ASP.NET AJAX, I have done a lot of performance analysis on AJAX framework in order to improve first time load and perceived speed of javascript rich pages. Check out my analysis.

Think you know how to write UPDATE statement? Think again.
Learn how to optimize common UPDATE statements

Make a surveillance application which captures desktop and then emails you as attachment
Some time back I needed to capture a certain computers desktop in order to find out what that user is doing every day. So, I made a .NET 2.0 Winforms Application which stays on system tray (optional) and capture the desktop in given time interval (say every 60 secs) and emailed the captured images to me as message attachment (say every 30 mins).

Today I received MVP award for the 3rd time on Visual C#. Thanks to Microsoft for the award and setting up my new blog. I will continue both my MVPS Blog and this blog from now on.

A significant part of sql server process memory has been paged out. This may result in performance degradation

omar — Wed, 19 Sep 2007 07:46:00 GMT

If you are using SQL Sever Server standard edition 64 bit on a Windows 2003 64bit, you will frequently encounter this problem where SQL Server says:

A significant part of sql server process memory has been paged out. This may result in performance degradation. Duration 0 seconds. Working set (KB) 25432, committed (KB) 11296912, memory utilization 0%

The number in working set and duration will vary. What happens here is SQL Server is forced to release memory to operating system because some other application or OS itself needs to allocate RAM.

We went through many support articles like:

918483: How to reduce paging of buffer pool memory in the 64-bit version of SQL Server 2005
905865: The sizes of the working sets of all the processes in a console session may be trimmed when you use Terminal Services to log on to or log off from a computer that is running Windows Server 2003
920739: You may experience a decrease in overall system performance when you are copying files that are larger than approximately 500 MB in Windows Server 2003 Service Pack 1

But nothing solved the problem. We still have the page out problem happening every day.

The server has 16 GB RAM where 12 GB is maximum limit allocated to SQL Server. 4 GB is left to OS and and other application. We have also turned off antivirus and any large backup job. 12 GB RAM should be plenty because there's no other app running on the dedicated SQL Server box. But the page out still happens. When this happens, SQL Server becomes very slow. Queries timeout, website throws error, transactions abort. Sometimes this problems goes on for 30 to 40 minutes and website becomes slow/unresponsive during that time.

I have found what causes SQL Server to page out. File System cache somehow gets really high and forces SQL Server to trim down.

You see the System cache resident bytes are very high. During this time SQL Server gets much less RAM than it needs. Queries timeout at very high rate like 15 per sec. Moreover, there's high SQL Lock Timeout/sec (around 15/sec not captured in screen shot).

SQL Server max memory is configured 12 GB. But here it shows it’s getting less than 8 GB.

While the file system cache is really high, there’s no process that’s taking significant RAM.

After I used SysInternal’s CacheSet to reset file system cache and set around 500 MB as max limit, memory started to free up.

SQL Server started to see more RAM free:

Then I hit the “Clear” button to clear file system cache and it came down dramatically.

Paging stopped. System cache was around 175 MB only. SQL Server lock timeout came back to zero. Everything went back to normal.

So, I believe there's either some faulty driver or the OS itself is leaking file system cache in 64bit environment.

What we have done is, we have a dedicated person who goes to production database servers every hour, runs the CacheSet program and clicks "Clear" button. This clears the file system cache and prevents it from growing too high.

There are lots of articles written about this problem. However, the most informative one I have found is from the SQL Server PSS team:

http://blogs.msdn.com/psssql/archive/2007/05/31/the-sql-server-working-set-message.aspx

UPDATE - THE FINAL SOLUTION!

The final solution is to run this program on Windows Startup:

SetSystemFileCacheSize 128 256

This sets the lower and higher limit for the System Cache. You need to run this on every windows startup because a restart will undo the cache setting to unlimited.

You can run the program without any parameter to see what is the current setting.

Download the program from this page:

http://www.uwe-sieber.de/ntcacheset_e.html

Go to the end and you will get the link to the SetSystemFileCacheSize.zip

What to do when you kill your database with your own hand

omar — Mon, 01 Jan 2007 11:21:00 GMT

Couple of months ago, we had an interesting day. It was a bright summer morning. We were all in the office doing our regular work. Developers writing codes, QA testing the site. We were all happy and smiling as there was no major bug. Some were drinking coffee and having nice chit chat. Our honorable Sys Admin came to office as usual late after noon. We all welcomed him remembering his day and night inhuman effort to keep our systems up and running fighting against germs and bacteria. As usual he started his day logging into servers one by one checking their health. He connected to our maintenance server via Remote Desktop. Did some routine check up and found that there was no space on hard drive. So, he decided to delete the database on the maintenance server as it was quite old. He started SQL Server Management Studio, entered sa and password, selected the database, pressed delete. As his habit, on the confirmation dialog he pressed enter without looking into it. Things went cool, database got deleted. But strangely the maintenance server did not free up any hard drive space.

In the meantime, there were sharp screams from other rooms. We all ran to see whether somebody fell down or had an electric shock or not. Seemed like every one was intact. But with horror, we looked into all the screens on our desktops and saw this:

This is our error handler page where users are redirected when there's any unknown catastrophic failure. This page appears when something goes wrong, really wrong. We do this by having the <customErrors> block in web.config:

< customErrors mode ="RemoteOnly" defaultRedirect ="GenericErrorPage.html" > < error statusCode ="403" redirect ="403.html" /> < error statusCode ="404" redirect ="404.html" /> </ customErrors >

If somebody fell down or had electric shock, we would not mind at all but this is generally beginning of a nightmare for us. Then we saw our Sys Admin was shaking and his face turned white out of blood. He somehow stood up trembling with fear and came to me and said, "My blood pressure is going sky high. I think I have deleted production database".

I told him, "no worries, I have done this before", of course on the outside. Let's not say what I felt inside. So, we first looked into Recycle Bin. Nope, no trace. It was a 30 GB file. There's no way Windows is going to store it on Recycle Bin. Then we searched for several undelete utilities both commercial ones and free ones. All failed on regular attempt. Some showed an advanced recovery can restore the file but it will take 2 hours of scanning. This will be disastrous if we go down for 2 hours. But we had no choice. Last backup we took was the previous day. If we restore it, we are going to lose thousands of users and their page setup. We will also lose hundreds of signups. So, we went for the 2 hours scan. After 2 hours, the scan reported it found the MDF file. But when we said recover it, it recovered a 0 byte file. We tried another product, same result. In the meantime 4 hours past by. So, we had to take a decision. We has no other choice but to restore the previous day's backup. So, we did, and the site went live.

Now it was time to find out all those poor souls who lost their data. We maintain a log file where we record all important activities like adding new page, registering, first visit etc. It was a tab delimited file like this:

01.01.2007 08:39:35 e5ca904c-0348-42cf-9d1b-6fb932ec930d Create Anonymous user if necessary 0,25
01.01.2007 08:39:36 e5ca904c-0348-42cf-9d1b-6fb932ec930d Get Page: __RSSFEED__ 0,109375
01.01.2007 08:39:36 e5ca904c-0348-42cf-9d1b-6fb932ec930d Load Modules in page #1413882 0,1875
01.01.2007 08:39:36 e5ca904c-0348-42cf-9d1b-6fb932ec930d GetPageflake 0,546875

It logs date and time of each action and also the duration of the process. This log helps us identify slow running operations and we can see almost in real time what's going on with the site.

After importing this log into database, we ran a query to find out user name of all users who signed up since database delete. Here, we discovered something interesting. We were expecting the date and time will be in server's own time. But Enterprise Library logs are always in GMT. So, we had to figure out what is the time difference from regional settings of the servers. Then we subsctracted it to match GMT. Then we calculated when the database was deleted and started finding the users after that. We got a list of email addresses whose signup was lost, almost around 400. We were lucky it happened on a weekend, so there was not much signup that day. Then we sent that email list to our marketing team and they sent apology letters to those users.

So, what we did wrong which you should make sure you never do:

Sys Admin became too comfortable with the servers. There was lack of seriousness while working on remote desktop. It became routine monotonous absent minded work to him. This is a real problem with sys admins. On first month, you will see him very serious about his role. Everytime he logs into remote desktop on production or maintenance servers, there's a considerable amount of curves on his forehead. But day by day, it reduces and he starts working on production server as if he is working on his own laptop. At some point, someone needs to make him realize what is the gravity of his actions. He should wash his hands before sitting in front of remote desktop (or perform wudhu if he is muslim) and then say his prayer: "O Lord! I am going to work on remote desktop. Grant me tranquility and absolute concentration and protect me from the devil who whispers in my soul foul words and lures me to cause great harm to production servers"
All databases had the same "sa" password. If we had different password, at least while typing the password, sys admin could realize where he is connecting to. Although he did connect to remote desktop on maintenance server, but from SQL Server Management Studio, he connected to primary database server as he did last time. SQL Server Management Studio remembered the last machine name and user name. So, all he did was enter password and hit enter and delete the database. Now we have put the server name inside the password. So, while typing the password, we know conciously which server we are going to connect.
Don't ignore confirmation dialogs on remote desktops as you do on your local machine. Nowadays, we consider ourselves super expert on everything and never read the confirmation dialog. I myself don't remember when was the last time I read any confirmation dialog seriously. Definitely this attitude must change while working on servers. When Sys Admin tried to delete the database, there was a confirmation that there are active connections on the database. SQL Server tried its best to inform him that this is a database being used and don't delete it, please. But as he does hundred times per day on his laptop, clicked OK without reading the confirmation dialog.
Don't put same administrator password on all servers. This makes life easier while copying files from one server to another, but don't do it. You will accidentally delete file on another server just like we do.
DO NOT use Administrator user account to do your day to day work. We started using a Power User account for our day to day operation which has limited access on couple of folders only. Using Administrator account on remote desktop means you are opening doors to all possible accidents to happen. If you use a restricted account, there's no possibility of such accidents.
Always have someone beside you when you work on production server and do something important like cleaning up free space or running scripts, restoring database etc. Make sure the other person is not sleeping on his chair beside you.

How to setup SQL Server 2005 Transaction Log Ship on large database that really works

omar — Fri, 15 Sep 2006 10:01:00 GMT

I tried a lot of combinations in my life in order to find out an effective method for implementing Transaction Log Shipping between servers which are in a workgroup, not under domain. I realized the things you learn from article and books are for small and medium sized databases. When you database become 10 GB or bigger, thing's become a lot harder than it looks. Additionally many things changed in SQL Server 2005. So, it's even more difficult to configure log shipping properly nowadays.

Here's the steps that I finally found that works. Let's assume there are 2 servers with SQL Server 2005. Make sure both servers have latest SP. There's Service Pack 1 released already.

1. Create a new user Account named "SyncAccount" on both computers. Use the exact same user name and password.

2. Make sure File Sharing is enabled on the local area connection between the server. Also enable file sharing in Firewall.

3. Make sure the local network connection is not regular LAN. It must be a gigabit card with near zero data corruption. Both cable and switch needs to be perfect. If possible, connect both servers using Fibre optic cable directly on the NIC in order to avoid a separate Switch.

4. Now create a folder named "TranLogs" on both servers. Let's assume the folder is on E:\Tranlogs.

5. On Primary Database server, share the folder "Tranlogs" and allow SyncAccount "Full Access" to it. Then allow SyncAccount FullAccess on TranLogs folder. So you are setting the same permission from both "Sharing" tab and from "Security" tab.

6. On Secondary database server, allow SyncAccount "Full Access" right on TranLogs folder. No need to share it.

7. Test whether SyncAccount can really connect between the servers. On Secondary Server, go to Command Prompt and do this:

9. Now you have a command prompt which is running with SyncAccount privilege. Let's confirm the account can read and write on "TranLog" shares on both servers.

10.

11. This is exactly what SQL Agent will be doing during log ship. It will copy log files from primary server's network share to it's own log file folder. So, the SyncAccount needs to be able to both read files from primary server's network share and write onto its own tranlogs folder. The above test verifies the result.

12. This is something new in SQL Server 2005: Add SyncAccount in SQLServer Agent group "SqlServer2005SqlAgentUser$....". You will find this Windows User Group after installing SQL Server 2005.

13. Now go to Control Panel->Administrative Tools->Services and find the SQL Server Agent service. Go to its properties and set SyncAccount as the account on the Logon tab. Restart the service. Do this on both servers.

14.

15. I use sa account to configure the log shipping. So, do this on both servers:

a. Enable "sa" account. By default, sa is disabled in SQL Server 2005.

b. On "sa" account turn off Password Expiration Policy. This prevents sa password from expiring automatically.

16. On Secondary server, you need to allow remote connections. By default, SQL Server 2005 disables TCP/IP connection. As a result, you cannot login to the server from another server. Launch the Surface Area Configuration tool from Start->Programs->MS SQL Server 2005 and go to "Remote Connection" section. Choose the 3rd option which allows both TCP/IP based remote connection and local named pipe based connections.

17. On Secondary Server firewall, open port 1433 so that primary server can connect to it.

18. Restart SQL Server. Yes, you need to restart SQL Server.

18. On Primary server, go to Database properties->Options and set Recovery Model to "Full". If it was already set to full before, it will be wise to first set it to Simple, then shrink the transaction log file and then make it "Full" again. This will truncate the transaction log file for sure.

19. Now take a Full Backup of the database. During backup, make sure you put the backup file on a physically separate hard drive than the drive where MDF is located. Remember, not different logical drives, different physical drives. So, you should have at least 2 hard drives on the server. During backup, SQL Server reads from MDF and writes on the backup file. So, if both MDF and the backup is done on the same hard drive, it's going to take more than double the time to backup the database. It will also keep the Disk fully occupied and server will become very slow.

20. After backup done, RAR the database. This ensures when you copy the database to the other server there's no data corruption while the file was being transferred. If you fail to unRAR the file on the secondary server, you get assurance that there's some problem on the network and you must replace network infrastructure. The RAR also should be done on a separate hard drive than the one where the RAR is located. Same reason, read is on one drive and write is on another drive. Better if you can directly RAR to the destination server using network share. It has two benefits:

a. Your server's IO is saved. There's no write, only read.

b. Both RAR and network copy is done in one step.

21.

22. By the time you are done with the backup, RAR, copy over network, restore on the other server, the Transaction Log file (LDF) on the primary database server might become very big. For us, it becomes around 2 to 3 GB. So, we have to manually take a transaction log backup and ship to the secondary server before we configure Transaction Log Shipping.

23.

24. When you are done with copying the transaction log backup to the second server, first restore the Full Backup on the secondary server:

25.

26. But before restoring, go to Options tab and choose RESTORE WITH STANDBY:

27.

28. When the full backup is restored, restore the transaction log backup.

29. REMEMBER: go to options tab and set the Recovery State to "RESTORE WITH STANDBY" before you hit the OK button.

30. This generally takes a long time. Too long in fact. Every time I do the manual full backup, rar, copy, unrar, restore, the Transaction Log (LDF) file becomes 2 to 3 GB. As a result, it takes a long time to do a transaction log backup, copy and restore and it takes more than an hour to restore it. So, within this time, the log file on the primary server again becomes large. As a result, when log shipping starts, the first log ship is huge. So, you need to plan this carefully and do it only when you have least amount of traffic.

31. I usually have to do this manual Transaction Log backup twice. First one is around 3 GB. Second one is around 500 MB.

32. Now you have a database on the secondary server ready to be configured for Log shipping.

33. Go to Primary Server, select the Database, right click "Tasks" -> "Shrik". Shrink the Log File.

34. Go to Primary server, bring on Database options, go to Transaction Log option and enable log shipping.

35.

36. Now configure the backup settings line this:

37.

38. Remember, the first path is the network path that we tested from command prompt on the secondary server. The second path is the local hard drive folder on the primary server which is shared and accessible from the network path.

39. Add a secondary server. This is the server where you have restored the database backup

40.

41. Choose "No, the secondary database is initialized" because we have already restored the database.

42. Go to second tab "Copy Files" and enter the path on the secondary server where log files will be copied to. Note: The secondary server will fetch the log files from the primary server network share to it's local folder. So, the path you specify is on the secondary server. Do not get confused from the picture below that's it's the same path as primary server. I just have same folder configuration on all servers. It can be D:\tranlogs if you have the tranlogs folder on D: drive on secondary server.

43.

44. On third tab, "Restore Transaction Log" configure it as following:

45.

46. It is very important to choose "Disconnect users in database…". If you don't do this and by any chance Management Studio is open on the database on secondary server, log shipping will keep on failing. So, force disconnect of all users when database backup is being restored.

47. Setup a Monitor Server which will automatically take care of making secondary server the primary server when your primary server will crash.

48.

49. In the end, the transaction log shipping configuration window should look like this:

50.

51. When you press OK, you will see this:

52. Do not be happy at all if you see everything shows "Success". Even if you did all the paths, and settings wrong, you will still see it as successful. Login to the secondary server, go to SQL Agents->Jobs and find the Log Ship restore job. If the job is not there, your configuration was wrong. If it's there, right click and select "View History". Wait for 15 mins to have one log ship done. Then refresh and see the list. If you see all OK, then it is really ok. If not, then there are two possibilities:

a. See if the Log Ship Copy job failed or not. If it fails, then you entered incorrect path. There can be one of the following problem:

The network location on primary server is wrong
The local folder was specified wrong
You did not set SyncAccount as the account which runs SQL Agent or you did but forgot to restart the service.

b. If restore fails, then the problems can be one of the following:

i. SyncAccount is not a valid login in SQL Server. From SQL Server Management Studio, add SyncAccount as a user.

ii. You forgot to restore the database on secondary server as Standby.

iii. You probably took some manual transaction log backup on the primary server in the meantime. As a result, the backup that log shipping took was not the right sequence.

53. If everything's ok, you will see this:

Cost effective, failsafe, 2 server load balanced web & DB servers

omar — Mon, 28 Aug 2006 06:53:00 GMT

We were very poor when we first started Pageflakes so we needed a really cheap but fully reliable deployment configuration. The requirement was:

Fail safe. If a server goes down, another one can take over without users noticing
Database must be mirrored in multiple servers
Web servers will be in Load Balanced mode so that incase one goes down, the other takes over immediately
Do it with least hardware and cost

So, we thought a lot and could not make a reliable solution without at least three servers. Then we found one with two servers only. Here's the plan:

We made the above plan for our hosting service provider.

We have 2 windows server both with IIS 6.0 and SQL Server 2005. Let's call these servers Web server and DB server.

Web server gets 60% web traffic configured via Network Load Balancing. We used Windows Network Load Balancing in order to save the cost of buying a separate Load balancer. We also used Windows Firewall instead of external firewall. You can imagine how poor we were. DB Server gets 40% web traffic and hosts the database in its SQL Server 2005. We started with SQL Server 2005 Workgroup Edition because it was the only version we could afford. So, we could not use the new DB Mirroring feature instead we had to use Log Shipping. The web server also had SQL Server 2005 but it was used for a standby database so we did not have to pay for licensing fee. Remember: You do not need to pay for SQL server license if the server is hosting standby databases only. The DB Server also runs the web site in its IIS.

So, we have 2 servers running the website in NLB. So, web server is properly load balanced and failsafe. Then we have database on the DB Server which is log shipped to the web server. So, incase DB server goes down, we can divert all traffic to the web server and bring up web server's standby database in order to run the site solely from web server. When DB server comes back online, we configure log shipping the opposite way and divert most of the traffic to the DB server. Thus DB server becomes the web server and previously web server becomes the DB server. It requires some manual work and not fully automated. But the cheapest solution for a reliable configuration.

So when you add more servers in future, it's very easy to add new ones.