Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Windows Vista - built-in disk partition resize utility

When I tried to install windows vista on my computer, the first decision I had to take was to partition the hard disk. I started looking for software on the Internet that can resize existing disk partition. I found some free software but none of them looked convincing enough to install and try on my computer. So I just decided to go on and install Windows Vista anyway. I had two partitions on my computer. The first partition was the C drive which had windows XP installed on it and the other partition was D drive which had all my data. There was no way I can install Windows Vista on C drive.  So, I installed Windows Vista on D drive. After installing Windows Vista I went to the computer manager by right cliking on the "computer" icon and selecting and "Manage". There you will find the Disk Management option. From there, select the D drive and choose "Shrink". It will shrink the parition to all available space. This will take a long time to complete without any notification. So, don't think Vista has hung. Let it finish and you will find D has been shrunk and there's a new parition.

Now move all your data from D drive to the new drive, say E drive.

If is better to do a disk defragmentation on D drive after doing this.

Note: I found this half written post in my draft box after 7 months. So, this may not be a new information. Just to let you know, for my regular development, I went back to Windows XP because there's significant difference between IIS 6 and IIS 7 and I need to see how IIS 6 behaves with my applications. Moreover, Visual Studio 2005 UI does not mix well with Vista. Since I went back, somehow XP actually feels quite light and faster than Vista. Finally, Vista burns out laptop battery within 40 mins where XP can withstand up to 1.5 hours. This can be because of Sony Advance Power Management drivers not working with Vista.

A significant part of sql server process memory has been paged out. This may result in performance degradation

If you are using SQL Sever Server standard edition 64 bit on a Windows 2003 64bit, you will frequently encounter this problem where SQL Server says:

A significant part of sql server process memory has been paged out. This may result in performance degradation. Duration 0 seconds. Working set (KB) 25432, committed (KB) 11296912, memory utilization 0%

The number in working set and duration will vary. What happens here is SQL Server is forced to release memory to operating system because some other application or OS itself needs to allocate RAM.

We went through many support articles like:

• 918483: How to reduce paging of buffer pool memory in the 64-bit version of SQL Server 2005
• 905865: The sizes of the working sets of all the processes in a console session may be trimmed when you use Terminal Services to log on to or log off from a computer that is running Windows Server 2003
• 920739: You may experience a decrease in overall system performance when you are copying files that are larger than approximately 500 MB in Windows Server 2003 Service Pack 1

But nothing solved the problem. We still have the page out problem happening every day.

The server has 16 GB RAM where 12 GB is maximum limit allocated to SQL Server. 4 GB is left to OS and and other application. We have also turned off antivirus and any large backup job. 12 GB RAM should be plenty because there's no other app running on the dedicated SQL Server box. But the page out still happens. When this happens, SQL Server becomes very slow. Queries timeout, website throws error, transactions abort. Sometimes this problems goes on for 30 to 40 minutes and website becomes slow/unresponsive during that time.

I have found what causes SQL Server to page out. File System cache somehow gets really high and forces SQL Server to trim down.

You see the System cache resident bytes are very high. During this time SQL Server gets much less RAM than it needs. Queries timeout at very high rate like 15 per sec. Moreover, there's high SQL Lock Timeout/sec (around 15/sec not captured in screen shot).

SQL Server max memory is configured 12 GB. But here it shows it’s getting less than 8 GB.

While the file system cache is really high, there’s no process that’s taking significant RAM.

After I used SysInternal’s CacheSet to reset file system cache and set around 500 MB as max limit, memory started to free up.

SQL Server started to see more RAM free:

Then I hit the “Clear” button to clear file system cache and it came down dramatically.

Paging stopped. System cache was around 175 MB only. SQL Server lock timeout came back to zero. Everything went back to normal.

So, I believe there's either some faulty driver or the OS itself is leaking file system cache in 64bit environment.

What we have done is, we have a dedicated person who goes to production database servers every hour, runs the CacheSet program and clicks "Clear" button. This clears the file system cache and prevents it from growing too high.

There are lots of articles written about this problem. However, the most informative one I have found is from the SQL Server PSS team:

http://blogs.msdn.com/psssql/archive/2007/05/31/the-sql-server-working-set-message.aspx

UPDATE - THE FINAL SOLUTION!

The final solution is to run this program on Windows Startup:

SetSystemFileCacheSize 128 256

This sets the lower and higher limit for the System Cache. You need to run this on every windows startup because a restart will undo the cache setting to unlimited.

You can run the program without any parameter to see what is the current setting.

Download the program from this page:

http://www.uwe-sieber.de/ntcacheset_e.html

Go to the end and you will get the link to the SetSystemFileCacheSize.zip

Request format is unrecognized for URL unexpectedly ending in /SomeWebServiceMethod

At Pageflakes, our webservers; event logs are being flooded with this error. In ASP.NET AJAX 1.0 version, Microsoft added a check for all web service calls to have Content-Type: application/json in the request headers. Unless this request header is present, ASMX handler fires an exception. This exception is raised directly from the ScriptHandler which handles all web service calls made via ASP.NET AJAX.

This is done for security reason. This prevents anyone from feeding off from your webservices. For example, you might have a webservice that returns some useful information that others might be interested to consume. So, any one could just add a <script> tag pointing to that web service URL and get the json. If that webservice is a very expensive webservice in terms of I/O and/or CPU, then other websites feeding off from your webservice could easily bog down your server.

Now, this back fires when you have HTTP GET supported webservice calls that produce response headers to cache the response. For example, you might have a webmethod that returns Stock Quotes. You have used response caching so that browser caches the response of that webmethod and repeated visit do not produce repeated calls to that I/O costly webservice. What will happen is proxy gateways or proxy servers will see that their client users are making this request frequently and it can be cached. So, they will make periodic calls to that webservice and try to precache them on behalf of their client users. However, during precache, they won't send the Content-Type: application/json header. That's what I have seen for several proxy servers. As a result, it produces exception and you get your event log flooded with this exception.

The reason why MS might have not seen this or anyone else is because there's no way to make HTTP GET response cacheable on browser from webservice calls unless you do the hack I mentioned in earlier blog post.

However, one thing that puzzles me is we get this error so frequently and the request headers look so legitimate that I am not 100% sure whether it's the proxies not producing the content-type header properly all the time. Somehow the error patterns look like an registered user is trying to use the site and failing. It is possible that some firewall might see this content-type as invalid content type and filter out the header.