Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 : powershell

Reduce website download time by heavily compressing PNG and JPEG

omar — Mon, 07 Apr 2008 13:17:47 GMT

PNG and JPEG are two most popular formats for web graphics. JPEG is used for photographs, screenshots and backgrounds where PNG is used for all other graphics need including cliparts, buttons, headers, footers, borders and so on. As a result, these two types of graphics file usually take up 80% of the total graphics used in a website. Of course, there's the GIF, which is very popular. But as it supports only 256 colors, it is losing its popularity day by day. PNG seems to be a all rounder winner for all kinds of graphics need. As all browsers support PNG well enough and PNG supports alpha transparency, it's surely the best format so far on the web for all purpose graphics need for websites. So, if you can optimize all PNG and JPEG on your website and compress them rigorously, you can easily shed off several seconds of load time from your website without doing any coding. Especially if your website is graphics rich like Pageflakes, 30% reduction in total size of graphics throughout the website is a big performance win.

Optimize all PNG on your website

PNG has a lot of scope for optimization. Generally regular graphics tools like Photoshop, Paintshop pro, Paint.NET all generate PNG using a moderate compression. So, PNG can be compressed further by using advanced compression tools. OptiPNG is such a tool that can compress PNG and sometimes produce 50% smaller output. At Pageflakes, we have around 380 PNG which when compressed using OptiPNG, gives us 40% reduction in total size. This is a big win for us.

Here's what wikipedia says about OptiPNG:

OptiPNG is an open source command line computer program that reduces the size of PNG files. The compression is lossless, meaning that the resulting image will have exactly the same appearance as the source image.
The main purpose of OptiPNG is to reduce the size of the PNG IDAT data stream by trying various filtering and compression methods. It also performs automatic bit depth, color type and color palette reduction where possible, and can correct some data integrity errors in input files.

Here's a poweshell script that you can run from the root folder of your website. It will scan through all the PNG files in the webtree and run OptiPNG on each file. This takes quite some time if you have hundreds of files. So, you should make it a part of your build script.

gci -include *.png -recurse | foreach
 { $fileName = $_.FullName; cmd /c "C:\soft\png\optipng.exe -o7 `"$fileName`"" }

Here I have stored the optipng.exe on the c:\soft\png folder.

OptiPNG gives very good compression. But there's even more scope for compression. AdvanceCOMP is the ultimate in compression technology for PNG as it uses the mighty 7zip compression algorithm. It can squeeze down PNG even further after being compressed by OptiPNG using its maximum compression mode. PNG files are compressed using DEFLATE algorithm. DEFLATE has 0 to 9 compression level, where 9 is the highest. AdvanceCOMP uses 7zip DEFLATE encoder, that extends the compression factor even more. During 7zip compression, a much more detailed search of compression possibilities is performed, at the expense of significant further processor time spent on searching. Effectively, the 10-point scale used in gzip is extended to include extra settings above 9, the previous maximum search level. There will be no difference in decompression speed, regardless of the level of compressed size achieved or time taken to encode the data.

Here's a poweshell script that you can run from the root folder of your website. It will scan through all the PNG files in the webtree and run AdvanceCOMP on each file. You need to run AdvanceCOMP after running OptiPNG.

gci -include *.png -recurse | foreach
 { $fileName = $_.FullName; cmd /c "C:\soft\png\advpng.exe 
 --shrink-insane -z `"$fileName`"" }

I have collected both optipng and advpng in this zip file.

Optimize all JPEG on your website

Unfortunately, there's not much powerful tool like OptiPNG for jpeg that you can run on all your jpeg files and compress them rigorously. JPEG file is compressed when it is saved. Generally all graphics applications provide you an option to select what's the quality ratio of the jpeg being saved. So, you have to consciously make the best compression vs quality choice while saving the jpeg file. However, libjpeg project has a jpeg optimizer tool that does some optimization on jpeg files. It has a jpegtran utility which does the optimization according to wikipedia:

jpegtran - a utility for lossless transcoding between different JPEG formats. The jpegtran command-line program is useful to optimize the compression of a JPEG file, convert between progressive and non-progressive JPEG formats, eliminate non-standard application-specific data inserted by some image programs, or to perform certain transformations on a file — such as grayscaling, or rotating and flipping (within certain limits) — all done "losslessly" (i.e. without decompressing and recompressing the data, and so causing a reduction of image quality due to generation loss).

However, when we ran jpegtran on all the jpeg files in Pageflakes, we are able to reduce about 20% total size of all jpeg. So, that was not too bad.

Here's how you run jpegtran to get all the jpeg files within your website directory optimized:

gci -include *.jpg -recurse | foreach
 { $fileName = $_.FullName; $newFileName = $fileName + ".tmp"; 
cmd /c "C:\soft\jpeg\jpegtran.exe -optimize -outfile `"$newFileName`" `"$fileName`""; 
copy $newFileName $fileName; del $newFileName; }

The libjpeg binaries are uploaded here for your convenience.

Warning: You have to run all these powershell commands in a single line. I have broken the commands in multiple line for better readability.

Let's save global bandwidth, go green.

Automate website deployment using Powershell

omar — Fri, 15 Dec 2006 08:58:00 GMT

I published an article which automates website deployment.

http://www.codeproject.com/dotnet/DeploySite.asp

The Powershell script does the following for you:

· Maintains different configuration information for different deployments. For example, different connection strings for development servers and production servers (one or more production servers).

· Creates a deployment folder using the deployment date, time, and version so that you have a separate folder for each deployment and can keep track of things deployed on a day, e.g., 20061214-1.

· Copies only the change files and some predefined files to the deployment folder. So, you don't deploy the whole website every day.

· Copies the web.config and customizes the <appSettings>, <connectionString>, <assemblies> etc., as per the deployment configuration. For example, you can have different connection strings for different servers.

· Updates all JavaScript files with a version number so that in every deployment, a new file gets downloaded by client browsers.

· Updates default.aspx automatically with the modified script file name.

· Compresses all JavaScript files that gets deployed.

· Compresses all static HTML files using an Absolute HTML Optimizer.

· Creates a zip file which contains the deployment package.

· FTP the zip file to a target server.

So all you need to do is, run the script, extract the zip on server and voila!

If you like it, please vote for me.

Generate Javascript string from HTML using Power Shell script

omar — Sun, 06 Aug 2006 17:29:33 GMT

Say you want to generate a javascript representation of HTML fragment so that you can use document.write(‘’) or use element.innerHTML to inject it in the document dynamically. You need to do the following steps in order to convert HTML to BLOCKED SCRIPT

Bring HTML in one line
Clear unnecessary spaces
Replace all quote with backslash quote (\’)
Replace all backslash with double backslash (\\)

Here’s a power shell script which does it for you:

1 ###################################

2 # Make JS string out of HTML

3 #

4 # Takes a HTML fragment and converts

5 # it to a JS representation so that

6 # the JS can be used in innerHTML or

7 # document.write

8 ###################################

9 Param( [string]$filePath, [string]$outputPath, [Boolean]$generateDocumentWrite=[Boolean]0 )

11 $streamReader = new-object System.IO.StreamReader($filePath, [Boolean]"True");

12 $fileContent = $streamReader.ReadToEnd();

13 $streamReader.Close();

14 $streamReader.Dispose();

16 $fileContent=$fileContent.Replace("`n", "").Replace("`r", "").Replace("`t", " ").Replace(" ", " ").Replace(" ", " ").Replace(" ", " ").Replace("\", "\\").Replace("'", "\'");

18 if( [System.IO.File]::Exists($outputPath) ) { [System.IO.File]::Delete( $outputPath ); }

20 $streamWriter = new-object System.IO.StreamWriter($outputPath, [Boolean]"True");

22 if( $generateDocumentWrite)

23 {

24 $streamWriter.Write("document.write('");

25 $streamWriter.Write($fileContent);

26 $streamWriter.Write("');");

27 }

28 else

29 {

30 $streamWriter.Write($fileContent);

31 }

32 $streamWriter.Close();

33 $streamWriter.Dispose();

Just run the script by passing a input file name and output file name and optionally a “1” in order to generate document.write(‘<converted js>’) statement.

Using Power Shell to automate incremental updates to websites

omar — Sun, 06 Aug 2006 17:10:00 GMT

Here’s something we used to do manually every time we had to deploy a patch to production server:

Get all the files changed since last deployment date & time from latest code base that has been tested well enough
Copy them to a folder which is dated.
Update the web.config using Production server configuration
Update some other settings in <appSettings> inside web.config
Run through all the HTML files and compress them
Run JSMIN on all JS files
Increase version number in all external <SCRIPT src=”…”> tags. Increase only those which have been changed
Zip the entire folder
FTP it
Extract the folder on production server and overwrite the files

Now doing all these manually is pretty difficult and there’s always change of human error. Thanks to Power Shell, we have automated the entire process up to FTP and copy on Production Server. We have a script named “deploy.ps1” which takes a date time, server name and a version number, and does all the preparation for us and gives us a nice zip file to upload. The deployment script can prepare zip files for our developer testing server, for beta server and for individual production server in NLB.

Let me show you how the file looks like:

The file starts with some header and constants:

#************************************************************

# (C) Omar AL Zabir Pageflakes Ltd.

# Create a deployment compressed file which

# contains the latest files after the given

# date.

# 1. Find all the files after the specified date

# 2. Create a deployment folder using the date time

# 3. Copy all the changed files to the deployment folder

# 4. Copy web.config and modify it according to settings

# 5. JSMin the script files, if any

# 6. Update default.aspx with latest version of Framework1-###.js & Framework2-###.js

# 7. Use Absolute HTML Compressor to compress all HTML files

# 8. Create a zip file

#************************************************************

Param ([DateTime] $lastDatetime = [DateTime]::Now.AddDays(-1), $SERVER = "www", $VERSION = 1)

The script takes a date time after which modified files are copied to a deployment directory. It also takes the server name and a version number.

if( $SERVER -eq "nlb1" )

{

$DB_HOST = "(local)";

$DB_NAME = "PageFlakesDB";

$CREDENTIAL = "...";

$EXTERNAL_SITE_PREFIX = "http://www.pageflakes.com/";

$EMAILTEMPLATE_FOLDER = "d:\www\EmailTemplates\";

}

These are some constants that we use in web.config. As these settings are purely server and site specific, we decide what will be their value based on the server type specified.

After this comes some path constants to files and folders:

$CONNECTION_STRING = "data source=" + $DB_HOST + ";initial catalog=" + $DB_NAME + ";" + $CREDENTIAL;

$JSMIN_PATH = "jsmin.exe"

$HTML_COMPRESSOR_PATH = "ahc.exe"

$HTML_COMPRESSOR_PARAMS = "-y -s -l -w -t -c -d -b- -m`"*.htm;*.html`"";

$HOSTNAME = "ftp://69.41.181.68";

$TARGET = "/" + $SERVER + "/updates/";

$WEB_PROJECT_PATH = "..\src\EarlyPrototypes";

$DEPLOYMENT_PATH = "..\Deployments\" + $SERVER;

$FRAMEWORK1_FILENAME = "Framework1.js";

$FRAMEWORK2_FILENAME = "Framework2.js";

$DEFAULT_ASPX = "Default.aspx";

These paths are determined by the folder where we have the deploy.ps1 file located and where the web site code and deployment directory is located. We currently have the following folder structure in our local code base:

Pageflakes\

Src\WebSite

Deployments

Scripts\

Deploy.ps1

So, all the paths are relative to the deploy.ps1 file.

Now we determine the absolute path of locations from their relative path:

$srcPath = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine( $pwd.ToString(), $WEB_PROJECT_PATH ) );

$destinationFolder = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine( $pwd.ToString(), $DEPLOYMENT_PATH ));

$pwd is an internal variable in Power Shell which gives you current path.

Then we create a folder for deployment based on current date time. The folder name is derived in yyyyMMdd format. Which means if we are deploying on February 31 ^st, 2006, the folder name will be 20060231.

$deployFolder = [DateTime]::Now.ToString("yyyyMMdd") + "-" + $VERSION;

$deployZipFileName = $deployFolder + ".zip";

Before starting to copy the file, we first delete the deployment folder if it already exists:

if( [System.IO.Directory]::Exists($deployPath) ){ remove-item -Path $deployPath -recurse; }

remove-item with the –recurse is equivalent to the deltree.exe we had during DOS days. Unfortunately there’s been no equivalent to deltree since Windows 98. But now with Power Shell you can delete a directory along with all subdirectory and files using remove-item –recurse.

In order to zip files to a zip file, we use ICSharpCode.SharpZipLib.dll from SharpZip. Here’s how you can load an external DLL and instantiate a class using powershell:

$zipLibraryPath = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine( $pwd.ToString(), "ICSharpCode.SharpZipLib.dll" ));

[void][System.Reflection.Assembly]::LoadFile($zipLibraryPath);

$zip = new-object ICSharpCode.SharpZipLib.Zip.FastZip;

Now we will copy all the changed files after a specific date:

"Copying files to deployment folder..."

# Get the files which were changed after the $lastDateTime

$changedFiles = get-childitem $srcPath -exclude "*.log" -Recurse | where { $_ -is [System.IO.FileInfo] -and $_.LastWriteTime -ge $lastDateTime -or $_.Name -eq "Default.aspx" }

# Get each file and copy the file to the proper destination directory

foreach( $file in $changedFiles )

{

[string]$filePath = $file.ToString();

[string]$relativePath = $file.DirectoryName.substring($srcPath.Length);

if( -not $relativePath.StartsWith("\users\") )

{

# if the relative path contains a subdirectory, then create a subdirectory under the deploy path

[string]$copyPath = [System.IO.Path]::Combine( $deployPath, $relativePath.TrimStart('\') );

if( ![System.IO.Directory]::Exists($copyPath) )

{

$newDir = [System.IO.Directory]::CreateDirectory($copyPath);

}

copy $filePath $copyPath;

}

"Copy complete."

The above script copies changed files from one folder to a destination folder recursively. Here’s the line that selects the change file:

$changedFiles = get-childitem $srcPath -exclude "*.log" -Recurse | where { $_ -is [System.IO.FileInfo] -and $_.LastWriteTime -ge $lastDateTime -or $_.Name -eq "Default.aspx" }

Note: if we do not put the filter “$_ -is [System.IO.FileInfo]” then directories also get selected.

Now’s a fun part, how do you modify web.config using Power Shell? Do you know how easy it is to work with Xml files in PowerShell? See this:

"Configuring web.config..."

$webConfigFilePath = [System.IO.Path]::Combine( $deployPath, "web.config" );

if( [System.IO.File]::Exists( $webConfigFilePath ) )

{

" Web config found";

[System.Xml.XmlDocument]$doc = new-object System.Xml.XmlDocument;

$doc.Load($webConfigFilePath);

$root = $doc.get_DocumentElement();

$root."system.web".compilation.debug = "false";

$root.connectionStrings.add.connectionString = $CONNECTION_STRING

foreach( $item in $root.appSettings.add )

{

if( $item.key -eq "SitePrefix" )

{ $item.Value = $EXTERNAL_SITE_PREFIX; }

if( $item.key -eq "InternalSitePrefix" ) .

}

foreach( $item in $root."system.web".compilation.assemblies.add )

{

if( $item.assembly.Contains("nunit") )

{ $removedNode = $root."system.web".compilation.assemblies.RemoveChild( $item ); }

}

$doc.Save($webConfigFilePath);

}

See how conveniently you can access nodes just like object properties. For example, you can change the <compilation> tags “debug” attribute this way:

$root."system.web".compilation.debug = "false"

There’s no need for SelectNode and then SetAttribute call on XmlElement.

Similarly you can iterate all <add key=”…” value=”…” /> nodes in <appSettings> and modify their value like this:

foreach( $item in $root.appSettings.add )

{

if( $item.key -eq "SitePrefix" )

{ $item.Value = $EXTERNAL_SITE_PREFIX; }

if( $item.key -eq "InternalSitePrefix" )

{ $item.Value = $INTERNAL_SITE_PREFIX; }

Power Shell makes Xml file manipulation peace of cake. You can add nodes and remove nodes just like adding objects in arrays. For example, here’s how I am removing all “nunit” assembly reference nodes in <system.web><compilation><assemblies> tag:

foreach( $item in $root."system.web".compilation.assemblies.add )

{

if( $item.assembly.Contains("nunit") )

{ $removedNode = $root."system.web".compilation.assemblies.RemoveChild( $item ); }

}

If you use NUnit, you will see they get added to the web project for no reason and if you don’t deploy nunit DLLs, you cannot run your web site due to the assembly references. The above code cleans up nunit reference from web.config.

Now’s another interesting point. If some JS file changes, we increment its version no by renaming the file with a versio number. For example, when our framework.js file gets changed, we rename the file to Framework1.js, Framework2.js and so on. When we do that, we need to update the <script src=”Framework.js”> in Default.aspx to <script src=”Framework1.js />. Here’s how we do it:

updateReferenceWithNewVersion $FRAMEWORK1_FILENAME $DEFAULT_ASPX $VERSION $srcPath $deployPath

function updateReferenceWithNewVersion( $fileName, $referenceFile, $versionNo, $srcPath, $destPath )

{

$filePath = [System.IO.Path]::Combine( $destPath, $fileName );

if( [System.IO.File]::Exists( $filePath ) )

{

"$fileName exists. Upgrading its version in $referenceFile"

$referencePath = [System.IO.Path]::Combine( $destPath, $referenceFile );

if( -not [System.IO.File]::Exists( $referencePath ) )

{

" Copying $referenceFile because it's not updated"

$referenceSrcPath = [System.IO.Path]::Combine( $srcPath, $referenceFile );

$result = copy $referenceSrcPath $referencePath

}

$newFileName = $fileName.Split('.')[0] + "-" + $versionNo + "." + $fileName.Split('.')[1];

ren $filePath $newFileName

$referenceContent = [System.IO.File]::ReadAllText( $referencePath );

$referenceContent = $referenceContent.Replace( $fileName, $newFileName );

[System.IO.File]::WriteAllText( $referencePath, $referenceContent );

}

else

{

"$fileName not available"

}

Pretty long script. Basically it checks if the JS file that we want to increase version exists or not. So, if Framework.js exists in the deployment folder, then it is changed. So, we know we have to change default.aspx in order to put the correct reference. Now what if Default.aspx does not exist in the deployment folder? We first copy the default.aspx to the deployment folder and then update all the places where “Framework.js” has been referenced.

Now’s even more interesting part, we run JSMIN to compress all .js files that is in the deployment folder. JSMin is a filter which removes comments and unnecessary whitespace from JavaScript files. It typically reduces filesize by half, resulting in faster downloads. It also encourages a more expressive programming style because it eliminates the download cost of clean, literate self-documentation.

So, we launch jsmin.exe and pass input and output JS file name as argument. Here’s how we launch exe files from Power Shell script:

$jsFiles = get-childitem $deployPath -include *.js -Recurse | where { $_ -is [System.IO.FileInfo] }

$p = new-object System.Diagnostics.Process;

$si = new-object System.Diagnostics.ProcessStartInfo;

$si.FileName = $jsminPath;

$si.CreateNoWindow = [Boolean]"true";

$p.StartInfo = $si;

foreach( $file in $jsFiles )

{

[string]$filePath = $file.ToString();

[string]$newFilePath = $filePath + ".new";

$si.Arguments = $filePath + " " + $newFilePath;

$result = $p.Start();

$p.WaitForExit();

if( $result )

{

" Minified: $filePath";

copy $newFilePath $filePath;

del $newFilePath;

}

else

{

del $newFilePath;

}

$p.Dispose();

This saves javascript file size. Now we need to optimize HTML files also and squeeze them as much as possible in order to save bandwidth and make site load faster. There’s a free utility Absolute HTML Compressor which has a command line version. We use the ahc.exe file to compress HTML files in deployment directory the same way we use JSMIN to cleanup javascript files.

"Compressing html files..."

$p = new-object System.Diagnostics.Process;

$si = new-object System.Diagnostics.ProcessStartInfo;

$si.FileName = $htmlCompressorPath;

$si.CreateNoWindow = [Boolean]"false";

$compressorParam = "`"" + $deployPath + "`" " + $HTML_COMPRESSOR_PARAMS;

"$compressorParam"

$si.Arguments = $compressorParam;

$p.StartInfo = $si;

$result = $p.Start();

$p.WaitForExit();

if( $result )

{

" Compressed";

}

else

{

" Error occured";

}

$p.Dispose();

Now’s the season finally, we zip up the entire deployment folder and make one zip file to upload.

"Zipping up...."

# zip the entire deployment folder and make one zip file

$zip.CreateZip( $deployZipFilePath, $deployPath, [Boolean]"true", [string]::Empty );

$zip.ExtractZip( $deployZipFilePath, $deployPath+"_Extracted", [string]::Empty );

"Zipping done..."

So, after all these, we have a nice patch that we can apply on test, beta, production servers. The patch ensures:

Correct configuration, server specific settings
All JS files are compressed as much as possible
All HTML files are also compressed.

Hope you enjoy it!