The good, the bad, and the ugly - custom action types
I'd like to add my thoughts to the recent discussion about how bad custom actions are, and which types are better or worse than others.
First of all, I believe there's a general consent that custom actions shouldn't be used for tasks that Windows Installer supports natively. Or, as Rob Mensching puts it: "Custom actions are (generally) an admission of failure" - either by the setup developer, or in the application, or by Windows Installer itself, because it's lacking some commonly required functionality.
Rob is the project leader of the open source WiX Toolset for creating Windows Installer setups. Part of the WiX project is a set of custom actions for common tasks such as interacting with IIS and SQL Server. Some people wish that such functionality would be added to Windows Installer itself, but I believe that won't happen. Alternatively the SQL, IIS, etc. teams could provide custom actions for their respective platforms, like the DIFx people did, but in most cases this hasn't happened either.
So we'll have to use custom actions, to retrieve information and set properties in the install, and to modify the target system. In general, the latter is more dangerous than the first, and requires additional effort such as creating a corresponding roll-back action.
There are even harmless types of so-called "custom actions" which are actually more like standard action: type 51 (set a property), type 35 (set a directory) and type 19 (exit with an error message).
But the discussion really is about whether EXE is worse than DLL and which programming language should be used to create custom actions. Windows Installer natively supports the following types of custom actions:
- VBScript, JScript - These don't work on some machines, if scripting is disabled. They can also make your setup look very evil if a virus scanner flags it as potential malware (think of Office macro viruses). If the script becomes complex, it's also hard to debug.
- EXE - Works, but doesn't have a way to communicate back to the install. So you can't return information in properties, you can't increment the progress bar and the only return values recognized by Windows Installer are success or failure. Robert Flaming of the Windows Installer team at Microsoft lists some caveats for EXE custom actions. Note that some of the problems he lists apply to "bad EXEs", so there is a chance to make it better. Also there are valid reasons to use EXE custom actions, e.g. you may already have a working and tested configuration tool which you want to re-use instead of rewriting it as DLL.
- DLL - Typically a very robust and versatile way to create a custom action. However you should avoid runtime dependencies.
There are third party extensions that enable you to use other types of custom actions. Christopher Painter argues that "InstallScript Custom Actions are GOOD". To the Windows Installer engine they appear as DLL (the InstallScript engine) and InstallScript is a language particularly for setup related tasks. However I believe that the tasks of a custom action often aren't typical installation tasks (copying files, accessing the registry) but require calling some APIs to interact with other applications. On the other hand, it's tempting to use InstallScript to perform tasks that actually should be left to Windows Installer's standard actions. So InstallScript custom actions, while they may be useful, should be used with care. And they add the overhead, in package size and runtime, of the InstallScript engine. In general, I think it's useful to write custom actions in the programming language you know best.
As mentioned in the beginning of this article, a custom action should never be used for tasks that Windows Installer supports natively. Unfortunately this still happens. Sometimes because the setup developer is lacking knowledge about the functionality included in the MSI engine, but probably more often because the authoring tool he's using doesn't expose the full Windows Installer functionality. This leads the developer to re-invent the hidden functionality using custom actions. Namely Setup & Deployment projects in Visual Studio suffer from this problem.
The ultimate form of such a custom action is a "wrapper msi": a msi setup that mainly consist of one custom action that runs an existing setup program. Instead of re-building the setup as true Windows Installer setup, the legacy install package is just wrapped in a msi. While this enables you to deploy the wrapped setup using corporate deployment tools, you don't get the benefits of a real Windows Installer setup, such as a transacted install with rollback, customizability with transforms, robustness by auto-repair, etc. A recent example is the MSI package created by the Internet Explorer Administration Kit.
Here are some pointers to built-in Windows Installer functionality which can help you to avoid unnecessary custom actions:
[Edit 2007-11-25:] Additional information and mitigation tips for EXE custom actions can be found in Heath Stewart's blog.