SQL Server 2000 post SP4 hotfixes don't install on all nodes of a cluster
I recently installed a new Windows 2003, SQL Server 2000 cluster.
Then added SQL Server 2000 SP4 and applied hotfix 2040 (KB 899761 FIX: Not all memory is available when AWE is enabled on a computer that is running a 32-bit version of SQL Server 2000 SP4)
By chance I looked at the sqlservr.exe file and noticed it was still 2039 on all non-active nodes for the instance, whilst the node I installed the hotfix from was at 2040.
I looked at the SQL2000-KB8997618.00.2040-x86-ENU.log log file that 2040 creates.
3.781: SQL DLL: Successfully transferred payload to remote product instance target \\NodeB
3.781: SQL DLL: Successfully transferred payload to remote product instance target \\NodeB
3.797: SQL DLL: Successfully created new scheduled task for product instance target \\NodeB
3.828: SQL DLL: Error, failed to save new scheduled task for product instance target \\NodeB
3.828: SQL DLL: Error, failed to create scheduled task for product instance target \\NodeB
4.000: SQL DLL: Successfully removed remote folder for product instance target \\NodeB
4.000: SQL DLL: Error, remote process failed for product instance target NodeB
5.031: SQL DLL: All remote clustered nodes have completed patch installation
5.031: SQL DLL: Checking that at least one remote clustered node was successful
5.047: SQL DLL: Remote clustered node NodeB was successfully patched
5.047: SQL DLL: One or more remote clustered nodes were successfully patched
What?
On step 3.828 it fails, but then still claims on 5.047 to have checked that the hotfix was there.
Not the case, it failed and 2040 is not on the other nodes.
As you may nor may not know, Microsoft has changed the hotfix installer for SQL Server. What they did not tell us is that it now relies on Windows Task Scheduler to do it's work on the other nodes of the cluster.
What it boils down to is that if Windows Task Scheduler is disabled or not running on the remote nodes, the hotfix install on those nodes fails, but fails very silently.
Most corporates disable Windows Task Scheduler as it is a potential security risk.
So, before you apply any SQL Server 2000 post SP 4 hotfixes on a cluster, enable and start Windows Task Scheduler on all the nodes of the cluster (stop and disable them once you are finished before your security team come and get you).