....but that isn't the whole story.
There must have been a problem with this XP Machine for a while, but the Wave 15 upgrade made it stand out.
The problems seemed to stem from a missing certificate which meant that the outlook.com chain was broken. I was getting "The Issuer of this certificate could not be found" on the outlook.com cert both in Outlook and Webmail / IE. In the chain, I found MSIT Machine Auth CA 2 and Microsoft Internet Authority mentioned.
The fix itself was simple in the end, but it did take a lot of searching. Which is why I am putting this here for others.
Download the latest Root Certificate and install in the Trusted Root Certificate Store on your machine.
Link to cert