Disaster Recovery - Derecho on June 29, 2012 was wakeup call
On June 29, 2012, one of the top storms of a lifetime roared through our area and even a week later there are numerous power outages. Our power company noted that the 90 high power transmission lines are not something that is easily fixed with a bucket truck. The article below notes that this "storm of the century" is a wake up call to brush the dust off our DR and Contingency planning manuals and always be prepared for the worse.
Storms of June 29th 2012 in Mid Atlantic region of the USA
QUOTE: On June 29th 2012 a severe windstorm reffered to as a derecho tore through the Midwest and MidAtlantic regions of the US. Over 1,750,000 homes and businesses were left without electricity. Datacenters supporting Amazon's AWS, Netflix and other large organizations were taken offline, and there were several deaths reported. I work for a company with a NOC and primary data-center in the path of the storm. A number of events took place. With day time temperatures near 108F and the windstorm coming through the battery on the backup generator powering the data-center cracked and was not able to start the generator.
So on to old lessons learned – geographic redundancy is desirable, document everything in simple accessible procedures, some physical servers may be desirable, such as DHCP, and AD. Keys services such as RADIUS must be available from multiple locations. Securely documenting addresses and passwords in an offline reachable manner is essential as well as documenting system startup procedures. Some new to me lessons learned are a little more esoteric. Complacency is a huge risk to an organization. Our company is undergoing a reorganization that is creating a lot of complacent and lackadaisical attitudes. It is hard to fight that.