In the wake of Hurricane Sandy – we are reminded to update and rethink our disaster planning
Living in Dallas, the news stories about the devastation of Sandy is a constant somber reminder to count our blessings. It’s devastating to think about loosing everything you have worked so hard for – so quickly.
Not only are we reliant on our physical personal things, we are also reminded about the impact disasters have on our business. We are helplessly dependent on our data. Companies with Datacenters in the northeast are seeing first hand how crippling it can be to business when computing power is all but stopped.
“93% of companies that lost their data center for 10 days or more due to a disaster filed for bankruptcy within one year of the disaster. 50% of businesses that found themselves without data management for this same time period filed for bankruptcy immediately. (National Archives & Records Administration in Washington)”
Found on:
http://www.bostoncomputing.net/consultation/databackup/statistics/
Is EPM and BI Mission critical? You bet it is.
Busting the common complacency myth: I hear it all the time
“If we had a disaster, the company would be more concerned with more mission critical systems such as revenue and billing systems…they would not be concerned with systems like Hyperion”
Maybe. I agree those are more mission critical systems… but lets face it… … those systems have their own procedures for disaster recovery. We still must protect our house and ensure our internal finance customers are protected. Yes, perhaps company billing, revenue, and payroll systems are the priority but there is nothing that should stop us for implementing our own DR procedures and business continuance plans. Blaming perceived “higher priority” systems is simply an excuse from our duty as IT to provide service to our users.
Lets say the datacenter flooded. All servers lost. And not just you – all other companies and businesses in the area are flooded. What do you think the lead time would be to clean up and get more servers an rebuild the environment? 1 year? 1 ½ years? All businesses in the area will be scrambling for servers and other hardware. It’s time to start thinking about scenarios.
We must protect this house
Protecting our EPM and BI instances from hardware failure is an inherent job and responsibility as corporate Finance IT administrators.
What grade do YOU get?
1.) Grade F: Not doing anything to prepare for the worse.
2.) Grade: D: Regular Backups.
Of course many organizations take backups of their environments. But are they being taken correctly? Some products in the EPM suite acquire exclusive locks on files that make backing them up while the system up almost garbage. Backup systems like NetBackup simply come by and backup servers whenever it darn well feels like it and we don’t know the status of the system when that happens. This is a volatile situation and may not be recoverable. Also, this system embeds hostnames everywhere. Don’t think that you can save your backups off-site somewhere and restore them on another server with another hostname later. While there are things we can do to force that to work, its not enough for a mature DR solution.
3.) Grade: B: DR site with artifact level exports.
Is your DR site hot or cold? A hot DR sites is a separate environment just like DEV, TEST, PROD, etc in a DR datacenter ready to take over. But how do you keep them in sync? Some solutions automate a nightly migration between the environments using things like Life Cycle Management. Some organizations ensure that DR is part of their path to production when promoting and migrating objects from DEV-TEST-PROD. This solution offers an immediate environment in the event of a disaster of the primary datacenter, however there will always be the possibility of old data.
4.) Grade: A: Load balancing between datacenters
Global world class organizations do active load balancing between datacenters which means that the primary and DR datacenters are actively participating in servicing end users in normal operation. Companies in this situation are tolerant to ANY datacenter going down, and still instantly servicing users with no data loss or downtime.
These are just a few examples of DR strategies. Especially in the wake of this storm, it important for companies to think of the unthinkable and plan for the worse. It’s a relatively small effort to put together a DR strategy, documentation, testing and implementation as an ultimate insurance policy and worth the exercise. I find it so funny that organizations will spend so much time and money into a comprehensive Highly available EPM and BI solutions and not spend the small amount of time to do a deep assessment and implementation of a business continuance strategy to protect it.
Eric specializes in advanced high availability and disaster recovery solutions tailored for Hyperion EPM and BI.
The I.T. Side 