Disaster recovery plan objectives

The key objective of a disaster recovery plan is to detail the key activities required to reinstate the critical IT services within the agreed recovery objectives. The most effective start point for any DR plan is the ‘declaration of a disaster’ once an incident has been deemed serious enough that ‘forward fixing’ at the primary location is impractical or is likely to result in an outage expending beyond the maximum tolerable outage.

There are a number of common mistakes which organisations make when creating a disaster recovery plan, these relate to the level of detail they contain and the ‘standalone’ nature of their construction.

What level of detail should the plan contain?

The answer will depend on who you ask, the more people you ask the varied number of replies you will receive. It is advisable to keep the IT DR plan as concise as possible and focus only on the key information required at the time of a disaster.

What information should the disaster recovery plan contain?

As a minimum the plan should contain the following information:

• A statement detailing the scope and capability of the disaster recovery plan, exactly when should this plan be used and what ‘consequences’ are covered. It is advisable to focus on the consequences of an incident rather than the cause. Why focus on consequences rather than the cause?

Is it really important why the data centre was destroyed? As far as the DR plan is concerned the answer is no. The same process and recovery stages will be followed regardless of the cause, fire, flood, terrorist incident, or the proverbial aircraft impact will all result in the partial or total destruction of the data centre.

The only relevant question is what is the impact and can I realistically continue to host services from my primary site or should I invoke and recover/resume the critical services at my secondary site.

• A description of the key roles and responsibilities so that anyone assigned to a particular role in the recovery team understand what is required of them. Should you name individuals in the plan? Ideally individuals who are to be expected to perform a particular role should already be aware that they are likely to be called upon and should have received the relevant training. It is advisable to record the names and contact details of individuals in the relevant section of the overall BCM plan rather than the DR plan. There is no reason why the individual names at the time can’t be entered into the recovery log as the ‘designated recovery manager’ or other predefined role.
• A summary of the critical services, their recovery objectives and recovery priorities, this information may be lifted from the BIA which was performed as part of the overall BCM process. Summarising them in the disaster recovery plan will remove the inevitable discussions at the time of the incident and provide a reference point for the recovery teams.
• Third party contact details, particularly those that may be required to assist in the recovery effort or those that provide recovery services, for example:
  • The secondary (DR) data centre service provider, you will need contact details, address, maps, and of course the invocation process and codes. It is advisable to do this as soon as it becomes clear the incident is likely to become a disaster recovery situation. You can always ‘stand down’ if the incident can be forward fixed (some service providers may levy a charge for this);
  • Your media handling company. Are your disaster recovery tapes removed from your data centre and vaulted off-site? If so you will want to arrange for them to be retrieved and sent to your recovery centre at the earliest opportunity;
  • Mobilisation of the recovery teams. What teams and individuals need to be contacted to recover the services, at this stage of the recovery the incident management team will already know the extent of the incident and should (if not you need to make sure you do at the earliest opportunity) have placed the recovery teams on standby. The plan should record teams and skills required, not individuals. Individual contact details have to be recorded somewhere, it is normal practice, as part of the overall business continuity management program to have contact lists, rather than repeat the detailed contact information the DR plan should reference the relevant sections in the BCM plan.
• Detailed recovery activities and sequence of events, including pre-requisites, dependencies, and responsibilities.

What level of detail should you include in this section of the DR plan?

This is very much down to personal choice, however, as a minimum you should include:

• the recovery process and flow of activities;
• high level activities, for example, load operating systems, install application software, restore data, synchronise database, make configuration changes, post recovery checks, open service to users;
• pre-requisites and dependencies for each activity;
• responsibilities, who will perform each activity.

Should you include the detailed activities for installing an operating system or restoring a database? The detailed recovery activities should be held locally by the team responsible for performing these activities. There are several reasons for this: the “how do I install Windows” instructions will be used for business-as-usual activities, minor incidents, AND disaster recovery. The DR plan only needs to reference these documents, if you find it an absolute necessity to include these in your DR plan then do so as appendices and not in the main body of the document, don’t allow the key purpose of the DR plan to be lost in un-necessary or duplicated detail.

Testing the disaster recovery plan

IT DR testing should be performed on a regular basis, the exact frequency very much depends on your own organisational needs. However, it is usual for ‘full deployment’ tests to be performed, as a minimum, on an annual basis. There are of course other ‘trigger points’, for example, a change in your infrastructure that affects your disaster recovery strategy, i.e. moving from an active/contingency recovery model to an active/passive recovery model.

What do I test? This question is probably the most common question asked, and the answer is simple, you test the plans, the process, the people, and the infrastructure, in fact every component required to recover and resume your critical IT services.

What are the key objectives of a DR test? There are several key objectives of a test, the main ones are:

• exercise the recovery processes and procedures
• familiarise staff with the recovery process and documentation;
• verify the effectiveness of the recovery documentation;
• verify the effectiveness of the recovery site;
• establish if the recovery objectives are achievable;
• identify improvements require to the DR strategy, infrastructure, and recovery processes;

What is the scope of a DR test? The scope will very much depend on the maturity of your DR strategy and capability, it is important to scope the test to stretch the objectives and success criteria of the previous test, for example, if this is your first test, you may not want to have the entire user community scheduled to come in and perform lots of testing, you may wish to limit the scope to just IT staff and maybe a couple of ‘friendly users’ to test functionality. Depending on the complexity of your environment it may take several tests to build confidence and perform a ‘full deployment’ test.

Common DR testing mistakes are:

• operating within your comfort zone, for example, recovering the servers you know you can recovery whilst avoiding the more difficult components;
• not testing the recovery of a service but focusing on the hardware, systems, and applications. Remember, a particular service may require several servers to be recovered, it may also require data held on local drives and network attached devices, and network connectivity from the data centre to the user.
• trying to achieve too much too soon and overstating your DR capability and readiness;
• not planning appropriately, testing and live invocation are very different. In a live invocation you do not have a live environment to protect. Consider the impact that testing may have on your live services. Engage with the appropriate people at an early stage, a “full deployment” test may take several weeks to plan.

Carl Bradbury, MBCI, is senior consultant, Siemens Enterprise Communications Limited.

Disaster recovery checklist

When most people think of disaster recovery, their initial thoughts are about recovering from a complete disaster like Hurricane Katrina. In reality, this type of disaster is rare and often hits specific areas prone to earthquakes, hurricanes, tornadoes and other types of natural phenomena. Although it is good to be prepared for a complete disaster, the likelihood of needing this type of recovery is rare.

More often, you’ll have to handle accidental data deletions, hardware failures and other application failures. While you may not classify these occurrences as catastrophes, they could easily cause serious system failures; preparing for them is as important, if not much more important, than preparing for a complete disaster. The exception, of course, is if your servers happen to be in area where natural disasters are likely to occur.

The following checklist will help you prepare for disaster recovery.

Document

One key component to any process that needs to be done in an emergency is to have proper documentation as to what needs to occur and when. In a crisis, people usually don’t think methodically and things are often done on the fly. Having a script or checklist of what needs to occur will help you stay calm in the event of a system failure. In addition to having a script to follow in a recovery event, a checklist also gives team members the information they might need for the recovery.

One pitfall is that documentation becomes outdated very quickly. This is where you should try to keep things simple, but still include enough information to perform the recovery. I have seen situations where the documentation is very thorough, but not tested. Keep in mind that documentation is just one component of your recovery process.

Practice

As I mentioned above, documentation is the starting point for your recovery. You also need to spend a fair amount of time using your documentation/checklist to practice a recovery. In some situations it will be next to impossible to carry out a full system failover, but if you never test it, you’ll never know if it will work. Things can be done on a smaller scale to simulate the procedures you will follow. But, keep in mind, if you are not testing the real thing you will probably have holes in the process.

Another benefit of testing is that you will be calmer when must do a recovery. If you have already taken the steps, you will know what to expect and how long things take.

Script

I strongly believe in scripting out as much as possible. It is much easier to apply a script to automate the recovery than it is to go through each step along the way during the process. As you are preparing your recovery checklist, identify the things that can be automated via scripts. Take the time to write them down and document their use. This will save a lot more time and confusion later when you actually need to use them.

In addition, SQL Server gives you the ability to script out just about every object that exists within the DBMS, so you should use these tools and periodically script out your objects. These things include logins, stored procedures, table definitions, triggers, SSIS packages and jobs. You may never need to use all the scripts, but having them may be a lifesaver.

Sign off

After recovery is complete you need some way to close the loop. There should be some type of sign-off process that signals recovery was successful. This may include several people, depending on the type of recovery. This again should be part of your checklist. You want to make sure the recovery was successful, the data is intact and there is no chance of further data loss prior to getting the user base back on the system.

Escalate

Have a list of people to be contacted in case of a failure. This should include people who need to do the recovery as well as those who must sign off after recovery is complete.You should have both primary and secondary contacts for each role on the list and at least two ways to contact each person. If you need an answer to move forward on your recovery plan, make sure you have the proper people lined up to make the decisions quickly.

Back up your database

What would a SQL Server disaster recovery plan be without database backups? Plenty of other tips talk about selecting a recovery model and backup plan. Backups are a key component and more can be found in the following resources:

• Checklist: Backing up SQL Server
• Step-by-Step Guide: How to properly back up a SQL Server
• Worst practices for SQL Server backup and recovery

Failover hardware

Having the necessary hardware on hand for a failover is also very important. You may have a one-to-one failover ratio for servers, but that is not very practical for many companies, let alone for every single server. You need to identify which hardware will be used for a failover and make sure it will be available when needed. Also, servers might fall into different classifications — small, medium and large — so you may need to have several failover boxes on hand.

Media

It is also important to have the necessary media on hand in case you need to do a complete system rebuild. Software versions are always changing, so you can’t assume that you will always have the exact version in your media cabinet. Keeping servers standardised will make this much simpler, but having the correct versions should be a requirement. Create a media kit that includes all of the necessary software versions and patches. You can make multiple copies; one for on-site use and another to be stored with off-site tape storage.

Change control

In addition to documenting and testing your recovery process, having a good change-control process is very important. Just think about all of the effort you put into documenting your failover process and testing it to make sure everything works without a hitch, only to have someone make a modification that invalidates all that you did. The way to combat this is by not letting it happen. Once you get a server into your recovery umbrella, make sure it is under a change-control process, so when changes are introduced, your recovery process will be modified to reflect whatever changes are necessary.

Recovery priority list

Another key piece of documentation is a priority list that designates which servers need to be recovered and the order of recovery. In most cases, it is probably not likely that all of your servers will go down, but if it did happen, it would be next to impossible to recover them all at once. Having a priority list will help you concentrate on critical servers first and leave the others for later. When preparing this list, keep in mind which servers depend on other servers. It’s no use recovering a server if it can’t be used until another is recovered first.

It’s impossible to be prepared for every possible failure that could occur, but being more prepared versus not in the least prepared is definitely a better place to be. Take time to think about what you would do if there was a failure and begin by putting together a checklist of steps that you would follow.

From my SQL Server DBA handbook.