Contractual Arrangements for Disaster Recovery Services

In theory a commercial hot or warm standby site is available 24 / 365. It has staff skilled in assisting recovery.  Its equipment is constantly kept up to date, while older equipment remains supported.  It is always available for use and offers testing periods once or twice a year.

The practice may be different.  These days, organizations have a wide range of equipment from different vendors and different models from the same vendor. Not every commercial standby site is able to support the entire range of equipment you may have. Instead, vendors form alliances with others – but this may mean that your recovery effort is split between more than one standby site. When you invoke the standby site, it may not be equipped as you need it – for instance, the desks may have mainframe or midrange terminals on them, instead of the PC / Local Area Network (LAN) / server environment you need.  There may be a lead time to de-install the terminals and install the PC / LAN environment.  Perhaps the standby site is already occupied by another client who may be testing.  This client has to be allowed to close down and move out.  Again, there may be a lead time for this.  Typically four hours notice may be required before the client can actually occupy the standby facility (although they may gain access to meeting rooms earlier.

The standby site may not have identical IT equipment: instead of the use of an identical piece of equipment, it will offer a partition on a compatible large computer or server.  Operating systems and security packages may not be the same version as the client usually uses.  These things may cause setbacks when attempting recovery of IT systems and applications.

Call Center standby sites may not always be compatible in equipment nor able to replicate the whole of an integrated customer relationship management system. Indeed, most of the Call Center recovery plans we have seen simply would not work.

Commercial standby sites vary in the standard of facilities they provide.  Many have workspaces with small desks and limited storage space.  Some have meeting rooms, restaurants, shower and rest facilities while others may have very basic facilities.  The presentation and location of some is superb: others are basic buildings in insalubrious surroundings that may cause concern for staff arriving and leaving at night. Note that some commercial standby sites only have a limited amount of equipment actually on site and will contract for quick resupply of additional equipment to their site if a client invokes use of the standby site in a disaster. Some are not convenient for public transport.  Others may not have sufficient parking space.

Telecommunications issues may also arise: it is important to ensure that relevant links are in place and that communications capability is compatible.  The adequacy of voice and data capacity needs to be checked.  Telephony needs to be switched from the disaster site to the standby site: can this be done?  Can your staff operate the switchboard at the standby site? (Incidentally, having telephone and fax numbers mixed up sequentially does not help in recovery – it is much easier if fax and telephone numbers are separable in distinct blocks.)

Most commercial standby sites offering IT and work area recovery facilities do not guarantee a service: the contract merely provides access to the equipment.  Although most reputable vendors will negotiate a Service Level Agreement that specifies the quality of the service, it is rarely offered.  It is important to ensure that your service will not suffer from unacceptable downtime or response.

Not all commercial standby sites keep up to date with equipment.  They exist to make a profit.  They do not, therefore, usually invest in the latest equipment and then try to sell subscriptions for it.  They are probably more likely to wait for client demand to justify acquiring the new equipment so that its cost is quickly recovered.  This means that those organizations that buy the “latest and greatest” may find a time gap between acquiring this equipment and having a standby recovery site available.  Then they are likely to pay a high price for their subscriptions, since the vendor has little competition and will seek to recover as much of the investment as possible from the first customer.

The vendor charging structure needs to be carefully considered.  It is not unusual for vendors to seek to recover fixed costs plus basic profit margin for the whole recovery site from the first five subscribers.  These first five may be tempted by discounts for three or five year contracts.  However, as soon as the fixed costs plus basic profit margin are achieved, the vendor can afford to discount significantly, since his additional costs are marginal and new subscriptions go almost entirely to the profit line. Knowing this, in one case, we achieved a discount of over 70% from one vendor for the same service. However, the initial subscribers may be locked in to paying top dollar for the next three or five years.  When one considers that each facility may have typically up to 35 subscribers, it can be a lucrative business for the vendor.  That, coupled with future earnings being covered by medium and long term contracts, is why one service vendor came to the market on 68 times earnings.

Some vendors offer a drop-ship service as an alternative to occupying the standby site.  That is, in the event of equipment failure, for instance, they will drop off a replacement rather than insist the client occupy the standby site, with all the inconvenience that may involve. Some vendors include this as part of the standard subscription, while others treat it as a premium service and charge extra for it.

The vendor may have skilled staff available – but this is rarely guaranteed and they come at a cost. In terms of cost, there may be additional fees to pay for testing, on invocation of a disaster, and for occupation in a disaster. Some of these costs may be covered by your insurance as extra cost of working, but this should be checked before the disaster.

Mobile / portable alternate facilities sound attractive, but it is essential that a site survey is undertaken to ensure they can be parked on the required site.  We know of at least one disaster invocation where a mobile unit arrived outside the disaster site, only to be moved on by the police.

The better vendors have algorithms and checks to ensure a minimal possibility of a client invoking and finding the standby site occupied (i.e. they calculate an actuarial basis for standby).  They will welcome your attendance at user group meetings and will provide references (subject to confidentiality agreements with their clients). Larger vendors will have a number of other sites to which clients can overflow if necessary.  Among the questions we need to ask are:

• Is the vendor financially sound?
  
• If I invoke, will the recovery site be occupied?
  

  How many subscribers are there for the whole recovery site?
  How many subscribers are there for the equipment we need?
  How many subscribers share our normal office building?
  How many subscribers are there within half a mile of our normal office building?
  Within one mile?
  Who has priority?

• If the recovery site is occupied when I invoke, where do I go to recover?
  
• How does the vendor define “disaster” – on what conditions may I invoke?
  
• How quickly can I occupy the recovery site on invocation?
  
• How much will the annual subscription cost?
  
• How much are invocation fees?
  
• How much will it cost to test?
  
• How much testing time is allowed?
  
• Can the vendor personally cover the full range of our equipment and telecommunication needs now?
  
• If not, how will these needs be met?
  
• Does the vendor have standby generators and uninterruptable power supply adequate to maintain the whole installation?
  
• Does the vendor have alternate telecommunication suppliers with separate routing?
  
• Will the vendor keep in step as I buy new equipment?
  
• Will the vendor support aging equipment for as long as I need it?
  
• Can I store my equipment at the vendor site?
  
• Will the vendor drop-ship small equipment at my site to save me having to relocate to the recovery site in the event of hardware failure or loss of a single component?
  
• If so, will they charge extra or is this included in the annual subscription?
  
• Is the location of the standby site safe for my staff?
  
• Is the recovery site convenient for public transport?
  
• Does it have rest, shower and catering facilities for my staff?
  
• Does it have adequate parking space?
  
• Is the site secure, and will my data remain confidential?
  
• What are the qualification and skills of the vendor’s support staff?
  
• Are they certified as members of the DRII or BCI?
  
• Will the vendor’s support staff help me recover?  If so, how many?
  
• Will the vendor’s support staff help me test?
  
• Does the recovery site agreement contain a Service Level Agreement specifying availability, reliability and performance?
  

This section may seem negative – or even hostile – to vendors.  It is not intended to be so. The majority of standby site vendors provide sound service at reasonable cost and are genuinely dedicated to assisting their clients under the most difficult circumstances.  They have an enviable record of successful recoveries.  But, as in any industry, there are a few unscrupulous suppliers.  It is the responsibility of the BC manager to ensure that he or she entrusts the survival of their company only to those vendors who apply the highest standards and that this is backed up by a stringent contract, clearly defining service specifications and technical requirements and Service Level Agreements.

© Andrew Hiles