I cannot handle it in silence anymore so let me give you a lesson in backup, disaster recovery and what my industry calls business contingency. Check out my Diary for the complete lesson...
Even more troubling, the White House has now admitted that until October 2003, the White House recycled its back-up tapes, which contained the only copies of emails deleted prior to that date. What the White House has not explained is why it changed its policy of preserving all back-up tapes.
In all my years I have NEVER come across a customer who makes the decision to "recycle" tapes to save money or tapes.
HOW TO GET TO THE BOTTOM OF IT QUICKLY:
The government hired either the backup software maker's consultants or a trusted consulting company to architect and implement the solution.
Subpoena the consulting company to detail the backup solution they were contracted to implement and the documents the government signed off so the company could get paid. If the backup environment does not look like the one the consulting company set up you know the white house has been up to no good.
These consultants are your witnesses and smoking guns who are not under executive privilege.
I cannot handle it anymore without stating this "backup" excuse is worse than the "my dog ate my homework" stunt I tried to pull in grade nine. So let me give you a lesson in backup, disaster recovery and what my industry calls business contingency.
The White House has also admitted that the only safeguard it has to its patently inadequate method for preserving email (dumping them in files that are put on EOP servers) is back-up tape media. These back-up copies, however, are only a "snapshot" of what was on the server at the time of the back-up. In other words they are not comprehensive, as the White House concedes.
Before I start our lesson, I will qualify myself. I have over five years as a consultant for The Largest Backup Software and Security company in the world. During these years I traveled the United States consulting for large companies and government offices. Our consulting focused on reviews of our customer's current backup / recovery strategies and detailed solutions to meet legislative requirements. Some of these solutions spanned over a three year implementation period. Now let's get down to the lesson...
There are many ways to back up data and many layers to ensure you catch it all.
Tape Backups
When backing up data to tape you MUST set an expiry date for the data in a pre-defined policy. This policy is a global policy and will apply to any data that fits the policy criteria. In general, most regulation's dictate 1 year, 7 years or forever. You cannot no just "recycle" a tape arbitrarily. Tapes will not let you tape over them nor can you just go and change the policy. This is how you would have to do it:
- Pull a list of all the tapes that contain the data you need to restore
- Order the tapes from your offsite storage company, wait for them to arrive and hope one has not been missed.
- Manually mount all the tapes BACK into the tape libraries - not a small feet.
- Change your policy in the backup software to expire the tapes. This will now "freeze" your tapes. Freezing means your tape expiry information does not match the new policy information.
- You must now manually un-freeze all the tapes by entering command line scripts.
- Once un-frozen you can erase the tapes by writing over them or magnetically erasing them.
The above watered down walk through should give you an idea of how non-trivial the decision to recycle tapes is.
In all my years I have NEVER come across a customer who makes the decision to "recycle" tapes to save money or tapes.
When performing evaluations, we were always the ones highlighting how our customers could more effectively use tape space and how policies should be managed to ensure companies were not saving everything forever.
Don't Forget the Other Layers for Protection
To save you the heartache I will not go into detail but I can promise you that certain levels of government will use other layers of backup to ensure data is backed up and easily accessible. These would be:
- Email Specific Backup and Logging. For years companies have been using software to not only save every email down to the last little letter icon in your electronic inbox. They can sort and journal words for easy searching should there ever be a court case.
- Full monthly/annual backups of top of incremental backups. This ensures that you only loose 10 Thousand emails vs. 5 Million.
- Clustering or Mirroring. I promise you almost all important servers are configured in a way that if one goes down a second server acting as a mirror of the first will pop up within minutes and be running without a blink of an eye. This is how VISA makes sure you can keep shopping through the holiday season.
- Bare Metal Restore. Is a process in which a full image of a server is saved. You can save several images over time and at any time restore the ENTIRE computer back to its original state.
I have been in meetings where government agencies purposely do not engage their backup capabilities for one or both of these reasons:
- They do not want to expose themselves legally because they know there may be sensitive data accidentally communicated by employees.
- They actively know not all dealings are above board and therefore cannot risk the exposure and consequences.
HOW TO GET TO THE BOTTOM OF IT QUICKLY:
There is one surefire way to get to the bottom of this fiasco quickly.
The government hired either the backup software maker's consultants or a trusted consulting company to architect and implement the solution.
Subpoena the consulting company and have them detail out the backup solution they implemented, when it was implemented and the documents the customer signed off on so the company could get paid for the implementation.
If the backup environment does not look like the one the consulting company set up you know the white house has been up to no good.
These consultants are your witnesses and smoking guns who are not under executive privilege.
Happy Hunting for Backups... they are there you just have to ask the right people!