James,

Most often the biggest challenge is in figuring out "what to keep" and "how long to keep it". What appears on the surface to be two very simple decisions? If the company is regulated, (i.e. Financial Services) there will be certain records and retention periods that are cut and dry, or so you would think – because of the “smoking gun” effect, which says, maybe I rather keep everything, so at least “I know” vs. purging and down the road “not knowing” even cut and dry requirements become vague.

So understanding and creating policies is most often the largest and most daunting challenge.

The second is in understanding the impact of the technology.

How often does any corporation, large or small, collect millions of "objects" that have to be technically “harvested” from source systems such as email, which are designed and hardened over the year to be super secure - and what does the archive technology do? Pluck it right out, and stick it someplace else. Anyway, collected with the proper security attributes (ACLs and other), placed (processed) into its proper category, in order to be managed, and then stored (hopefully efficiently) for potentially 10,20,50 years or maybe forever.

The impact spans the “entire corporation”, from the board room to the helpdesk, and most IT organizations are not ready to deal with long-term retention of "critical" data objects within their data-centers. It's a learning curve that requires different approaches to managing not just the "technology" risk associated with business continuity and DR for example, but the "business" risk that is contained within the data.

And a tremendous amount of information is at risk, look at just email for 15,000 users, that averages 500mil messages daily, and about 140mil yearly- figure close to 35TB. And that's the easy part, provisioning storage, the hard part is "processing" to get it "in" with accuracy, and then the even larger requirement, which is to "search" and "extract" with accuracy.

The legal requirements create a whole host of technology requirements, to preserve "fidelity", "original meta-data", manage "provenance", etc. The list goes on and on.

In a nutshell, what I see most often is an attempt to "boil the ocean" on the first pass, instead of gradually easing an organization into an "archival" and "compliance" culture. It's a “culture”, which actually starts from the "point of hire" with continual training that needs to be formed and made part of the DNA of an organization – this will eventually negate most of the legal and compliance issues (not the technology) and protect corporate America.

Good Luck,
Peter
April 2008