Caches, as far as the eye can see…

One of the things I always recommend when giving IBM CMOD Best Practices seminars is that an Application Group should always have its own Storage Set.  The lesson that bore that nugget of truth was a painful one for a mid-sized American bank.

I got a phonecall from a contact at IBM, asking if I was available for an emergency assignment.  Given that I’m not a doctor or firefighter, the word ’emergency’ isn’t thrown around a lot.  I was told to book a flight to New York, and all I was given was a name and a cell phone number.  I arrived the next day and got settled into my cube.  After meeting the team, the reason I’d been called in was finally revealed — I was replacing their OnDemand Administrator who had fallen ill, and was put on six weeks of bed rest as a result.  What made it an emergency, was that the bank was embroiled in a lawsuit, and access to the documents in CMOD was absolutely critical.  I still failed to see the problem — Content Manager OnDemand servers often run for weeks or months with a bare minimum of care & feeding.

The snag was that as part of this lawsuit, the bank was under a court order to retain ALL documents on a variety of servers, and Content Manager OnDemand was just one of them — but OnDemand processed the highest daily volume of the three systems that were the subject of the court order.   That, and the Application Groups that contained huge volumes of transient data that was set to only be stored in the cache, with no (legal) way to delete it, or ability to offload it onto inexpensive tape storage through their existing Tivoli Storage Manager server.  When our managers pushed to have the data redirected onto tape, the lawyers refused to allow us to change anything, for fear of violating the court order.

By the time I left almost 6 months later, there were 25 4TB cache filesystems.  The monthly cost of storage was probably more than our small team earned in a year — and the overwhelming majority of that data completely useless and uninteresting to anyone due to it’s transient nature, but still needed to be retained until the lawsuit was completed.

If only the Application Groups had been properly defined with individual Storage Sets, the Storage Set could have been updated to move the data to tape, and there would be little to no problem, and an order of magnitude less cost.

The next year, the court case was resolved, and then the painful task of reclaiming & consolidating the CMOD cache filesystems was undertaken.  But that’s another story.