Tuesday, November 27, 2012

Harpooning the whale

One data set, two locations - each with an EMC VNXe array. Big giant file sets. How do engineers in the US and Mexico share 3D models quickly and seamlessly? We receive engineering drawings and models from our customers. We comb through the files extracting bill-of-materials information, critical dimensions and other vital attributes. A single product may have dozens of associated files totaling several hundred megabytes. Open the main assembly and a significant number of associated files may also be opened. So how can engineers connected only by VPN work on the same filesets quickly and seamlessly, just as if they were sitting in the same room? That's the first problem.

On to the next problem. I hate to change tapes. Actually since there are so few people in our US world domination headquarters, the question of who will change tapes each day is important. I may not be in the office on any given day. We have a sixteen-tape rotation - eight Friday tapes, eight weekday tapes. We can go back to any Friday in the last eight weeks, or any weekday in the last two weeks. That's the extent of our retention needs. I have a nice chart showing which tape goes in the drive on which day. Pop out last night's tape, put it in your briefcase to take home with you. Pop in today's tape. Bring back from home the tape from two nights ago. There should always be one tape in the drive, one tape in your briefcase, one tape at home and fifteen tapes next to the backup log book.

In my fevered imagination I'm plotting a single grand solution to both problems. Real-time, two-way site-to-site data synchronization along with offsite backup set replication allowing for at least eight weeks of retention and file versioning. This is my white whale, the one problem I am compelled to solve before I leave my post for good.

DFS, Robocopy, Unison, Cobian, Synchronicity. SureSync, GoodSync, Syncrify, XXCopy. Those are the non-EMC synchronization options that seem to be available to us, 'cause the EMC solutions aren't cheap enough for our budget. Asigra, CloudBerry, CrashPlan PROe, Egnyte, EVault, Nasuni and Zetta are some of the enterprise-ready online backup contenders - and a few of them claim to offer server-to-server sync too!

Here's what I want from sync. Real-time, with bandwidth throttling by schedule. Honest-to-goodness two-way synchronization. Data compression. Bit- or block-level, not file-level. VSS or open-file support. A management console. For backup I want the following: Retention and versioning policies, some level of backup sets (not just a single copy of our data), a single vendor (don't make me buy AWS separately!), agents for my target servers and laptops, and a management console too. I figure it costs me about $600/month to run tapes - hardware lifespan of four years, tape lifespan of eighteen months and about an hour of my time a week changing tapes and reading logs. Can I beat that number?

I've conferenced with vendors, I've read the manuals, I've installed several trials and now I've made some choices. But tell me, how would you harpoon this whale?