Social backup

now here’s an interesting idea …

I’ve had a server set up in one of my houses for some time now, it’s got a number of whopping hard drives in it and a copy of foldershare. Various members of my family also have foldershare installed and then invite me to their library. Thus all their files are backed up securely to a different location.

I also use foldershare for keeping my three active computers in sync with all my work files.

Up until now this has worked seamlessly – i’ve never had a problem with the caps (10000 files per library). But recently i’ve noticed that my work iMac is not keeping in sync with my macBook. i tried to force a resync by recreating the library, but this just resulted in a whole load of files being deleted (recoverable) from my macBook.

So foldershare was getting things wrong. And at that point I lost faith: i can’t have a backup or sync app getting sync directions wrong. Fatal!

So I’ve spent the last few days looking at alternatives. The winner has to be cross-platform (as most of my family use PC and I use a mixture of linux and mac).

I’ve been quite impressed with DropBox: but it’s still in beta AND i have to put folders into the DropBox folder, which is a limitation too far. I can get around this in mac/linux with symlinks, but not windows. DropBox devs are working on a ‘folder watch’ and at that point i’ll take another look. Although it does suffer from the same problem I outline below.

I’m also looking at SugarSync, which i have actually installed on a couple of computers. It looks like it’s a neat solution. I prefer the DropBox interface – though: their web interface is very neat.  

Neither DropBox nor SugarSync are free.  Both cost about $100 a year for a bunch of MB.  Which is fine: i have no problems paying for a comprehensive solution.

I should add that both do version management (nice to have) and both allow files to be emailed/made public.  This is a nice feature as on big outsourcing contracts, the tech spec/input specification etc sometimes gets très bloated with visio diagrams that end up killing transaction workflow simply by many email hosts refusing to handle attachments over 10MB.

Drawbacks

I’m (very) concerned about the concept of my data being on line in someone else’s storage facility: this isn’t like a hosting facility where i have stacks of my own blades. I have no control over what they do with the data, no control even over where they store it! i might be able to get an NDA (non-disclosure agreement) from them but the chance of being able to enforce it renders it useless. So i’d be relying on reputation alone. Not a great situation to be in.  I might offer some advice to these guys:  allow users to specify their own storage locales: say their own account on Amazon/ their own webDAV server / ftps server / svn server or whatever. Charge me for the interface, charge me for the management: find some other hook to get me to pay: but make me feel comfortable about the data.  I’m happy that my webhost is secure (kind of – as I have root access to the box and trust them not to do any physical snooping), i’m happy that my own servers are secure against anything other than physical break-ins.  So: to messrs DropBox and SugarSync: be storage abstract please.

There are workarounds to these issues.  One is to put a truecrypt folder into the dropbox/sugarsync folder. another is (probably) to use a sparseimage.  I have problems with both of these for two reasons:

• i’m having to change my working habits just to use software.  This is reminiscent of the pain I have seen accounts departments go through when they are told to start using SAP R3.  Suddenly all processes have to change.  Why? because someone in authority thinks a SAP implementation will get them a nice bonus because … it costs lots of money and so ‘must deliver value’.  A bit like hiring McKinsey to tell you how to do your job.
• i’ve had problems with corrupt archives before.  and corruption kills all your files, rather than just one.  I’m not confident that the failure-cases for these products are robust enough: operating systems can just die in too many different ways…

But the solution that tempted me into writing this post was Cucku. This has no websharing, and is primarily a backup tool.  The truly neat thing is that it backs-up your designated files (and deltas) to a designated friend’s computer (using skype as an intermediator [curiously]).  It’s Windows only (which rules it out for me) but I really like the idea of social backup.  It mirrors what I have been doing for a number of years. The folks at cucku have made sure that the backed up version is secure from even your friend’s eyes too.  and using skype means that the transport channel is encrypted.

But of all of these, only foldershare comes close to doing what I actually want.  Which is simply to keep a set of folders absolutely faithfully replicated across multiple computers.  And to do so automatically, in the background, using minimal bandwidth (all of the above products sync only the file deltas – byte wise). Yes, I could set up VPN’s and use rsync or chronosync or whatever: but this results in a manual solution and is not a byte-wise update.  Foldershare also provides a web interface to retrieve files.  But unlike the other products, this web-interface requires one of my computers to be online as MS do not store my files: it’s p2p only.  This limitation is just fine for me given my scenario as a social backup centre.  there’s iFolder too, which could perhaps be made to work for me as I could become an iFolder server.  But it no longer looks to be under development and the interface is horrendously clunky.

Other solutions also exist and the above is far from comprehensive.  Powerfolder touts itself as a foldershare equivalent.  I’ve tried to set it up a couple of times but it’s too clunky too.  

Oddly enough, another product which really did work was MS Windows with Windows Server, using roamed accounts with offline storage.  that worked well, albeit with some annoying quirks.  Macs have something similar but I’m not keen on setting up a Leopard Server and I don’t really want all the macs as account based computers.  Not to mention that it’s not a cross-platform solution.

I’m not even going to talk about iDisk/Mobile Me.

i wish there was a solution that ‘just worked’.  I can’t be the only person that needs a reliable, automatic, painless syncing mechanism.  And it’s not complex either.  Really!.  It’s just a p2p interface with some straightforward business rules on conflict handling.  

If anyone can recommend something that gets close to fulfilling my needs: i’d be very much obliged.