Getting Organised

Well it isn’t even close to being Spring, but it was time to do some hard drive cleanup, so I’ve spent the past few days sorting through the hard drives of my in-home server, backups and the machine I use for photo editing and video processing. As I wrote here last year,  I have been using the Amazon S3 service as an off-site place to store my photos and I use Bucket Explorer as the front end/user interface. The combination works incredibly well, and so when my brother-in-law John was asking about what off-site storage option he should consider for his several thousands of photos, I had no trouble recommending it to him as an option to explore.

John is the same fellow whom I convinced to by an iMac, as he really wanted to just use a computer rather than have to figure out how it worked, and he has been very pleased with it. I am still running Microsoft’s XP for photo work only because I’ve invested time learning Microsoft Digital Image 10 suite for processing my photos, which is similar to iPhoto on the Mac, and I am comfortable using it. Sadly, Digital Image Pro 10 has been dropped as a standalone product by Microsoft, although some of its elements have since been merged into Vista. While I suppose I could change out XP for Vista, (and I actually do have a couple of copies of Vista sitting in boxes unopened)  based on my experience while working on the pilot to deploy Vista in Shell, it would frankly seem more like a down grade from XP rather than an improvement.  I suppose I’ll either have to wait for Vista’s replacement called Windows 7 to be released  and look at it then (actually not till after at least its first service pack is released of course), or perhaps move fully over to Ubuntu’s version of Linux which is what I have running on most of my other machines – or just buy a Mac and not have to mess around as much as you do with Windows products. We’ll see what happens.   

As I started into the disc cleanup, it became quickly obvious that I needed something that would allow me to easily check folders and files for duplications and differences in file size, content, etc. I wanted to be able to ensure the photos were sorted, and at the same time be certain that none of the versions of the photos were lost or possibly over written due to having the same file name and yet be different in some way (colour corrected copies, cropped or scaled copies etc).  There are quite a few utilities available to help you manage these sorts of issues, and it is disappointing that something isn’t built into the MSWindows OS as it is a fairly basic requirement. After doing some checking around, I decided to try  a freeware product called WinMerge, and I have to admit I really like it. It sort of reminds me of another utility I used many years ago called F31, although it is much more sophisticated. You can do light weight comparisons based on just date and size, or more extensive binary file comparisons to determine differences in files sharing the same name, and then synchronising the contents of the directories is a snap. Quite useful.

As I type this, I’m just waiting for my updated photo library uploads into Amazon’s S3 environment to complete, then I’ll refresh the copy that I keep on a removable hard drive which I keep stored in our safety deposit box after which I’ll be ready to start on my next little project – scanning and storing my slides, negatives, family films and also a few audio cassette tapes that my grandparents made for us as children.  As they are long gone these many years, it is nice to still be able to listen to them once and a while – and ideally I’ll get the tapes converted before it becomes impossible to find a way to play them !  

I expect this will keep me busy for a little while …….

Amazon S3, Backups and Other Things

Most folks these days have a digital camera, and as a result have hundreds and possibly thousands of digital photos and videos littering the hard drives of their computers. Certainly this is true for me and of course it highlights one of the dilemmas if living in a digital age. Crash your hard drive, and your photos are gone forever. Admittedly not all of them are ‘perfect’ shots, but out of every hundred or so there are bound to be a few that are precious to you and which you really do not want to lose.

In prior generations, there were generally lots of ‘hard copies’ of photos either in photo albums or boxes stashed away in closets, and copies of the more memorable photos would be made and sent around to relatives etc. This was a good thing, and back then, if I were to lose a copy of the photo of my great aunt Lulu (I actually don’t have an aunt by that name, but you get the idea) there would have been a good chance I could either dig through my shoe box full of negatives and have a new photo printed, or perhaps could ask a family member for a copy of a photo they had and I’d be back in business. That was then, and this is now.

Like many people, I starting burning CD copies of my photos, and then as the numbers and sizes of the images grew and video clips were added to the mix, eventually DVD copies which I keep in a safety deposit box at a local bank. This has the advantage of it being a relatively cheap option and also of the photos being kept off site so if the house burns down they will still exist. Of course it has the disadvantage of me having to constantly update and replace these discs as new photos are added, which is time consuming and like most people, I’m not quite as disciplined as I should be.

Off site storage for a few photos in something like WebShots is an option which many people use, and there are many others like it, as well as the various ’social network’ solutions such as Facebook, but in general the advertisements drive me crazy, so I refuse to use them for photos. To avoid all the ads, and for displaying a handful of photos that you really like, there is the option of putting a few photos up on your own web site as I have done, or as a friend of mine has done. But what do you do to replace the shoe box full of negatives you used to keep filed away, ‘just in case’, and how do you ensure it is not computer dependant as well as being easy to use ? Well, I’m not sure what you are using, but I am using Amazon S3.

Before retiring from Shell, I did a writeup on the various Amazon Web Service offerings, and I like what they have done – they have platforms for applications development, commercial platforms for managing and running applications, collecting money etc., and they also have a bulk storage solution. The storage solution (called Amazon Simple Storage Service, or S3 for short) distributes your data across many servers, is highly fault tolerant, is accessible from any computer (Linux, Mac or Windows) anywhere that you have a decent internet connection, is hugely scalable, and is pretty cheap (you just pay for what you actually use). All of this makes it quite attractive.

By itself Amazon S3 is not even slightly user friendly, as it is targeted toward developers, but there are a few companies who now offer beta (of course !) software you can use as a user interface for the S3 service to allow you to store all your ’stuff’ on the Amazon S3 servers. I actually have about 17 GB of files sitting up there ‘in the cloud’ and other than a couple of horrible moments which I’ll get to in a minute, I recommend it. For me at least, as a back end storage service Amazon S3 has been stable, reliable and accessible. But you do need a front end interface of some sort to use it, so read on.

Before starting, you need to decide what it is you actually want to be able to do – in this instance, although you can use Amazon S3 as the storage location for the data displayed on your web site, what I was looking for was a replacement for the shoe box I had for my negatives. I admit only a few of the photos are masterpieces (to me at least), but on the other hand saving everything gives me the freedom to pick and choose whenever I want. Having decided on the ’shoe box’ approach, there are quite a number of commercial alpha and beta products out there – many of them are listed in the Amazon Solutions Catalog. Some of these are good – some of these are frankly awful, but all of them demonstrate the possibilities and will get better with time.

Three of the interface solutions that I tried out to allow me to just be able to drag and drop files from my hard drive into Amazon S3 buckets (which is what they call the storage locations you have access to in S3) are S3Fox, Bucket Explorer, and Jungle Disk.

S3Fox looks nice, but I (and many others) could not reliably get it to work – no idea why. For the moment, I’d suggest it be avoided till it has a few of the kinks worked out. It also requires Firefox and does not work with Microsoft Explorer, which I think is OK as Firefox is what I’d recommend anyone to use if asked, but it does freeze them out of a large part of the market

Bucket Explorer and Jungle Disk are roughly similar to the casual eye, but did offer me a few stressful moments. After creating a bucket with Bucket Explorer and filling it with my photos, I then tried to open it with Jungle Disk – no go. Everything appeared to be gone, other the the top most level which only had the bucket name. Likewise when I created a bucket in Jungle Disk, Bucket Explorer could not read the name of the bucket, or the names of the files it contained. I’m sure things like this will get fixed over time, but for the moment – for the casual user – I’d suggest picking one or the other and not to try to mix and match.  By themselves, both work and are good choices.

I will continue looking at new options as they become available, but for anyone looking for a digital shoe box for their digital images, I think the Amazon S3 product with the user interface of your choice is definitely a good option. For the moment I’m happy – and sticking with Bucket Explorer !