Getting Organised – continued

Life is never simple is it ?

Following along after my last post, I was just finalising my upload to the Amazon S3 storage, and then had planned to update the removable hard drive we keep in a safety deposit box, but I first decided to take a few minutes and do a bit of spot checking. In this case, the ‘few minutes’ turned into a few days …….. 

As I checked through the photo files I noticed that here and there – not very often, but often enough to be annoying – the thumbnail view of the photo files was not being displayed. I have seen this before, and if you Google this problem you’ll find that it is a commonly known Microsoft feature. In the past I’ve typically just refreshed the file from one of my other backups and continued on. However this time, I happened to try and open a few photos on either side of files which were missing the thumbnail view and found a new problem – here and there although the thumbnail of the photo looked fine, when I tried to actually open the file there was either no valid JPG file there, or the file opened as a black screen. The result varied with the tool that I used to open the file.

I reached for my bottle of antacids, as this was not good at all.

Everyone knows hardware will fail – that’s just a given as stuff does just wear out and frankly you are a fool if you don’t plan for it, but a search of the web showed that, as for the missing thumbnails, file corruption is also a fairly common problem for which the most common connection is the use of Microsoft products. In fairness given the percentage of the market Microsoft  successfully controls this is not surprising, and it could happen with any vendor’s software – it just doesn’t make it any less annoying.  

I’ve done much Googling for possible solutions, but have not found anything that I would call a definitive answer, although file transfers for USB devices and file transfers in general do seem to be a common issue area. In my own case I suspect that my problems may have started around XP SP1 time as there were some reported driver related issues, and I just had not noticed. Vista users seem to have had this sort of problem a lot, giving me even more incentive to not use that product and its related applications, but at the same time I’m not convinced either MAC or Linux based solutions would be trouble free either as it is all software, so by my definition can’t really be trusted. What I have done is gone back to basics, and double checked that I have all the XP patches and applications updates, plus as per a few suggestions I happened across, I have also updated the firmware on the ASUS motherboard of my XP machine. While I was at it, just for good measure I also updated the firmware on my D-Link DNS-323 home server as a number of incompatibilities with this device were flagged when Vista was released. Doing firmware updates to otherwise working hardware is generally something I try to avoid as you can create more trouble than you solve. In this instance although I’m not using Vista, it wasn’t clear to me whether Vista ‘features’ may not have been introduced to XP with the later SP2 and SP3 XP updates, which may then also have possibly introduced similar Vista file corruption problems to my XP platform.

I have spent a lot of time this past couple of days, doing file transfer tests and I can’t reproduce the problem so for the moment at least it is on ‘monitor’. On top of the testing, I have also spent many, many hours going through my full digital photo collection accumulated over the past 10 years, and at the end of doing so have determined that about 15,000 or 25% of the photos were suspect or had been trashed ! I have been able to recover these files from other backups (I’ll admit that I am anal about backups), and then have reloaded these back into my Amazon S3 storage space.

When you consider the digital age we live in, and increasingly the many years of photos and fond memories that exist only as bits either on a local hard drive or as bits kept somewhere else, it is sobering to consider ones digital past could so easily disappear due to a glitch, be it either software or hardware related. Following this experience, my suggestion to anyone out there who is using either XP, Vista or frankly anything else, and who thinks that their photos are all safely stored on either a local, removable or network hard drive, CD or DVD should perhaps take a look at them and do some in depth checking. You may be surprised, and not in a nice way. In the meantime, I’m going to restock my supply of antacids …….

Amazon S3, Backups and Other Things

Most folks these days have a digital camera, and as a result have hundreds and possibly thousands of digital photos and videos littering the hard drives of their computers. Certainly this is true for me and of course it highlights one of the dilemmas if living in a digital age. Crash your hard drive, and your photos are gone forever. Admittedly not all of them are ‘perfect’ shots, but out of every hundred or so there are bound to be a few that are precious to you and which you really do not want to lose.

In prior generations, there were generally lots of ‘hard copies’ of photos either in photo albums or boxes stashed away in closets, and copies of the more memorable photos would be made and sent around to relatives etc. This was a good thing, and back then, if I were to lose a copy of the photo of my great aunt Lulu (I actually don’t have an aunt by that name, but you get the idea) there would have been a good chance I could either dig through my shoe box full of negatives and have a new photo printed, or perhaps could ask a family member for a copy of a photo they had and I’d be back in business. That was then, and this is now.

Like many people, I starting burning CD copies of my photos, and then as the numbers and sizes of the images grew and video clips were added to the mix, eventually DVD copies which I keep in a safety deposit box at a local bank. This has the advantage of it being a relatively cheap option and also of the photos being kept off site so if the house burns down they will still exist. Of course it has the disadvantage of me having to constantly update and replace these discs as new photos are added, which is time consuming and like most people, I’m not quite as disciplined as I should be.

Off site storage for a few photos in something like WebShots is an option which many people use, and there are many others like it, as well as the various ’social network’ solutions such as Facebook, but in general the advertisements drive me crazy, so I refuse to use them for photos. To avoid all the ads, and for displaying a handful of photos that you really like, there is the option of putting a few photos up on your own web site as I have done, or as a friend of mine has done. But what do you do to replace the shoe box full of negatives you used to keep filed away, ‘just in case’, and how do you ensure it is not computer dependant as well as being easy to use ? Well, I’m not sure what you are using, but I am using Amazon S3.

Before retiring from Shell, I did a writeup on the various Amazon Web Service offerings, and I like what they have done – they have platforms for applications development, commercial platforms for managing and running applications, collecting money etc., and they also have a bulk storage solution. The storage solution (called Amazon Simple Storage Service, or S3 for short) distributes your data across many servers, is highly fault tolerant, is accessible from any computer (Linux, Mac or Windows) anywhere that you have a decent internet connection, is hugely scalable, and is pretty cheap (you just pay for what you actually use). All of this makes it quite attractive.

By itself Amazon S3 is not even slightly user friendly, as it is targeted toward developers, but there are a few companies who now offer beta (of course !) software you can use as a user interface for the S3 service to allow you to store all your ’stuff’ on the Amazon S3 servers. I actually have about 17 GB of files sitting up there ‘in the cloud’ and other than a couple of horrible moments which I’ll get to in a minute, I recommend it. For me at least, as a back end storage service Amazon S3 has been stable, reliable and accessible. But you do need a front end interface of some sort to use it, so read on.

Before starting, you need to decide what it is you actually want to be able to do – in this instance, although you can use Amazon S3 as the storage location for the data displayed on your web site, what I was looking for was a replacement for the shoe box I had for my negatives. I admit only a few of the photos are masterpieces (to me at least), but on the other hand saving everything gives me the freedom to pick and choose whenever I want. Having decided on the ’shoe box’ approach, there are quite a number of commercial alpha and beta products out there – many of them are listed in the Amazon Solutions Catalog. Some of these are good – some of these are frankly awful, but all of them demonstrate the possibilities and will get better with time.

Three of the interface solutions that I tried out to allow me to just be able to drag and drop files from my hard drive into Amazon S3 buckets (which is what they call the storage locations you have access to in S3) are S3Fox, Bucket Explorer, and Jungle Disk.

S3Fox looks nice, but I (and many others) could not reliably get it to work – no idea why. For the moment, I’d suggest it be avoided till it has a few of the kinks worked out. It also requires Firefox and does not work with Microsoft Explorer, which I think is OK as Firefox is what I’d recommend anyone to use if asked, but it does freeze them out of a large part of the market

Bucket Explorer and Jungle Disk are roughly similar to the casual eye, but did offer me a few stressful moments. After creating a bucket with Bucket Explorer and filling it with my photos, I then tried to open it with Jungle Disk – no go. Everything appeared to be gone, other the the top most level which only had the bucket name. Likewise when I created a bucket in Jungle Disk, Bucket Explorer could not read the name of the bucket, or the names of the files it contained. I’m sure things like this will get fixed over time, but for the moment – for the casual user – I’d suggest picking one or the other and not to try to mix and match.  By themselves, both work and are good choices.

I will continue looking at new options as they become available, but for anyone looking for a digital shoe box for their digital images, I think the Amazon S3 product with the user interface of your choice is definitely a good option. For the moment I’m happy – and sticking with Bucket Explorer !