How to reliably (and cheaply) store data?

Like many computer users, I have a lot of data, such as games, programs, photos, etc, that I wouldn't like to lose. And like many computer users, if I did lose this data, then I could just re-download a lot of it, and of what I couldn't re-download, probably half of it I wouldn't miss anyway, since I couldn't remember what it was. But anyway, I was wondering what good, inexpensive methods of back that there are.

I used to backup everything to DVD-R, but that's proven less than ideal, since they do occasionally start displaying errors and data loss, sometimes. Not often, but enough to make DVD-Rs untrustable for storing data that you want to last for years or even decades.

I now just use external USB hard drives, as they are reliable, fast, cheap, and they are re-writable. Being re-writable is good, but it does introduce of worry of a virus (or human stupidity) erasing forever valuable data. Plus the hard drive might just die, or become corrupted, and of course there are worries that an electro-magnetic pulse of some sort might destroy the data (this last one doesn't bother me, since if it happens then I presume we'll be at war so my PC will be the least of my worries, but I know that governments and businesses do worry about this sort of data loss, especially since it might well take out business/bank/criminal data records, but for me it's just my collection of games, utilities and songs).

Blu-Ray-R might be good, if in practical terms the discs are more reliable than DVD-Rs. How are they for reliability? And why are Blu-Ray-Rs still only 25GB, I thought that this could be doubled per disc?

I imagine that when SSD drives become cheaper, they'd make great backup devices. Do SSD's lose data over a long time of disuse? And incidentally, would they be vulnerable to EMPs?

What methods of backups do you use for your data?

 

I personally use HDDs for backup, but that's because you get a large amount of space, for a relatively cheap amount of money. I considered tape backup, but unfortunately the drives costs way too much money, as well as the tapes. Not to mention, they can't hold as much information per tape.

Considering you can get multiple TB hard drives for like $100~150, it's probably the cheapest thing to do.

 

M-Discs are very durable (http://www.zdnet.com/article/torture-testing-the-1000-year-dvd/).
DVD-R ones require compatible (typically LG) burners, BD-R are compatible with any decent BD-R burner.
Other than that, Raid 5 or 6 and copies in different physical locations.

An SSD basically stores electrons on transistors, so it'd probably get destroyed by an EMP, unless it's stored in a light-tight metallic enclosure (Faraday cage, doesn't need to be light-tight, but at least a grid smaller than the shortest wavelength emitted by the EMP; that's light-tight for that wavelength anyways).

 

Unlimited amazon cloud storage is 60 bucks a year. I have enough bandwidth and no data caps to make this feasible for me.

 

Hard drive, tape backup, remote (i.e. online) storage. Combination of all for safety.

Yup, Dropbox / Amazon etc. offer reasonable pricing - if you trust your data with them.

 

Just to entertain the scenario of EMP, I googled the situation, and apparently wrapping your hard drive with plastic (or some other insulation material) and then aluminum foil could protect from it. Do 2 layers of it.

As retro stated, a combination of backup methods is the best surefire method.

 

My backup setup is a raid5 nas with a home server running syncthing, regular backups of the nas on an external USB HDD kept elsewhere and soon M-Discs for long-term backup of personal stuff I really don't wanna lose, also stored away from home too. I wish I could also do online backups on Amazon, but my downloads/uploads are limited.

 

I'll never trust online/cloud storage. Too many backdoors and privacy issues, even with encrypted data. And always the chance that it will just disappear anyway.

 

RAID isn't backup.

 

That's why I got a backup of it.

I also use it as the first backup of laptops files using syncthing.

Using raid5 allows to hold lots of data with a one disk fault tolerance, it's a safer than nothing setup (not protecting against similar drives dying at similar times). But I backup what I can't buy back into another USB HDD regularly. Yes my DVD rips are not that safe, but I'm a few handbrake hours away from reencoding them anyways, while the self-made code, souvenirs and school stuff are properly backed up multiple times.

You should look into syncthing. Using you personal server (e.g. a raspnerry pi or what not) you sync directly between your devices without a third party. Everything is transferred securely and encrypted. The only third party service is the discovery server, but it's open source and you could run your own with a domain name/static ip if you're paranoid (not a bad thing).

 

Curious, if you encrypt before uploading, what other issues are you worried about? Worried they will disappear? Worried the client might be snooping on you?

 

If you fuck up that file (I sometimes accidentally write over a photo after watermarking rather than doing Save As (new file), for example), your fuck up is instantly backed up. If you get a virus in that file, the virus is backed up. That's the problem.

The safest way to back up is still to do it yourself. Yes, it's nice to have an automated system, but anything that's vital - back it up yourself. DVDs aren't a bad way of doing that immediately, to be fair. When I come back from a shoot, I'll copy my files from memory card to my computer, which will get auto backed up... but I copy them to an external or DVDs before I work on them.

Your computer is stored in your home. There's a locked front door - maybe even a locked porch door or entrance door if you're in an apartment. That's two doors to get through. There's also perhaps a back door - that's only one door to get through, and likely easier - for one, a burglar is not seen by public breaking in round the back. Then the burglar would have to locate your computer, either fire it up or steal it. And (hopefully) there'd be passwords and/or encryption to get past. In other words, your files are relatively safe in your house. However, your computer is connected to the Internet - so long as you don't do anything stupid and have a router with firewall, you are probably safe there. Yes, you could be physically broken into - but it's hopefully unlikely.

Online storage is on a server or maybe many servers in a cloud. They're relatively safe, as you most likely have to have ID to get past security in the data centre. However, your Dropbox account needs two pieces of information to be accessed - your e-mail address and your password. If someone really wants your info, they probably have your e-mail address. So they just need your password.

Have you ever had an account on Adobe, Major Geeks, Snapchat, Money Bookers, Xbox-Scene, PSX-Scene, Sumo Torrent, Android Forums, Forbes, Foxy Bingo, Paddy Power, vBulletin, Yahoo, Avast, Malwarebytes, Black Hat World, YouPorn, Ashley Madison or Adult Friend Finder? These, and many other sites, had vulnerabilities that allowed hackers to steal the database. They have your e-mail address. They have your (hopefully encrypted) password... which they may or may not be able to break. Do you use that password anywhere else? Dropbox, perhaps?

NOW tell me Dropbox is secure.

 

I really don't get your point @retro. How is using a raid array more dangerous than using a simple HDD? I mean the raid IS the backup, and I keep five versions of my files on syncthing, and my second backup of the file I can't regenerate/buy/download on the raid drive to another USB HDD is done incrementally every three months or so, or when I feel like a project deserves it...

The only files that are solely on the raid drive at any point are mostly reencoded of my DVDs done on handbrake, I can handle loosing them.

Edit: To clarify, my home server with the raid only automatically backs up a few specific folders with syncthing, keeping 5 versions of each files. The rest is all done manually.


2de edit: What you say is true about online sercoces, although if you use a long single used password to encode files locally, it should be safe to store them on Amazon servers for archival, the Amazon security features are just sugar coating on that case, your file could be public they'd be undecipherable until we get a quantum computer working.

 

Firstly, RAID IS NOT BACKUP.

Let's say you're saving a new version of a 4k video that's 90 minutes long, over the top of itself. That's a big file. You have a power outage. What happens to your RAID? What happens to both versions of your file?

Spoiler: Answer

The RAID probably corrupts, you've lost the original file as you were overwriting it, you've lost the new version as it hadn't saved - you have one garbage file... and a (partial) parity copy.

Now let's say you'd got the file and backed it up to a separate drive before editing it. NOW what happens when the power goes out as you save the file?

Spoiler: Answer

The new file doesn't save, the file you were overwriting is most likely corrupt... but you at least have a copy of it on your proper backup media

Real-life example: I created a RAID 1 (mirror) setup at my friend's photo studio. He is bad at backing up, other than weddings - he downloads the memory card RAW files to the computer, then backs them up to DVD. He then adjusts levels and converts them to JPEG, edits the JPEGs and backs the edited JPEGs up to DVD. With studio shoots, he merely copies the RAW files to the PC, converts to JPEG and edits... so there is at least a RAW "backup" locally.

A customer wants a photo restored, so he scans it in as a JPEG. There's no backup. He spends hours editing the photo, gets it looking great and saves it. He decides to watermark the image and make it smaller, to e-mail the client for approval. He should go file > save as.... but instead, he doesn't think and closes Photoshop, clicking yes to the save prompt. He saved over the hi-res photo with a lo-res watermarked version! What happens?

Spoiler: Answer

The RAID mirror copies the lo-res watermarked file, giving him two copies of the new file and no copy of the original!

RAID is NOT backup. RAID is FAULT TOLERANCE. RAID will overwrite your files as they are updated. RAID will copy the virus that attached itself to your file. RAID will copy the mistake you made and accidentally saved over the original file. RAID will copy the corruption in your Access database. RAID IS NOT BACKUP.

Likewise, syncing is not great backup if it syncs by automatically overwriting files with the latest version. If it creates a new version of the file - great! Your use of more than one technology is, as I previously said, the best method... but RAID on its own IS NOT BACKUP. Never mind the fact that RAID rebuilding is painfully slow... and if another drive failed (or fails during rebuilding), you're screwed.

As for Amazon storage, there's still the fact that someone only needs your password to access your unencrypted files - or is there some kind of two-step authentication? The average Joe sucks at using strong passwords. In fact, I believe Amazon allow really weak passwords, which is worrying!

 

For Amazon itself I can't say, but if you store properly encrypted files on the less secure network it still wouldn't be accessible to anyone but you. You could use multiple passwords for different files too, or two layers of encryption if you are really paranoid, but if you can't keep a single password secret, keeping two should be as much of a failure.

For the raid thing, I specifically said the raid was THE backup, I know raid itself isn't safe, and I've made that clear. I think your intention is to make it clear that raid alone isn't a viable backup solution and that's completely true but beside the point of arguing with me.

As for automatic syncing stuff, it's not a proper way to do a "real backup" but it's a life saver for that time when your laptop died at a conference or similar situations. You should still keep old versions of files and do a proper incremental backup of that folder if you wanna make it safe.

 

I actually have a bigger problem with using RAID-5 instead of 6. With hard drive sizes the way they are now, there is a larger than acceptable chance of a second drive failure before the rebuild of the first drive completes. I think that retro pretty much covered everything else...

 

Not to mention that they'll have to first know that you're a target of value, otherwise spending... years... on cracking your archives would be really of no value. That's really about the purpose of encryption - to make it very hard (and hence unfeasible) for anyone to brute-force their way to the data.

 

With crappily built hard drives the way they are now... That's the main issue - yes, sizes have made the problem more inherent, but hard drives seem to fail a lot more nowadays than was previously the case... not that any hard drive has an infinite lifespan.

 

I think FamilyGuy is trying to say that his backup server (in this case a NAS where he stores his backup files) uses a RAID array. His workstation doesn't use RAID. I agree with FamilyGuy that it's better his backup storage is on a RAID array than a single hard drive. Less chance of failure.

From what I understand, keeping multiple versions of files via syncing is very similar to incremental backup. The main difference is that current incremental backup implementations conserve storage space. (Of course you'd want to configure the syncing to keep versions of a file that are older than 24 hours, e.g.)

If you encrypt a file before uploading it to DropBox, you could post your DropBox password on Facebook for everyone to see, but your data is still secure. This is what I instruct friends and family to do who are insistent on using "cloud". (Not the posting password on Facebook, the encrypting before uploading.)

 

If your data is not in your control, it is not secure.