Backing up a Linux system to Usenet

https://arstechnica.com/information-technology/2017/02/bored-with-ho-hum-cloud-backups-use-usenet-yes-usenet-instead/

by Tomte • 9 years ago

80 79 9 years ago

arstechnica.com

thieving_magpie• 9 years ago

Bored with ho-hum cloud backups?

Well I can't say that's something I've ever thought.

jasode4• 9 years ago

With access to a Usenet news server, you can simply upload your backup there, and it will be stored redundantly in news servers all over the world. Best of all, this approach typically costs considerably less than a cloud backup service.

I saw no mention about "retention policies" in the article. For newsgroups with binaries, the other Usenet peer servers can choose to download only text and ignore binaries. Or they only hold binaries for 30 days or whatever discretionary time period they choose based on available disk space.

With AWS Glacier, Backblaze, etc the data retention would be explicitly specified.

[1] example of limited retention period: http://www.giganews.com/blog/labels/retention

problems6• 9 years ago

Most paid Usenet services these days advertise their retention time of > 1000 days.

Some of the big providers in the business right now:

https://www.easynews.com/usenet-plans.html 2600+ days

http://www.supernews.com/ 2300+ days

http://www.news.astraweb.com/ 3000+ days

Easily enough for some backups. Of course, there's no guarantee of integrity, but it might work for some people.

jasode4• 9 years ago

Of course, there's no guarantee of integrity,

Exactly. It's been a decade since I've used Usenet but I remember there were always tons of messages complaining, "part43.rar and part62.rar is missing please reupload!!!"

As another anecdote, I personally posted a huge list of AWK one-liners 20 years ago to Usenet (comp.lang.awk) and I can't even retrieve that message. That's just pure ASCII text (probably less than 2k) and even the Google Groups (dejanews acquisition) archives don't have it.

loeg4• 9 years ago

Common Usenet binary retention is north of 3 years today, shared across many providers. Yes, you will need an account at one of those providers to download the backup in the future. And yes, nothing is guaranteed about retention.

dillonb4• 9 years ago

While interesting, this seems like a bad idea. You're uploading your backups, no matter how encrypted, to a place where they will be publicly available to download.

problems6• 9 years ago

Most cloud backup services are worse - they do no client side encryption, your files are freely available to the service provider or anyone who can break in.

I'd be much more comfortable with this personally. Trust the math, not the people.

naiyt2• 9 years ago

Or you could just use Tarsnap, where you can trust the math and have private backups for cheap.

dillonb4• 9 years ago

Absolutely, but having another layer blocking access to your data is definitely a good thing. It's a good idea to encrypt your files yourself before uploading them to a public cloud.

therealmarv7• 9 years ago

Exactly. I rather trust well proven math more than people or infrastructure. One famous example nowadays is Bitcoin ... nobody was able to break the fundamental math behind it.

therealmarv7• 9 years ago

Nobody cares about data junk. Especially your personal data junk if it is all encrypted. I don't think that a lot of persons will look at your data there. If this sort of good encryption you consider for public cloud backups breaks we have a lot more problems than exposed backups.

dillonb4• 9 years ago

Saying "nobody cares" about your data is not a good security policy.

bb1012• 9 years ago

Who knows? Your private backups might even end up being conveniently available test data for nascent quantum decryption software.

834572• 9 years ago

Idea: An open source peer-to-peer backup service where data is encrypted and backed up in pieces across countless systems around the globe. No one person had your data but in some way it would be guaranteed your data would always be available. Everyone who uses the backup service would be required to also accept data pieces for backup.

dublinben• 9 years ago

That sounds just like Freenet or IPFS.

tambourine_man• 9 years ago

That is the dream yes.

Maybe with: https://ipfs.io

lb1lf• 9 years ago

It is called BitTorrent. Generate the backup file as described in the parent article, give it a saucy name involving a (female) celebrity and let it out into the wild. As disk space is effectively free and people tend to hoard files, I suspect you'd have no problems recovering your file a year or two down the line.

(I tried this way back when using WinMX (anyone remember THAT?) to see whether it would work. It did.)

We dubbed it 'hardcore backups.'

lozf• 9 years ago

You could even throw a little steganography in to the mix, so those who check for the content they want, but pay little attention to file size continue to participate in storing your data!

problems6• 9 years ago

IPFS, Freenet, Zeronet... maybe more significantly Maidsafe is trying to do this with a cryptocurrency so you get paid a bit to store files. It's a quite clever idea.

HuggableSquare• 9 years ago

There was a paid version of something like this called Digital Lifeboat. IIRC it ran on top of the BitTorrent protocol, and users would store some pieces of other users backups. It didn't really catch on and shutdown a few years ago.

madjam002• 9 years ago

There was something like this for years called Symform but they shut down recently: http://www.backupreview.com/symform-to-discontinue-peer-to-p...

I guess this was more of a commercial solution though.

coolspot• 9 years ago

Etherium developers working on Swarm[1][2] which is p2p storage with incentive and smart contracts!

[1] https://blog.ethereum.org/2016/12/15/swarm-alpha-public-pilo...

[2] https://github.com/ethersphere/swarm

therealmarv7• 9 years ago

Has anyone experience with that in practice? What are the data limits on this kind of backups?

loeg4• 9 years ago

It works well enough. You can find binary providers that charge by download quantity and use that style of plan to upload for free. (And upload is usually free -- usenet providers want more content, after all.)

Name your backups something inconspicuous and have your client generate an .nzb file for later recovery (has the unique mail ids for each component mail in it).

It's a little risky in that maybe providers would delete your backups if they identify a pattern, or if you name them wrong some studio may issue a (bogus) DMCA takedown against them.

Tarsnap is cheap enough that I wouldn't bother with any of this unless you're already using a binary usenet account anyway.

ashark• 9 years ago

Related-ish-ly, I've had a notion bouncing around in my head that it should be possible to build a fairly complete social networking interface on top of email and ical (or similar), and some kind of contact-management standard, such that the whole thing (or at least enough of it) could be rebuilt from e.g. your (possibly dedicated to this purpose?) email account when necessary, possibly in stages—say, a new installation of the application builds what it can from the last 30 days in a few seconds, then displays a populated interface while it continues digging to some arbitrary point in the past (until it exhausts caching resources it has been allotted on a given platform, basically).

An initial search has revealed a couple abandoned partially-complete efforts, but that's it.

teddyh• 9 years ago

I believe that p≡p is aiming to implement something like that in due time. They had a talk at FOSDEM this year: https://fosdem.org/2017/schedule/event/pep/

bb1012• 9 years ago

This has to be a joke, surely? A good example of the Tragedy of the Commons...

As ProfessorGuy succinctly commented: > Why not an article on how you can get a bed for yourself in the local hospital so you won't have to pay rent? Hey, if they're going to build a public institution like a bunch of suckers, they deserve to be taken advantage of!

wowtip• 9 years ago

This is why we can't have nice things...

aw3c2• 9 years ago

This would be a drop in the bucket compared to the terabytes of warez uploaded daily. https://en.wikipedia.org/wiki/Usenet#Usenet_traffic_changes

WillPostForFood• 9 years ago

While there are different, and maybe bigger ethical issues there, at least it is using the system as designed.

arjie2• 9 years ago

Amusingly, there's alt.binaries.backup https://www.binsearch.info/browse.php?server=2&bg=alt.binari.... I found out about its existence through the following two earlier threads regarding this approach:

https://newsgroupdirect.com/blog/2012/04/07/usenet-for-file-...

https://www.linkedin.com/pulse/non-scalable-anti-social-back...

Does anyone know what the deal with alt.binaries.backup is? Why would anyone mirror?

apricot• 9 years ago

Also, if you ever get bored putting junk mail in your recycling bin, why not mail it to the author of this article?

comstock• 9 years ago

Another idea:

Split your data into ping packet sized chucks and throw them out on the Internet. Keep all your data bouncing around on the Internet and use that as your storage provider.

Not really practical, but a neat idea. Saw in the excellent "Silence of the wire": https://www.nostarch.com/silence.htm

arjie2• 9 years ago

It's analogous to delay line memory, and a common joke people when they encounter that idea of storage (which is rather clever).

Florin_Andrei• 9 years ago

Point a laser at the empty sky, modulate it with your backup file.

It's gonna last forever.

lightbulbjim• 9 years ago

Already implemented. See: http://esoteric.codes/post/123650565028/pingfs-storing-your-...

stevekemp• 9 years ago

My personal favourite was sending data in outgoing emails to invalid destinations. Your data would get queued and a few days later you'd receive it back in an undeliverable-message.

However this stopped being effective when mailserver stopped sending back the complete body of the bounced mail. Instead just a "Your mail couldn't be delivered, blah blah". You could deal with that by encoding your data in the subject-lines, though I guess ..

kosma• 9 years ago

For most users, however, you’ll probably find it costs you around $10 per month to maintain the Usenet service in this manner.

Which is more than a dedicated cloud backup service like Backblaze or Crashplan. Genius.

wsha• 9 years ago

If you read the whole article, the author mixes in caveats throughout and says that it requires a special set of requirements for the usenet backup to be worthwhile. At some point, he addresses your point by saying that in some cases (possibly requiring careful coordination between all of the servers) you could backup many different servers with the same usenet account whereas services like Backblaze and Crashplan usually restrict the number of machines.

That said I still think the point you make is the most important critique. Online storage/backup is such a crowded market (Google, Amazon, DropBox, Box, Backblaze, Crashplan, Carbonite, Mozy, tarsnap, Spideroak, rsync.net, OneDrive, etc) that if you go with something significantly cheaper than all of the mainstream options you are probably going to get what you paid for.

tdkl• 9 years ago

Seems like someone got the task to write something controversially stupid to get traffic, because if you proposed such scenario to any real life company, you'd be laughed out of the room instantly, disregarding "20 years of experience".

benzinschleuder2• 9 years ago

Some time ago, I implemented a little tool for myself which backups folders incrementally on usenet (deterministic message-id creation from a secret key, append-only style with metadata, parity, encryption etc. so you only have to remember one unique key to access all of your data, even if new data is added).

It can mount the current state of the usenet backup with FUSE and it's possible to browse through the files and listen to music etc. I understand that it might not be a good idea to store all your data only on usenet, but I thought that was an interesting concept and a fun little project to work on :)

ycmbntrthrwaway3• 9 years ago

Open source or didn't happen.

benzinschleuder2• 9 years ago

at the moment its very tailored to macOS and has plenty rough edges because its for personal use. But I'll gladly clean it up a bit and put it on github if enough people are interested and/or i will write-up how its "protocol" works. Just let me know (twitter pm or sth.) who is interested.

EDIT: basically it's one deterministic stream of messages for journaling everything, one recursive stream of folders and linked raw files. deterministic lookup rougly like: HMAC(type|index|revision|replication, key_for_locating) and it iterates through that

EDIT2: and one for parity of meta and raw

buzer• 9 years ago

Reminds me of

"Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" - Linus Torvalds

ddp• 9 years ago

This is one of those ridiculously pointless exercises that keep me coming back to Hacker News for more. What a waste.

http://www.tarsnap.com/

ycmbntrthrwaway3• 9 years ago

Encrypting your backup with a passphrase and making it public is a bad idea. It is not forward secure. Anyone can try to bruteforce it offline. You better use a keyfile and backup it somewhere else, such as your other computers, external drives etc. And even then, publishing your backup is a bad idea due to possible breakthroughs in cryptography. No crypto is secure forever, you better assume that it will be broken in the next 5-10 years.

storrgie• 9 years ago

this kills the usenet

cestith2• 9 years ago

From Amazon Lightsail I can get this for $5 a month:

* 512 MB Memory

* 1 Core Processor

* 20 GB SSD Disk

* first 1 TB of outbound Transfer

That's a whole VPS I can put a git repo on and upload encrypted files to using ssh. So I get gigabytes of storage, secure transfer, a commit log, and can do some light other work on it. Or I could just rsync or scp them there.

GitLab is even one of their preinstall-able images, or you can pick some other application/stack (MEAN, LAMP, Nginx, Drupal, Node.js, Redmine) or a bare OS (Amazon Linux or Ubuntu) and initialize your own bare repo.

For the $20 mentioned in the post, I could set up four boxes in four AWS zones. I could configure my backup scripts to do a duplication rate of only two full copies and have roughly 40GB of stuff backed up fully duplicated in physically disparate data centers.

This isn't even getting into elastic storage offerings or paid cold storage which is dirt cheap by comparison. This is just simplified point-and-click deployment of predefined images, and it's already far more reliable for the money.

If one really wants to spend monthly to rig up their own backup solution rather than just using Amazon Drive, Google Drive, or Dropbox or something similar, this at least makes the stuff easy to find, redundant, and simple enough without polluting the public forums.

If one really needs two or three year data retention offsite with multiple copies and wants to only pay for the upload and download without paying for storage in between, just buy outright some 32 GB USB flash drives for $8 a pop and mail them to three friends. There, you've paid $30ish one month for hardware, postage, and taxes and will pay one friend $1 in postage to get the data back.

boomboomsubban• 9 years ago

For the $20 mentioned in the post, I could set up four boxes in four AWS zones.

And for the $2.50 also mententioned, you could set up zero. Or you could buy a bandwidth based plan, where $20 will get you at least 250GB of bandwidth.

Fwirt• 9 years ago

It's actually quite possible to get an unmanaged VPS for under $2 a month if you shop around. I'm currently renting one for use as a personal email server for £0.002 per hour, which works out to around $1.85 a month for 1 core, 1GB ram, 20GB storage, and 2TB bandwidth.

https://lowendbox.com/ is a good resource for finding cheap VPS offerings to mess around with.

ourcat• 9 years ago

I've often thought that Usenet binaries would be perfect for podcasts.

Backing up a Linux system to Usenet

Related Stories