Sia – Decentralized cloud storage network

https://blog.sia.tech/meet-sia-the-most-viable-non-financial-application-of-blockchain-technology-afe6e7412a25

by Geee • 7 years ago

276 192 7 years ago

blog.sia.tech

yasp• 7 years ago

How do services like Sia handle illegal content?

jameslevy• 7 years ago

I believe that censorship resistance is a key value proposition for decentralized file storage. So its agnostic handling of illegal content is a feature, not a bug.

yazan94• 7 years ago

In the spirit of blockchain and decentralization, I think you're right. But practically, explaining that to the authorities will probably not be a pleasant experience if they manage to find cp or other illegal data on your hard drive.

pradn• 7 years ago

I wonder about whether it's right to be renting your storage space away without knowing what it's being used for. I don't want to directly contribute toward bad things happening.

Ajedi3211• 7 years ago

All data is fragmented and encrypted, so it's impossible to know whether a particular host is storing a fragment of illegal content or not. I suppose if you somehow _did_ find out you were hosting a fragment of illegal content you could breach the storage contract (and pay the resulting penalty) but that's very unlikely because, again, all content is encrypted.

alexkcd2• 7 years ago

Seems like this could be a common use case for pirates. Similar to how Megaupload or Fileshare worked a few years ago.

Sia's encryption doesn't protect the file hosts, if the uploader posts a link + decryption keys for pirated content to a public file sharing forum.

Does the take down notice go to everyone that hosts a fragment of the file? Who is liable if the uploader can't be traced?

asdkhadsj2• 7 years ago

Depends on how it's implemented though, does it not? Eg, I can encrypt a file and give it to 10 people and they'll be clueless of what is in it, yes. But, because of how I encrypted it it is clear that the contents are the same on those 10 people.

So if I am busted and my key/etc compromised, it is possible that those hosts are compromised as well.

This says nothing of Sia's implementation, just that encryption itself does nothing to prevent a party from being prosecuted from illegal content.

mtlynch3• 7 years ago

Encryption on Sia happens on the client-side, so the host has no guarantee that the file is actually encrypted.[1]

For encrypted data, I think it will be similar to the situation with other storage providers. If law enforcement searches your computer and discovers that you've uploaded illegal content in encrypted form to S3/GCS/Mega (especially Mega because they make it so easy to upload client-side encrypted data), then law enforcement will order the provider to destroy all copies of the data.

It will be interesting to see what happens if the provider is a Sia host. Law enforcement entities have standard processes for reporting illegal content to Amazon/Google/Mega/etc, and those companies have teams responsible for handling those requests. Casual Sia hosts currently wouldn't know how to handle such a request. The outcome might be that the compliance costs are too big for casual home users, so hosting on Sia becomes a specialized task that only dedicated companies can provide.

[1] https://lukechampine.com/irresponsible.html

uwuhn• 7 years ago

It doesn't. The last real application of Sia that I heard of during the previous crypto boom basically sounded like renting your HD space out to people who would potentially use it to store illegal content.

arcticbull3• 7 years ago

That's the biggest risk of permissionless / decentralized -- you learn quickly why permissioning and centralization are important.

antpls2• 7 years ago

The next step is to fight the idea of "illegal content". Bits are just bits, some just upset the elites, just like nipples.

galaxyLogic2• 7 years ago

I find the concept of "Illegal Content" conceptually questionable. Authorities do not care much, I assume, if you possess illegal content. What they care about is if you share illegal content.

Say you have a pro-democracy pamphlet on your hard-drive in China. If they find it they would imprison you no doubt but they don't care so much to look for it, and it is hard to find if you never share it with anybody.

SIA provides private storage so if your only crime is possessing illegal content but never sharing it with anybody no big deal (I would assume). But if you share it then SIA can not protect you because the exchange must happen somewhere outside of SIA.

dymk4• 7 years ago

Authorities do not care much, I assume, if you possess illegal content. What they care about is if you share illegal content.

You'd be wrong - people get nabbed for having or creating child porn, not just distributing it.

camjohnson265• 7 years ago

How does http handle illegal content? Is there any reason that content should be blocked at a protocol level instead of punishing the person who created it through existing regulations? If Sia has a way to remove such content it loses its decentralized nature, which is the whole reason it exists in the first place.

Should it exist? Well sometimes important content is illegal, such as political content in China. It’s good to have a way to publish important illegal content, even if bad stuff slips through.

What if someone posts illegal violent or sexual content? That is unfortunate, but it means that a real person took the time to post such content on the network. Keeping them from speaking will not change their warped beliefs, it will just hide them from the world and keep people from remembering that evil people exist. The person who published the content should be punished through existing regulations and the content should stay on the network to remind us that some people are horrible. Hopefully no one accesses it.

Fornax96• 7 years ago

Perfectly said. Removing "illegal" content does not solve the fact that the content exists. The problem should be tackled at the source.

anonymous51334• 7 years ago

All data is encrypted automatically before it is uploaded to the network. It is impossible for individual hosting nodes to see what is being stored.

theonlyklas• 7 years ago

Although the cost of cloud backups are egregious compared to the cost of raw storage, I'll pay a premium for the peace of mind that comes from letting someone else be held liable for storing my data and credentials to access it. I can easily upload/download files to google, iCloud, or Dropbox from almost any device knowing only my email and password, which I find preferable to having to remember an arbitrary 29 word seed. With the amount of exit scams in cryptocurrency, too, I just don't trust any project to continue to provide the same amount of utility that they do now, if they provide any at all.

I suppose you could use a custodian site to link an email and password to the seed, but then you enter a centralized third party to the mix.

I see value in Sia but it's just not for the average person in its current state.

Ajedi3211• 7 years ago

You could store your Sia seed for free on Google Drive. That way you get the benefit of Google being "liable" for your data and credentials, but with the much lower storage costs of Sia. Yes, it's a centralized third party, but you already implied you have no problem with that.

Scaevolus• 7 years ago

Sia's seeds are ridiculous-- the 29 words provide 300 bits of entropy. 100 bits would be a sufficient security margin against brute forcing, assuming a modern memory-hard KDF like Argon2.

With a 100 bit password, assuming every flop of the 1.8 exaflops of the Top500 supercomputers tested a new password, it would still take 25,000 years to crack. Key stretching should add at least 30 bits of security by taking a billion operations--

    $ perf stat argon2 asdfasdf -id -m 16 -t 16 <<< asdf
    Type:  Argon2id
    Iterations: 16 
    Memory:  65536 KiB
    0.781 seconds

         6,331,020,712      cycles                    #    4.021 GHz                    
        13,467,211,117      instructions              #    2.13  insn per cycle

Here's what 100 bits of security margin looks like with a more sophisticated scheme (abbrase): "Hope raised between unpleasant bellows. Devil rode sullenly, refugees waiting." => (first three letters) hopraibetunpbeldevrodsulrefwai.

perlgeek• 7 years ago

and their prices are suspiciously similar at about $20 per TB per month. Today it is possible to purchase a refurbished 1 TB hard drive for about $20 — so buyers are essentially paying Amazon each month the entire cost of the hard drives used to store their data!

Really? That's the argument? That you could store your files cheaper on a single, refurbished disk without considering the cost of housing, electricity, network, physical security, redundancy, management etc.?

Ajedi3211• 7 years ago

The implication is that with Sia you can get those same benefits without having to essentially pay the entire cost of a new hard drive every month. (Sia's rates are much cheaper.)

lunchables• 7 years ago

Aren't all those costs priced in? I assumed the data storage with Sia was redundant and encrypted, which should cover basically all of those things, right?

Looks like it from looking at the site:

File segments are created using a technology called Reed-Solomon erasure coding, commonly used in CDs and DVDs. Erasure coding allows Sia to divide files in a redundant manner, where any 10 of 30 segments can fully recover a user's files.

[...]

Before leaving a renter's computer, each file segment is encrypted. This ensures that hosts only store encrypted segments of user data.

I'd love for someone to do an analysis on the performance and availability of the data. I wonder if you could pay more for faster speeds or higher levels of redundancy? That would be neat features.

rossjudson• 7 years ago

I think the pricing information should probably be updated. The current public page for Google One says $99/year for 2 TB.

b_tterc_p5• 7 years ago

How is this possible? By removing intermediaries
[Sia] makes a small transaction fee from each file contract.

Am I wrong in observing that the number of intermediaries between hosting data on AWS == the number of intermediaries oh Sia == 1?

What exactly is this doing? Blockchain contracts to rent out your hard drives?

Ajedi3211• 7 years ago

They explain what the mean by that two sentences later:

Hosts on Sia do not have to worry about building and maintaining enormous datacenters, employing thousands of employees, and marketing their services. They only need to worry about providing reliable storage capacity to renters on Sia.

And yes, it's essentially a decentralized storage market built on smart contracts. You can either pay to store your files (the client software handles redundancy, contracts, etc automatically) or _get_ paid to rent out space on your hard drive.

b_tterc_p5• 7 years ago

How do they verify that a host can actually store all of the data?

jason50• 7 years ago

I think they are saying they are removing "trusted" intermediaries. There are zero trusted intermediaries sitting between buyer and storage/bandwidth in Sia's model. Thus, value cant be extracted from providing trust/confidence.

snarf212• 7 years ago

I agree that the use cases don't seem to be lining up. So is this for personal use or corporate data center replacement? Consumer companies will give you 2TB for $10 with no bandwidth costs with guaranteed replication and sharding built in. What happens when one of these nodes goes down? I didn't see anything in the post about that. I think the thing that always gets lost in these ideas is that there is a reason that people pay these current prices. It is because is solves a real problem and high reliability requires paying people to make sure that is true. A smart contract and collecting micro payments on the hope a cryptocurrency will rise in price have much different skin in the game.

I've always thought the most important use case around decentralized storage was more around censorship. You can post content that is harder to block in countries that firewall AWS, etc.

_7fvc• 7 years ago

This is a flashback for me to 2006. I was working on storage technology and reviewing a few products in p2p storage. There's too much burden to apply p2p to get space. Storage is cheap enough for this idea to work.

SimianLogic22• 7 years ago

Can someone ELI5 why you need a blockchain for this? Another startup could make a non-blockchain version of this that uses similar syncing technology to GDrive / Dropbox file sync that just syncs other people's stuff instead of yours and pays you monthly.

hanniabu2• 7 years ago

Non-reliant on central services or entities.

snarf212• 7 years ago

This also cuts both ways. Your business needs a file but your customer can't connect to that node because of some political blocking of internet connections. Who can you rely on? What central service can you call to get this fixed asap?

idlewords2• 7 years ago

Except for the software developers.

wmf5• 7 years ago

There are a lot of details such as finding servers with available space, paying for storage, posting bonds, collecting bonds with fraud proofs, and doing all of that in a decentralized way. You pretty much need a blockchain for the decentralization.

idlewords2• 7 years ago

You need the blockchain to secure venture capital investment and a vocal online community.

Taek14• 7 years ago

Founder here. The blockchain gets you a couple of things.

The key thing in our blockchain is the storage contract, which is something like a blockchain-SLA. The host has an obligation to store some data, and the host also puts up money out of pocket as a promise that the data will be stored faithfully. The blockchain will occasionally challenge the host, and if the host cannot create a proof that the data is still being stored, the host's collateral is forfeit.

The renter also puts money into the file contract, this money is used to pay the host if the host is honest/keeps the data, and the money is destroyed if the host is dishonest or loses the data. The money gets destroyed so that the renter has no reason to interfere with the host, nothing to gain from seeing the host fail or working to make the host fail.

Thanks to the blockchain, the host has a guaranteed payment if they store the data, regardless of whether the renter sticks around for the duration of the contract or not. And also thanks to the blockchain, the renter knows that the host will not get paid until the end of the contract, and that they will actually lose some of their own money if they lose the data.

There's another great benefit though to cryptocurrency, which is that payments are super low friction. Every time you upload data to the Sia network or download data from the Sia network, you make hundreds to thousands of tiny payments to the various parties over state channels. This wouldn't be feasible using more traditional payment systems, we'd have to track everything in a centralized database, and that actually doesn't scale very well.

Sia is both low cost and high speed, but these are side benefits over the primary goal of uncompromising decentralization. Storage just happens to be an application where the decentralized version is actually highly competitive to the centralized versions.

I'm glossing over a fair amount here, if there's anything that doesn't make sense or seems off feel free to ask, I'm happy to clarify.

SimianLogic22• 7 years ago

How do renewals work? With S3 I upload something and then pay for it + bandwidth until I delete it. Would I need to know how long I need storage for ahead of time? Estimated bandwidth?

I run a video rendering service that currently uploads previews (call it 5-10mb) to S3 for ~3 days and then deletes them. Finished renders (~50-200mb) are stored for 2 weeks. I upload 500-1000 new files a day.

Amazon just bills me for this once a month. Would I need to load up a wallet and prepay each of these contracts individually? How would I send out a download link or play a video in a browser?

rolltiide4• 7 years ago

that just syncs other people's stuff instead of yours and pays you monthly.

with VC money? with a fickle settlement system that requires 5 layers of financial institutions before you get a REST API? where you probably still need 50+ money services licenses and compliance department anyway? where any of those financial institutions can still cut you off for any reason? to maintain operations in a single country?

why do all that when you can just print money, or more accurately create a scarce digital asset of value to the market you create, available internationally on day one

you don't have to agree with it, but the pendulum has swung waaay in this other direction and thats the answer for "why" before you even get into the implementation that the founder wrote.

rsync• 7 years ago

"The cloud storage industry is very similar to the financial industry in that trust is a necessity ..."

https://www.stavros.io/posts/holy-grail-backups/

"I Found the Holy Grail of Backups"

nullandvoid• 7 years ago

This takes me back I still have a few of there coins sitting around from years ago that have done nothing but tanked ( this was somewhat expected but I wanted to diversify in alts and this looked like one of the cooler options )

buildbuildbuild• 7 years ago

Abuse concerns and “does it need a blockchain” cynicism aside I find all of these “crypto” cloud computing projects fascinating. I highly recommend playing around on their testnets. Storj and Filecoin are also competing in the storage space, and BitTorrent is vying to be a CDN alternative.

SONM aims to be an EC2 competitor, and interestingly requires ID verification to participate in their testnet due to the clear abuse potential of unattributable internet-connected VMs. (which exist in plenty of other places anyways since at present date, hosting providers have no KYC requirements)

Will the sharing economy extend to computing resources? I’ll stay tuned.

Taek14• 7 years ago

Decentralized computing is a challenge because unlike storage, it's very expensive to encrypt computation. You see something like 10,000x slowdowns.

With storage, we can encrypt and checksum everything client-side in a cryptographically secure manner, and we can have full faith that even though we are giving our data to someone else, they have no way to view the data or tamper with the data.

If techniques like homomorphic encryption gain substantial speedups, I'm sure that we'll start to see decentralized/trustless computing, but for now you really need to be doing your outsourced computing with someone you can trust and someone who is regulated, because it's possible to act maliciously.

wmf5• 7 years ago

Unless SGX works.

fghtr• 7 years ago

https://golem.network is the project to make decentralized computing possible. They already have beta version released.

pmlnr3• 7 years ago

Storj was a brilliant idea then suddenly someone put blockchain into it.

anonymous51334• 7 years ago

You can use storj to host your files without using blockchain. I think they accept credit cards directly. Also storj is doing a lot to make sure only reliable hosts are allowed to continue to operate on the network. If you aren't serious about having a reliable hosting node then you'll eventually get booted from the network.

PlaneSploit• 7 years ago

Blockchain is just for payments, the actual object store doesn't use it

miguelmota• 7 years ago

Decentralized computing requires redundant computations for verification which seems like a waste for most applications and not to mention latency added to each request.

I'm sure there's use cases for this that require more sophisticated compute than what smart contract platforms like Ethereum offer but I'm not sure what those are. If someone is being prohibited from using a cloud platform and needs to resort to a decentralized hosting platform, I can see them simply using a VPN and hosting the application on a different platform that's not censoring them. SONM requiring KYC for their decentralized platform sounds like an oxymoron.

jonas21• 7 years ago

Hosts on Sia do not have to worry about building and maintaining enormous datacenters, employing thousands of employees, and marketing their services. They only need to worry about providing reliable storage capacity to renters on Sia.

I would expect that building large datacenters and employing people to manage them is exactly how you go about providing reliable storage capacity.

And while it's true that providers don't have to worry about marketing or branding, well... the flip side is that they can't do any marketing or branding and need to compete in a market where the product is completely commoditized.

Nursie• 7 years ago

I would expect that building large datacenters and employing people to manage them is exactly how you go about providing reliable storage capacity.

Further, I imagine that if there is money to be made providing reliable storage to Sia users, then those with experience of reliably providing low-cost, high-volume storage will corner the market pretty fast!

Taek14• 7 years ago

Sia is built on a completely different assumption than most large datacenters - uptime requirements for hosts are around 95%, which equates to about 36 hours of downtime per month.

This means that you can run a much leaner database and require substantially less expertise to keep things going. If a rack goes down, you don't need someone on-site to bring it up ASAP, you can fix it the next workday without upsetting your customers.

95% uptime for hosts translates to 99.99+% uptime for Sia users, because users store data across hosts in a 10-of-30 scheme. As long as 10 out of 30 of the hosts are online, you will be able to access your data. The probability of losing 20 hosts when each of your 30 hosts have 95% uptime is exceedingly small; your practical uptime depends more on the reliability of the software than it does the reliability of the hosts you use.

As for commoditization: that's the goal! If we can completely commoditize data storage, prices should come down dramatically, and the market should be a lot more efficient. You won't need to be an Amazon or Google to have a stable and competitive data offering.

q3k• 7 years ago

This means that you can run a much leaner database and require substantially less expertise to keep things going. If a rack goes down, you don't need someone on-site to bring it up ASAP, you can fix it the next workday without upsetting your customers.

This is already the case for cluster storage systems like Ceph.

abalone• 7 years ago

  Don't give up, I won't give up
  Don't give up, no no no
  Don't give up, I won't give up
  Don't give up, no no no

- Blockchain guys

(I know I know.. I couldn’t resist.)

buildzr8• 7 years ago

I'm trying to understand the reliability story better. I'm currently looking at this page:

http://siastats.info/contracts_status

Does that mean for every individual hoster-renter contract, of ones that were established and used, some ~9% of them failed?

That would essentially mean that to get equivalent numbers to those advertised by B2, which claims 11 9s I'd need to buy over 10 contracts. Is this actually how bad it is at the moment?

Taek14• 7 years ago

You are correct that the failure rate for contracts is somewhere around 9%, and you are also correct that you need over 10 contracts to get 11 9's of redundancy, however this doesn't translate to needing 10x redundancy on the Sia network to get high reliability.

Data is uploaded to Sia (per-default, it's configurable through the API) in a 10-of-30 scheme using Reed-Solomon coding, which means that each piece of data is held by 30 hosts, and out of those 30 hosts any 10 of them are sufficient to recover the original data. This has a total overhead of 3x, and the algorithms behind it are in my opinion super fascinating.

If you assume each host independently has 91% uptime, you get this amount of downtime as a result: https://www.wolframalpha.com/input/?i=sum+of+(30+choose+x)+*...

In reality, the software reliability is a bigger factor in downtime than the host's reliability.

buildzr8• 7 years ago

Data is uploaded to Sia (per-default, it's configurable through the API) in a 10-of-30 scheme using Reed-Solomon coding, which means that each piece of data is held by 30 hosts, and out of those 30 hosts any 10 of them are sufficient to recover the original data.

This sounds good - I'm just trying to understand how this is counted - are you establishing 30 separate contracts to achieve this?

pmlnr3• 7 years ago

These ideas are all the wrong direction to decentralize. We don't need more layers and more complexity and more blockchains, we need simple, reliable, accessible household appliances. For storage, for sharing, for posting.

Please stop adding layers and layers of calculations, it's a waste of energy.

gnode5• 7 years ago

Please stop adding layers and layers of calculations, it's a waste of energy.

The trend in electronics is that this has become less true over time. As transistors have shrunk, the relative cost of calculation has diminished compared to the cost of transmission over distances. A cryptographic operation can be cheap compared to accessing main memory because the distances signals travel within the crypto logic block are tiny.

Complexity (as in more to understand) does not equate sensibly to higher power consumption.

anonymous51334• 7 years ago

From what I understand, right now these projects are working on the backbone of the systems. In the future, other developers will be able to create services on top of these platforms to resell the storage space. So maybe someone will have an online cloud hosting service with a simple drag and drop online interface. They will charge a fee to the customer and the website operator will simply buy storage contracts on SIA or similar providers to provide the actual hosting and distribution of the files.

filebase3• 7 years ago

You would be correct!

Filebase[1] provides S3-compatible object storage at a fraction of the price. We happily use Sia on our backend. :)

[1] https://filebase.com

ukd13• 7 years ago

Long time user and follower of Sia. It's always surprising to me that it's overlooked compared to filecoin (which is similar, but seems like $250m+ vaporware to date). I use it to have tertiary backups.

Weird caveat I found: the smallest chargeable size is actually pretty large.

xur17• 7 years ago

Weird caveat I found: the smallest chargeable size is actually pretty large.

This and the cost to form contracts is is large enough that it doesn't make economical sense until you are storing > $1gb.

That said, they seem to slowly be working through a number of these concerns. They very recently released seed based backups, one of the biggest issues I had last time I looked.

ukd13• 7 years ago

Ya; that's another one - the same wallet on multiple machines = different set of contracts. Didn't notice that was fixed though...

neiman• 7 years ago

I tried Sia about 2 years ago. It worked "ok" for half a GB files, However, for 4-5 GB my upload got stuck while the help forum couldn't provide answers besides "wait for the next release".

There were also unclear issues back then. Are the files encrypted on my computer before being uploaded? are they sharded between many hosts? or are they stored in one host? How long will they be stored.

Most of these issues seem like they could be improved. Not an inherent faults of the technology or something. I like Sia's idea and tech in general; I'll give it a try again now.

Taek14• 7 years ago

Sia has come a long way in the past two years. Uploads, downloads, and repairs are both much more stable and also a lot faster. Sia can effortlessly sustain 100mbps upload and download if your connection is that fast.

Sia is happy with files that are hundreds of GBs in size today, and with a filesystem that that gets up to about 20 TB. Beyond that it starts to struggle, but we're continuously working to expand its capabilities.

Data has always been encrypted client-side, even in very early releases. Data today is sharded between hosts in a 10-of-30 Reed-Solomon scheme.

I think you will find that the experience is much smoother and more complete compared to the experience from 2 years ago. And, a critical feature is now available as of the latest release: seed based file recovery. After you upload your files, it's possible to create a snapshot of your data that you can recover later on a different machine using nothing more than your wallet seed. This makes Sia a practical and cost effective solution for secondary or tertiary backup.

nisten2• 7 years ago

There is one missing piece to all this, I/O.

Given they publish their apps on all platforms they could probably add this in the future as an ssd or PCIe storage tier by doing regular disk benchmarks.

However it seems like a major limiting factor, given that random access speeds on spinning drives or even cheap ssds can drop drastically by an order or two of magnitude (i.e. from 130MB/s to 2MB/s). Given how they're doing error correction in a distributed manner, having a fast storage tier would be a major advantage for them which would eliminate any problems from flash storage wearing out and fully gain the benefit of throughput, random access times and cheap new prices of nvme drives.

At the moment I probably can't use this for streaming my own 4k videos to myself because an up to 300mbps network is not good enough for that, the slowest storage in it would be a bottleneck for a significant amounts of the experience of watching the streamed file.

And I definitely can't use it for application storage given how important response time is to the user experience and user growth.

That all aside, those 2 are relatively extreme examples and there's a lot of usage scenarios in between them.

I do admire how far they've gotten this given all the blockchain hyped products that have failed. They seem to have a solid system here that will only improve on performance and put pressure on bringing online storage prices down.

Taek14• 7 years ago

Most 4K video is closer to 50mbps. If your home connection is that fast, Sia should be more than capable of delivering smooth 4K streaming to your home.

A single host with slow storage is not a blocking factor because Sia downloads in parallel from as many as 30 hosts at once when composing a stream of data. Even if each has an I/O bottleneck of 1 MB/s (8mbps), you'll be able to get your video out at 240mbps.

In practice today, Sia hosts are almost all network bottlenecked, most use HDDs which do not have trouble keeping up with network latencies (HDD seek times are 11ms, network latencies usually 50ms+) or throughputs (HDDs can usually get close to 100 MB/s, far more than the network throughput)

Can I ask, have you tried streaming 4K video from the latest version of Sia? Seek times might be 5-10 seconds, but I believe that once your stream has loaded you will see completely smooth playback. Our next release has a focus on reducing those seek times, but the code is not ready yet.

nisten2• 7 years ago

That's interesting, and very informative thanks.

No I have not tried it however, based on your information, this looks ideal for something like showing auto-playing video on a splash screen using your API to pull the file on the client side.

  /renter/stream/*siapath [GET]

jchw4• 7 years ago

Today it is possible to purchase a refurbished 1 TB hard drive for about $20 — so buyers are essentially paying Amazon each month the entire cost of the hard drives used to store their data!

What a refurbished hard disk does not give you:

- Redundant storage

- Redundant power/network/etc.

- Geographic distribution

- A network API

- Any networking, or uplinks at all

- Access control

- A team ensuring the reliability and security of your data

- Global network of datacenters and pops

- Integration with other cloud services

All that and Amazon et al. do not store your 1TB of data in a single refurbished hard disk. This is beyond non-sense.

Sure, some of these costs are amortized, but I still think this is a lame comparison. If all you wanted was at-cost block storage with crappy reliability and speed, you could probably accomplish that in easier ways.

Taek14• 7 years ago

One of the main innovations of Sia is providing a low cost means to turn that 1 TB refurbished hard drive into a useful cloud product.

Sia is able to provide, at 1/10th the cost of Amazon, high throughput, high reliability, high redundancy, geographically distributed cloud storage, and it can do that without high reliability drives or datacenters.

leetbulb3• 7 years ago

Off topic, but: I purchased three refurb 6TB HGST He HDDs in Jan. from a reputable shop. All three failed last month. I was so happy that the shop completely refunded all of them because the warranty is generally very limited on refurb HDDs. Normal RMA is 5yr on this specific model.

There's no way refurb drives are cost effective for these environments.

AdamJacobMuller• 7 years ago

On balance, I purchased 45 refurb 4TB SAS drives around the same time. I had two DOA with 0% failure rate since then.

Also, very anecdotal, I know.

bin02• 7 years ago

Backblaze's failure/longevity stats are indispensable when considering which drive to purchase: https://www.backblaze.com/blog/hard-drive-stats-for-2018/

gnode5• 7 years ago

What a refurbished hard disk does not give you: [...]

The whole point is that a decentralized storage network can add such features on top of the 1TB refurbished hard drive.

The refurbished drive may not be reliable, but if you replicate your data amongst many separate drives with independent failure (geographically distributed; different providing entities) then you can create a system with greater reliability. By using erasure coding schemes (similar to RAID5/6), it's possible to reduce the storage overhead which supports reliability to a fraction of the principle amount. Thus, providing 1 TB of storage to the network could earn slightly less than 1 TB of reliable storage in return.

Comparatively, a single cloud provider has the disadvantage that, because they are a single legal entity, they can decide for whatever reason to not continue your service.

jchw4• 7 years ago

What I am not saying is that Sia is not viable or that cloud storage providers have no downsides, which they clearly do. I did not mean to imply that there is anything wrong with this service, just to express my annoyance with the comparison. I am not happy that my comment got so much attention, because I had no intention to overshadow the announcement with my single gripe.

The line that is annoying me and probably others is this:

Buyers are essentially paying Amazon each month the entire cost of the hard drives used to store their data

...They're not. Because redundancy. Because electricity, CPUs, network, software development, etc.

zherbert4• 7 years ago

Appreciate your response. My comparison to the cost of a hard drive today is not meant to be a direct comparison, but instead to demonstrate that traditional cloud storage is actually quite expensive. Sia is able to bring down storage costs by an order of magnitude and bandwidth costs by two orders of magnitude. While most HN readers understand the nuances of cloud storage costs, most non-technical readers are unfamiliar with the high costs of S3 storage and egress.

jchw4• 7 years ago

The issue I take is that the comparison is truly not meaningful. Yes, storage is cheap, but you are comparing the cost of a refurbished disk to a managed service.

This may lead people to think cloud storage is a waste of money, which I personally think is untrue (though Sia certainly may be able to do it for cheaper than cloud storage, too. Both things can be true.) While the cost to us is certainly more than the cost to the provider, the cost for us to replicate what a cloud provider offers is certainly greater at small scales where we benefit less from amortization and volume pricing.

A good point may be that we don’t actually need to replicate what a cloud provider does to offer a compelling alternative for various use cases. This I would probably agree with. It is certainly cheaper for me to have my own NAS and offsite backups than it would be to pay for cloud storage, though it is still surprisingly expensive (at least to the uninitialized. Check out what a fully loaded Synology 8 bay would cost! Multiply by number of backup sites. A lot of storage, but definitely not cheap.)

Dylan16807• 7 years ago

Obviously you're paying for more than the drive. That doesn't make it a bad comparison. There's no great reason for the drive to be more than half the cost, so even with 3 copies amazon still comes out as making a whole lot of profit. There's a reason that multiple competitors can provide all those things at significantly lower prices.

And more importantly, if you're focused on storage then don't pay for features you don't need.

weq• 7 years ago

I think the use-case of cloud storage is mostly marketing by cloud provides to show relevance to the consumer. They are are loss leaders at best; used to lock you in to their services / track your every move; at worst.

I think the comparison with a $20 1TB hdd is perfect. Cloud providers, and phone makers, have convinced you to have your photos, music, data, online, always. Like a bank. Among other things, its much easier to mine your usage patterns that way and enforce DRM of other cloud services.

I have my IDE PC drive from 20 years ago. It still works. I can burn a CD if im that worried about failure. I dont need the redundancy of a billion user website, a networking api, access control, all that other crap you use in enterprises that comply with ISO standards.

Before cloud providers, people didnt take their photos to Apple and tell them to store them. The only reason u do it now, is because of marketing and service integration.

jtbayly• 7 years ago

Wrong. The reason we do now is so that we can have our photo library accessible on our phones and synced between multiple devices.

siavosh• 7 years ago

Can someone explain the difference between Sia vs Filecoin/IPFS. The latter seems to be still working on solving some fundamental crypto primitives before they can offer what seems like what Sia already claims to have solved?

wmf5• 7 years ago

Yes, Filecoin is trying to come up with a "more efficient" and possibly "elegant" implementation based on algorithms that are still being invented. Of course, with 100x the budget it's possible that they could still overtake Sia.

tempsolution• 7 years ago

This reads like a typical blockchain non-sense hype article. Before you even go down the road of comparing sia to S3, what about some unbiased benchmarks? Show us the 99.999999999% durability, the pretty much unlimited bandwidth, the single digit milli-second latencies, the practically unlimited capacity.

If you want to compare Sia to anything, compare it to Dropbox or OneDrive. S3 was made for an entirely different purpose and Sia can't even remotely compete with it.

As a personal data store? Maybe. But wait until this thing becomes more popular and hackers start making a mess. Will be interesting to see how stable this "fully decentralized" network is, once it gets the full attention from bad actors as all the cloud providers.

rolltiide4• 7 years ago

coming from someone who is not a fan of Siacoin from when I last used it in 2017, I think you miss the point

what I like and want is encrypted distributed data that I can store and retrieve quickly and cheaply

I also like an economic model that aligns the hosts to maintain the data, and how the proliferation of that economic model lets me invest in its growth from merely being a passive non-productive speculator, and finally I would like a clear easy to understand ROI from being a productive host: can I subsidize the cost of my hardware that I'm already interested in having.

any cryptocurrency can do that, but how the software uses the cryptocurrency is a key factor, and unfortunately we can like an idea and team but get forced to accept an economic model that has flaws

so for me, it is easy for me to understand why teams want to promise the sky against S3, and it really isn't that relevant to using this securely. like, you might be able to subsidize all of your hardware and provide a service without worrying about if the SIA comparison is accurate because its really not relevant.

but finally, it turns out that unmonetized and non-cryptocurrency based decentralized cloud storage networks are good enough! so I stick with IPFS.

hope that was useful to someone.

carlosdp4• 7 years ago

IPFS isn't decentralized cloud storage though. Unless you host your file yourself on your IPFS node, your file is not guaranteed to be available. Hence why Filecoin is a thing they are working on.

carlosdp4• 7 years ago

I agree in that they are probably comparing the wrong aspects of S3.

You're not going to be able to compete on the speed anytime soon probably, and maybe even the capacity if it really takes off. But, anyone here that works on an Amazon-based internet company knows that S3 is the most unreliable (in terms of availability, not storage) universally used product they have, next to EC2.

A good chunk of incidents can be attributed to S3 downtime in some AZ. With a decentralized storage network, the probability of actual downtime in terms of availability go wayyyyy down. If Amazon us-east-1 has a bad deploy or a network event, you lose access to those buckets, full-stop.

With something like this, you'd need a much larger network event, and if the app is fully decentralized itself, blast radius can be contained to individual customer regions instead of where you host your servers.

Probably more useful to compare to GCS Storage honestly, since they have multi-region buckets.

Taek14• 7 years ago

You're not going to be able to compete on the speed anytime soon probably, and maybe even the capacity if it really takes off

Sia today running in the default configuration gets as much as 300mpbs upload and download, which I believe is comparable to S3. Latencies are several seconds, which is not comparable to S3 however this is a matter of software optimization, not a fundamental limitation of the network architecture.

simonebrunozzi2• 7 years ago

You are being deceived by Amazon's PR machine at its finest. The 11 9s of durability are a "design", not a guarantee. The official PR line at the time was "S3 is designed to provide 11 9s of durability".

There's still no SLA on durability AFAIK.

easymodex• 7 years ago

Actually it is 11 9s of durability, but availability is a different story.

zherbert4• 7 years ago

Hi all, author here, just wanted to say I am thrilled to see the HN community discuss decentralized cloud storage in more detail. There are many drawbacks today and narrow use-cases, but our goal is to continue to improve Sia – and over the coming years we will prove that decentralized storage can compete directly with Amazon, Google, and Microsoft. We feel confident that the marketplace dynamics, in particular, will foster competition between hosts on the network and maintain prices that are an order of magnitude cheaper for storage and two orders of magnitude cheaper for bandwidth.

zherbert4• 7 years ago

I would also add – give Sia a try, especially if you have tried it before our v1.4.0/v1.4.1 releases! https://sia.tech/get-started

solidasparagus2• 7 years ago

Pretty arrogant to say you're going to make cloud storage providers obsolete when you haven't even built a competing solution. You can't even share data yet...

And what about all the other features that cloud providers offer? You say "[the] Sia software is exponentially improving, and performance and featureset is quickly approaching Amazon S3", but I see absolutely no evidence of even the most basic features of cloud storage providers.

Authorization? Regulatory compliance? High-performance bandwidth? High-reliability public API/URLs? Customer support? Private cloud peering? Zero effort integration with many (most?) major data-related open source projects? Versioning? One-click, no-knowledge reliability?

How much do you need to grow to hit cloud scale? 1,000,000x? More? And you're already claiming that you're going to make S3 obsolete?

Your blog would hold more weight if you weren't so over-the-top, bullshit-level optimistic.

peterkelly• 7 years ago

https://news.ycombinator.com/item?id=9224

AmericanChopper3• 7 years ago

I went and tested this after reading the article

https://filebase.com/

Only took me a couple of minutes to get a file uploaded using the aws-cli. It’s the only decentralized object storage solution I’ve ever seen that seems to actually work.

fitzroy• 7 years ago

Gotta say, the fact that the installation instructions consist primarily of getting around the "unknown / unidentified developer" restriction doesn't inspire confidence.

ken666• 7 years ago

Hi,

Any chances that Sia will eventually support directly restic/Borg?

I did not know about filebase (and looks interesting), but honestly it is barely competitive with backblaze/wasabi for backups.

filebase3• 7 years ago

Thanks for checking out our service! Filebase[1] offers an S3-compatible API and we are happy to confirm that we have tested basic use cases with restic.

Using Sia on our backend enables us to offer cloud storage at very competitive rates. Currently, Backblaze charges egress fees (we don't) and Wasabi has a 90-day minimum charge for all objects. (we have no minimums)

[1] https://filebase.com

alexnewman3• 7 years ago

I was greatly skeptical of these types of systems but ipfs proved me wrong. It's fine, when combined with a CDN. I'm interested in what new use cases Sia handles when CDN + IPFS works so well. For instance my blog https://www.wuli.nu (source code https://github.com/posix4e/blog)

psKama• 7 years ago

Can you please clarify which CDN features are you referring to on your blog (can't find any) or anywhere else? Also can you elaborate what do you even mean by "so well"? Sia was started to be built before IPFS was even announced and has hosted PBs of data already as a working product. On the other hand IPFS hasn't even released it's public testnet so it would be great to understand your point. Edit: btw by IPFS I assume you refer to their Sia equivalent product, Filecoin. At least I do, as IPFS cannot be compared to Sia alone.

alexnewman3• 7 years ago

I am in no way referring to filecoin. I am talking about IPFS and cloudflare integration. https://blog.cloudflare.com/distributed-web-gateway/

alexnewman3• 7 years ago

I actually think filecoin is worthless and IPFS is the only valuable thing that came out of that group

truth_seeker• 7 years ago

Can somebody from Sia team compare it to Radix DLT and it's mechanism to shard and distribute data ?

https://www.radixdlt.com/

x3haloed• 7 years ago

Is forums.sia.tech down temporarily or permanently?

2youmich• 7 years ago

Sia is a future

gridlockd2• 7 years ago

I have a feeling this is an apples to oranges comparison.

Can you serve clients directly over HTTP using Sia in a permissioned manner? If not, then it's not usable like S3. If it's not usable like S3, then S3 pricing comparison is meaningless.

Also, how correlated is Sia pricing to the overall cryptocurrency market? Will my costs go through the roof when there's another crypto bubble?

Would it actually scale? If Sia became popular, would prices go up like Bitcoin transaction prices? Isn't the extremely cheap bandwidth due to spare capacity of average broadband plans? If so, wouldn't providers eventually cut that off? Broadband pricing relies on the fact that most users don't use all of the bandwidth most of the time.

hereme888• 7 years ago

Just switched to backing my data on Sia. Was previously using Nextcloud, which is great, but more expensive.

Took a little time to set up, but renting 200 GB for 10 cents/mo, in a decentralized system that encrypts my data, has great uptime, and doesn't mine my personal or "aggregated" data, is unbeatable.

I heard Sia is not great at handling large number of files, so I'm using a backup software to create automated, compressed backups first, and Sia handles those larger files.

I'll see how it pans out over time.

Sia – Decentralized cloud storage network

Related Stories