Decentralization is about diversity

https://sandstorm.io/news/2016-08-17-decentralization-is-about-diversity

by ingve • 10 years ago

169 81 10 years ago

wmf2• 10 years ago

To me this is the aspect that separates Sandstorm from the rest: it is not a suite of apps; it is a developer platform. This is less sexy in the short term but it has potential to be more powerful eventually.

mnutt2• 10 years ago

This, to me, is probably the most interesting thing about Sandstorm as it relates to open source web apps. Today, I can write an open source web app and post it to github, but it takes someone with operational knowledge to deploy and run it. (even if that's just "create a heroku account, create a new app, git push" you still need to know something about development) But it feels like open source web app distribution has been stagnant and possibly even a lull since the wordpress/drupal/etc php heyday.

My hope is that Sandstorm rejuvenates open source and indie web app development by providing a channel where end users can easily and safely run random web apps they find.

djsumdog• 10 years ago

I'm in the process of trying to setup some things I'd like to seriously try in Docker containers (GNU Social and iodine for example). Now I want to try to get Sandstorm working in a container now as well. :-P

Honestly though I was talking about this exact concept that Sandstorm has implemented, a few years back. I was telling a buddy, "We have general purpose desktop OSes where we install apps. Why isn't there a building-block type server OS we do the same thing?"

Of course I'm sure like a million other programmers had that same idea, and it's am incredibly difficult one. Sandstorm seems to have a C++ base with Javascript frameworks on top of it. I'll totally put it on my list of things to try out.

The future is distributed.

erichocean• 10 years ago

Why isn't there a building-block type server OS we do the same thing?

Because everything is built on Unix as an OS abstraction. Even Windows (since the early 2000s).

Consider: byte streams and file systems are less than ideal fundamental abstractions in a distributed world.

IMO we won't have what you're seeking until we rethink what an OS should be in an Internet-everywhere world. We're still using OS designs that predate the Internet entirely.

braderhart• 10 years ago

I can help create a very lightweight container image for this if you want?

newsat132• 10 years ago

Since you give the example of GitHub... Sandstorm itself doesn't use their own Gitlab app to host and collaborate. How can they convince others? Doesn't make me confident of their platform. Centralized platforms like GitHub exist for a purpose.

kentonv14• 10 years ago

When the Sandstorm project started, we obviously couldn't host it on Sandstorm. But, we are gradually moving towards more dogfooding. For example:

- We use Etherpad on Sandstorm for all documents we write, e.g. design docs or project plans.

- We use Wekan on Sandstorm for project / task planning.

- We use Rocket.Chat on Sandstorm for internal team chat.

- We commonly share and publish files using Davros.

- We host docs.sandstorm.io -- and the analytics for docs.sandstorm.io -- directly from Sandstorm.

- We host the Sandstorm app index (back-end for apps.sandstorm.io and for the automatic app update pipeline) as a Sandstorm app.

- We host the purchase flow for Sandstorm for Work as a Sandstorm app.

- We sometimes use private Gitlab or Gitweb grains on Sandstorm for non-public code, or for working on security fixes before they are ready to be disclosed.

That said, we still need to:

- Switch the main Sandstorm source repo to Gitlab or similar. This will obviously be a disruptive change, so we haven't tackled it yet.

- Host our e-mail on Sandstorm. Sandstorm's e-mail support admittedly needs a lot of work -- it's a much more complicated problem than most of the other things one does on Sandstorm, and we haven't put much effort into it yet. We'll get there eventually.

- Host our CI server on Sandstorm. This requires packaging Jenkins as a Sandstorm app, and also has a bunch of other complications relating to the fact that some of our tests need to run in VMs or use privileged syscalls, so it might take a while.

- Host our main web site on Sandstorm. Currently it's hosted as a plain-old static file server that we maintain. We could switch but there wouldn't be a ton of benefit in doing so, other than to say that we did.

We'll get to all these eventually.

amelius• 10 years ago

But the browser itself is an application delivery platform, and it includes a sandbox. So I'm curious why one would need something like Sandstorm on top of that to run apps?

kentonv14• 10 years ago

Many (most?) apps need server-side logic. Apps that have multiple permissions levels need to enforce those permissions somewhere. Apps that support real-time collaboration need to maintain the authoritative copy and coordinate the OT stream. Etc.

milansuk2• 10 years ago

I think decentralization is also about Users own/control their data. If someone create for eg. running app and I'm not able to easy move my old runs to new app then diversity is nothing.

"Software must be provided as a package – not as a service – with each user running their own private copy." This sounds like a going back to desktop apps. Or am I missing something?

losvedir• 10 years ago

"Software must be provided as a package – not as a service – with each user running their own private copy." This sounds like a going back to desktop apps. Or am I missing something?

Yep, you're missing something: sandstorm is trying to be something like an open source app store for servers.

I believe sandstorm can be run on anyone's server (or VPS). Then, you can manage through its web interface (one-click install) various sandstorm-compatible apps onto that server.

Since you still control the server, it's your data, but because it's a server it can be accessed from your desktop computer, or phone, or whatever. I think there's an easy backup / transfer solution if, say, you want to move your server from Linode to DigitalOcean or to your own hardware.

edit: to clarify how this relates to the post:

I'm a web developer.

In my free time I'd love to throw together a little free budgeting web app, but I have a distribution problem: Either I centralize it, in which case I have to deal with data security, authentication, and scaling, or I release it as a rails app and ask my users to go through that rigamarole of installing and running their own rails server.

Instead, with sandstorm, I package up my rails app as a sandstorm app and put it in their app store, and anyone who wants a budgeting app can easily install and run it on their own private server.

milansuk2• 10 years ago

Can I ask what will be your business model for your app? What options do you have in Sandstorm enviroment?

scotu2• 10 years ago

Either I centralize it, in which case I have to deal with data security [...] Instead, with sandstorm, I package up my rails app as a sandstorm app and put it in their app store, and anyone who wants a budgeting app can easily install and run it on their own private server.

you just made a case for NOT using a self hosted opensource app

EDIT: extended the quoted text for better context

fabrice_d• 10 years ago

Before the web, software publishing was inconvenient, both for users and developers: copying listings from magazines, mailing floppy discs & CDs, finding that there is a bad sector on disk #52 of your Linux distribution...

But once you had the software, you could use it on your local data, fully under your control.

The Web changed that landscape dramatically: it is arguably the platform with the lowest barrier to entry for all parties. Unfortunately this has been abused by huge siloed services. They are attractive because they are free, and ... actually useful, but they make you lose control over your data, how you can process and manage it.

Sandstorm aims at fixing that imbalance, putting back the user in control, as do a few other "easy self hosting" systems.

kentonv14• 10 years ago

(Article author here.)

I think decentralization is also about Users own/control their data.

Of course. I'm arguing, though, that while user control of data is important, software and developer diversity is more important. I realize this is likely to be a controversial claim. :)

[If] I'm not able to easy move my old runs to new app then diversity is nothing.

Depends. The fact that I have many (old) documents in Google Docs doesn't prevent me from using Etherpad today, even though there's currently no automated migration. Also, quite a few services do actually provide data export and API access that could be used for export, which all suggests that data mobility is not the real problem here.

Don't get me wrong: I obviously think data mobility is important. But I don't think approaches like solid.mit.edu -- which seems to focus entirely on data mobility and storage rather than compute -- will actually fix very much if there's still a very high barrier to entry to software delivery.

Incidentally, Sandstorm's fine-grained containerization approach (https://sandstorm.io/how-it-works) makes it easy to use a new app for new data while keeping your existing app for old data, while still getting a unified experience. You can have repositories in Gitweb and Gitlab side-by-side on the same server, for example. And, of course, moving data between physical servers is completely independent of moving data between apps on Sandstorm. So you end up with a situation where you can move between apps and servers without really needing a way to migrate data, which I think is, technically speaking, more realistic that expecting everyone to use standardized data formats or write good export/import code.

"Software must be provided as a package – not as a service – with each user running their own private copy." This sounds like a going back to desktop apps. Or am I missing something?

The next line is: "It doesn't really matter if the user chooses to deploy to 'the cloud' or to their own machine, as long as they can run any package they want."

mnutt2• 10 years ago

I should also point out that sandstorm only gives apps a single mounted directory that is writable and every "grain" (running instance of an app) has a download button that includes that mount point, so while sandstorm doesn't dictate data formats, you can get your data out and likely migrate it to another app.

roschdal• 10 years ago

+1 for a more decentralized Internet.

api• 10 years ago

I think it's also about permission-free innovation, and maintaining a software and Internet ecosystem that allows that.

If we allow the Internet to become too centralized, ISPs and major cloud providers will have a large incentive to join forces or even consolidate. If you follow this trend the end-game could be an Internet that's like iOS: only whitelisted traffic is allowed, every link and protocol has to be approved, etc.

Then all innovation will stop because only huge players will be able to do anything new.

ocdtrekkie• 10 years ago

Email is already doing this too. I've had sites tell me I can only register for x or y if I use an email address from a major email provider.

fiatjaf2• 10 years ago

If we could come up with data that could be used by multiple apps, the thing would prompt everybody to decentralization.

However, every data we created is always highly coupled with the app in which it was created.

There's that effort called Solid, from Tim Berners-Lee, that is trying to change this, but they are so obtuse about ultra-complex standards that no one is using... maybe they're right.

wmf2• 10 years ago

I guess that would require breaking Web programmers out of the SOA/microservices mindset where data is strongly encapsulated by code.

kentonv14• 10 years ago

Solid is a noble effort but I am skeptical that it would solve the problem, for two reasons:

1. Application functionality is often deeply coupled to its data format. When building on a standardized data format, how does one add new, novel features? If you extend the standard, then you now have a non-standard data format, and the features you added won't be recognized by other apps that use the format. No one wants to wait for the standardization process to complete before they can ship a feature, so inevitably apps will ship various incompatible extensions. Then, when you try to move your data between apps, you find that a bunch of stuff breaks. There's no clear solution to this, other than for everyone to stop innovating, which obviously isn't what we want.

2. How does having standardized data formats suddenly prompt decentralization? Yes, it means you can more easily try out competing apps, but those competitors still face the same high barriers to entry. VCs don't like to fund the second player in a market, much less the tenth. Open source and hobby projects aren't suddenly able to compete when they still lack resources to run a service.

This blog post actually started out being specifically a response to Solid, arguing that decentralizing storage alone (without compute) is not enough, but I ended up not feeling great about targeting them so I cut that part out...

fiatjaf2• 10 years ago

I, as a programmer, am almost always able to migrate my data from an app to another, by using APIs, the command line and tiny bugged scripts that fetch data from one service, reformat it, repackage it and save it to another service, which uses totally different formats.

Maybe a better solution would be to create easier ways to do that, so everyone will be able to move their data.

zAy0LfpBZLC8mAC6• 10 years ago

However, every data we created is always highly coupled with the app in which it was created.

The emails that I write are not coupled with any particular app.

The HTML files that I write are not coupled with any particular app.

The ogg vorbis files that I produce are not coupled with any particular app.

The vCard files that [...]

Really, that's always been a strategy of businesses trying to lock in their customers. But open formats with various interoperable implementations do exist and work just fine.

jwildeboer• 10 years ago

Unhosted.org pioneered this already years ago. As did pagekite.

jstayton3• 10 years ago

I want to push back a bit on this statement:

Or can one random person, working in their spare time, build just the right app and reach millions of people?

I agree that it's very difficult for one person; however, to win doesn't always need to be defined as becoming the most popular, or reaching millions of people. What if it's 1,000 people paying $10/month? That is possible for one person going the service-oriented route.

This isn't a nitpick about Sandstorm per se — I do love what they're doing — and I understand they're making this argument because what they do isn't a service. But I do think it's possible to find diversity/decentralization in small, independent service-oriented companies that don't care about being popular on the scale of Facebook or Google.

kentonv14• 10 years ago

What if it's 1,000 people paying $10/month? That is possible for one person going the service-oriented route.

It may be possible for a single person to pull this off today, but it's quite a lot of work and responsibility, and it's hard to tell if it will pay off until you've put in a lot of upfront investment. Someone who writes a useful application for themselves on a weekend isn't likely to want to engage in this. But if all they have to do is upload the package they already built to an app store, that's a much easier pill to swallow.

Conversely, as a user, I am not very likely to want to put my data into a service run by one person, especially if the data is sensitive or if I don't want it to suddenly disappear. But if I can run the app on Sandstorm where it is automatically sandboxed and firewalled, with the data staying on my own server, and the app can never disappear, I'm much more comfortable.

nullcipher• 10 years ago

No offense to sandstorm but most of the apps indeed look like they were done in weekend. The blog author grossly underestimates what is required to make a great functional app (even those mobile apps that were done on a weekend aren't popular by any means). If this is the goal, then they should reposition themselves as "OS for distributing your weekend project" (no snark, am serious about this). It's a great niche if that is their initial target.

and I understand they're making this argument because what they do isn't a service

They do provide a service.

jstayton3• 10 years ago

They do provide a service.

That's true. I should have worded it differently. They're trying to attract developers to build apps for their platform, rather than launching them as a service (i.e., SaaS).

ilovecookies• 10 years ago

From a more philosophical point of view; the idea that everyone should become a programmer is honestly just plain wrong. I like what these people are doing but I don't think this could/should be done for more serious projects considering the amount of hacking that's going on at more centralized services like google-drive and dropbox. But if that's not their aim then that's all good I guess.

About shared data, can't you just use google spreadsheets?

jstayton3• 10 years ago

For most people considering Sandstorm, Google is probably the last place they'd feel comfortable sharing their data.

newsat132• 10 years ago

Their main page uses Google Analytics. Just saying.

kentonv14• 10 years ago

the idea that everyone should become a programmer is honestly just plain wrong.

I'm not saying that everyone should be a programmer. I'm saying that everyone who chooses to be a programmer and who produces useful code should have the ability to share their application with other people -- who may not be programmers.

Think of it like YouTube. YouTube doesn't replace film studios. But there's a whole lot of content on YouTube that you probably like, but that would never be produced by a film studio.

Klockan• 10 years ago

Kinda like flash games?

zAy0LfpBZLC8mAC6• 10 years ago

the idea that everyone should become a programmer is honestly just plain wrong

because?

ahartman005• 10 years ago

specialization of labor. different people are good at different things. some people have great people skills, but are poor at math/logic. they might make great salespeople, but poor engineers. why not use everyones' strengths?

"when the only tool you have is a hammer, every problem looks like a nail" I remember my creative thinking class, we were asked to brainstorm a solution to clearing the sidewalks of snow. I proposed a snow shoveling robot. the mechanical engineers had ideas like magnifying glasses above sidewalks to heat them, or making the sidewalks black so they absorb more heat. those were probably more elegant/cheaper solutions. society benefits from different viewpoints.

__jal• 10 years ago

the idea that everyone should become a programmer is honestly just plain wrong

For this to be fruitful, we need to define terms.

I agree that the notion that "everyone" should have the skill set of a full-time software engineer is silly. But I don't think that's what people mean when they say that.

The idea that "everyone" should be capable of creating software automation, I think, is a good one. By that I am thinking of things like spreadsheets, IFTTT, the late Yahoo Pipes, maybe some very basic scripting. I don't see this as substantially different than asking that "everybody" learn multiplication tables, basic algebra or how to balance a checkbook.

Or put another way, it is possible to function without basic algebra or trivially basic accounting knowledge, but extremely limiting. And I think that the ability to automate routine, simple things you want your computer/phone/cloud-enabled toaster-bunion remover to do on your behalf is becoming a similar basic life skill.

EGreg• 10 years ago

Decentralization is the wave of the future, for the next 10 years. We have reached peak centralization on the internet (the most centralized being WeChat).

The Web is the most widespread decentralized user-facing platform, followed by email and then perhaps bitcoin and git. We should build on top of them.

There are two possible security models. One is sandstorm's - where everyone installs apps on their own cloud. You still need to get a hosting provider, but then it's a one-click affair. Like on DigitalOcean.

Or you can have apps use the browser security model to communicate across domains, with each app running on its own subdomain. All the powerboxes and other interaction between apps would happen through a user agent session under the control of a user, or through oAuth tokens that the user issues for apps to communicate behind the scenes.

There are pros and cons to each approach. The second approach allows the apps to be hosted anywhere, whereas the first one lets apps share capabilities.

But at the end of the day, you can have organizations host collaborative apps for members, and embed widgets from apps in other apps, which is really cool.

aethertron• 10 years ago

A key advantage, I think, of the first approach (Sandstorm's, in theory) is in the capability of apps to run 24/7 and do useful stuff in the background, non-interactively, when the user is offline. Like receive network messages, pull rss feeds, crawl the web on your behalf, or make your idle server's computing power available to your friends.

pastProlog• 10 years ago

The big companies don't want decentralization, so the trend has been away from decentralization towards centralization:

* Usenet was a decentralized forum, like a decentralized HN in a sense. The first attack on it were the small-time commercial spammers. Then the government, RIAA/MPAA and last mile net duopoly (AT&T/Verizon) conspired to more-or-less sink it, which they did ( https://www.verizon.com/Support/Residential/internet/highspe... http://www.tomsguide.com/us/AT-T-Usenet-Access-Closed,news-4... http://www.tomsguide.com/us/Newzbin-Usenet-MPA-Copyright-Inf... )

* Newer peer-to-peer file sharing apps never had a chance. Napster, Gnutella and so forth, under attack by the RIAA/MPAA and government.

The trend is toward more centralization, not centralization. With a last-mile duopoly, antagonistic government and MPAA and RIAA beating war drums, decentralization has been under attack for years and the centralization forces have been winning. I see nothing in the visible horizon that sees things changing any time soon.

erikpukinskis2• 10 years ago

The trend is toward more centralization, not centralization.

What are you basing that on? When I think about journalism, that seems a lot less centralized to me. Bitcoin is decentralized money. Recording companies are giving way to smaller more specialized independent production companies.

I think software is in a weird recentralization due to this push towards services and away from products. But it's exactly the kind of technology cited above, and stuff like Ethereum, which will allow that trend to go back in line with the macro decentralization trend.

Maybe you can help me see what you're seeing that I'm missing?

Nutomic• 10 years ago

Napster was centralized, that's why it was taken down. Now we have torrents. When a torrent site is taken down, 10 new ones replace it.

I think with projects like Ethereum, IPFS and others, there will be a huge push for decentralization in the near future. Of course it will be limited to enthusiasts at first, but not forever.

ericjang• 10 years ago

"Diversity" is quite an overloaded term nowadays and I'm hearing it more and more often in everyday vocabulary.

I wish I had some quantitative data on this, but it seems to be used in every day contexts now, from engineering to research (diversity of data, samples), and of course, social justice contexts (non-white, non-male, etc).

I wonder if the increased prevalance of the word diversity is a result of co-opting definitions from the association that "diversity is cool"

cortesoft• 10 years ago

I am a bit confused by what you are wondering; all of those examples you give are using diversity in the same way, to mean having a wide variety.

Why diversity is a good thing might vary from case to case, but it is almost universally a good thing.

In this case, I think the article author is arguing that the diversity of app distribution channels is good for the same reason biological diversity is good; if something happens to make one distribution channel go bad, there are lots of others that can take over the role.

madgar• 10 years ago

a result of co-opting definitions from the association that "diversity is cool"

If you think that "diversity is cool" is just a given then I have to assume you are not from the USA, or have otherwise been insulated from the American culture war. (Lucky duck!)

In the US, the word "diversity" instantly alienates tens of millions of people.

pluma3• 10 years ago

It seems that as far as marketing goes. "Diversity" is the new "green".

Both terms obviously describe a good thing but people seem to be willing to suspend their disbelief whenever they encounter the term and laud something just for such claims (whether they are backed up by actual numbers or not).

I wonder if we'll see green-washing style "diversity-washing" too.

micro_softy• 10 years ago

Don't know much about Sandstorm but I agree with the statements.

The best software is often written by one person.

It may defy common sense but I have seen this happen again and again.

I like relatively small, open source software I can edit if I so choose.

pluma3• 10 years ago

I'd like to modify that claim:

One person can be enough to create an outstanding product.

There are stories of solo developers creating awesome products on their own, but there are far more stories of solo developers failing to ever come close to a finished product at all. Betting on a solo developer is high-risk, high-reward. Even if the developer is unreasonably talented in every aspect (i.e. great programmer, great designer, great marketer, etc) it can still be a matter of how far they can go without exhausting themself.

A larger team of less skilled developers OTOH may still be good enough to deliver a finished product. Plus, even if each developer is less skilled at most aspects individually, their strengths can add up and easily surpass that of an extraordinarily talented individual.

The reason you hear about excellent solo devs winning all the time is that you don't hear about the millions of solo devs failing. It's plain old survivor bias.

Teams may hold back extreme outliers but they empower everybody else and drastically increase the chance of actually delivering.

diggan• 10 years ago

Well, you're talking about two different things. The comment you're replying to, is talking about software, while you're talking about products.

chj• 10 years ago

How can you make sure you can't read user's data?

kentonv14• 10 years ago

I don't understand the question.

If you mean: "How does Sandstorm the company ensure that it can't read Sandstorm users' data?", the answer is that we provide software that you can run on your own machine. The software is open source, so you can verify that it does not give us any ability to access your data.

saynsedit• 10 years ago

"Software must be provided as a package – not as a service – with each user running their own private copy."

My ideology resonates with this statement.

On a more rational level, I agree that an environment that promotes application diversity encourages competition and innovation and ultimately will keep the tech economy humming.

I can see that removing the infrastructure requirement for distributing and providing web applications opens up web application development to a larger group.

There are shades of similarity of this idea with copyright reform and the free culture ideas of Lawrence Lessig. Once IP hits a certain threshold of common knowledge, preventing others from freely using it inhibits economic growth. Once something becomes cultural, it's hard for innovation to not grow out of it.

An example is Pokemon: independent companies can't make Pokemon products without gatekeeping from Nintendo. Currently, If a third-party pokemon product benefits consumers and drives the economy but is specifically bad for Nintendo then it won't be allowed. The counter argument is to innovate somewhere else but that's ignoring the fact that innovation is born out of experience and if I had no control over experiencing Pokemon regularly in my life because it's a cultural phenomenon, my ability to innovate is essentially handicapped.

Decentralization is about diversity

Related Stories