Amazon’s quiet open source revolution

https://www.infoworld.com/article/3694090/amazon-s-quiet-open-source-revolution.html

by unripe_syntax • 3 years ago

126 67 3 years ago

tyingq2• 3 years ago

Directionally good, though I'm sure there's a decision tree that's mostly driven by how it benefits AWS. Something like "is committing this back going to save us time maintaining it separately without giving away IP we care about?".

nazgulsenpai• 3 years ago

I would say that's true for any company that contributes to open source projects.

tyingq2• 3 years ago

Yes, though there's some things that are somewhat unique where Amazon is competing with open source products. For example, they have to offer MKS, but would rather you use Kinesis, EventBridge, etc. So perhaps less incentive to make Kafka better for everyone.

ghaff• 3 years ago

It's probably not so much IP that they care about as IP that gives them some competitive advantage by keeping it secret.

You can probably overstate the degree to which Amazon didn't contribute to and interact with open source communities 5+ years ago (and probably also overstate the degree to which there's been a complete sea change) but, as you say I think it's fair to say there's been a directional shift with more substantive activity in addition to what I'd mostly describe as talking the talk previously.

The industry as a whole has also developed into this incredibly complicated coopetition landscape especially around containers in the past decade.

jrochkind1• 3 years ago

I feel like that kind of self-interest is the origin of open source, and always what has powered it when it's working.

The original apache httpd, for instance, was written by people working at different organizations collaborating together to build the web server that they needed. It wasn't something people did as just a hobby, although the individual contributors may also have had some altruistic and certainly cooperative motivations. It definitely wasn't something people did motivated by trying to make money off of selling apache httpd in some way.

Open source always requires self-interest to fuel it. The original model was people collaborating across organizations because it provided something useful to their organization -- exactly this. When this stops happening, because organizations cut budgets to the bone; because they gatekeep anything that might benefit competitors as well as themselves; because they can free-load off existing open source instead -- what other source of self-interest is there?

One is people trying to profit off of the open source itself, with "open core" models, or models where the license can appear open but tries to ensure that the creators maintain a monopoly on any profitable aspects. Or, we can just get lack of investment in open source everyone is trying to free-load off of, withering open source with burnt out maintainers.

Companies committing to open source because it saves them the time of maintaining it separately is the best case, and describes the environment of the "golden age" of open source.

pydry10• 3 years ago

I met an open source maintainer once working on a very popular tool everybody uses who was practically impoverished. From talking to him it became apparent that he was routinely being taken advantage of across all spheres of his life.

It made me wonder how much valuable open source is powered, quietly, by people who are just used to being exploited.

taftster• 3 years ago

This is insightful.

Open source always requires self-interest to fuel it.

That self-interest doesn't necessarily have to be profit motivated either. As one of the replies to your comment suggest, probably the majority of open source is maintained as a hobby by an invested/interested user. Open source does not have to have a profit motive to make sense.

But likewise, a profit motive is also congruent with open source. It greases a lot of axles, to put it nicely. There's definitely some disparity in open source contributors in terms of money being made, almost like the income gap between wealth and poverty in general. I wish we could do better at fixing that and helping the "little guy" that is contributing their soul into a project.

Still though, open source is driven by "selfish" means, in whatever form that takes.

pkkm• 3 years ago

Sure, but why does that matter? A lot of good software got developed by itch-scratching.

Dowwie• 3 years ago

Their s2n projects seem like valuable contributions to the ecosystem. Strong, well maintained libraries supporting networking and cryptography are important.

phpisthebest4• 3 years ago

I wish that same drive would come to their devices division which do the absolutely bear minimum not to be sued for GPL Violation when they use GPL code on their Fire Devices

Edit: Also doesn't AWS have a history of Microsoft style EEE, where they embrace an open source project, make a service from it, extend that service in a non-open way and then push people to use their extended non-open service instead of improving upstream?

Hasn't this resulted in a number of projects changing their licensing to prevent it?

smatija• 3 years ago

I think ElasticSearch license change was biggest such change, as was discussed previously on HN: https://news.ycombinator.com/item?id=25776657

pydry10• 3 years ago

Their "obsession" with contributing isnt really in evidence on their fork:

https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da...

Worth bearing in mind that the only reason fork even exists was so that they didnt have to pay Elastic a cut when selling hosted ElasticSearch.

If they cared about open source they could start by not undermining the business models of companies that do more than just pretend to support it.

detaro• 3 years ago

Not sure I'd count that as "EEE"? I guess in a way you could see "We can't compete with AWS at running our own software" as AWS "extending" it, but I don't think it is quite that?

xyzzy_plugh• 3 years ago

do the absolutely bear minimum not to be sued for GPL Violation

Could you expand on what you mean? Are they violating the GPL? What does it mean to do the bare minimum but not violate the GPL?

It's unrealistic to expect corporations to go above and beyond when complying with something like a code license. If you want someone to do something, put it in the text of the license.

cma• 3 years ago

If you lower expectations to literal legal text only, then they don't even have the "it would be bad for PR and developer relations not to do it" incentive.

Firmwarrior2• 3 years ago

They drop a zip file with their open source code somewhere on the website at some point, but don't try to upstream patches, work with Linux kernel people in public, or provide a public git history

To be fair, it's a tough sell to tell your management chain that you have to spend a week every month arguing and refactoring code just to make some internet people happy about it

phpisthebest4• 3 years ago

>It's unrealistic to expect corporations to go above and beyond when complying with something like a code license.

This is completely false as they (amazon) want to market themselves has developer friendly and supportative of open source

Their actions however say something completely different.

Further Loads of companies and organizations go beyond the legally required bare minimum in all kinds of things not just open source. You have a completely warped view of human interactions if you want to boil all humanity, and all interactions to just what is "legally required of us"

stonemetal122• 3 years ago

It's unrealistic to expect corporations to go above and beyond when complying with something like a code license.

I agree up to the point where they start going on about how much they love and support open source. At that point going beyond legal necessity should be a given.

marcosdumay• 3 years ago

Also doesn't AWS have a history of Microsoft style EEE

AFAIK (but I'm far from knowing about all examples), no, they have a history of taking open-source software, forking it out of the control of the creators, and extending it there without bothering to sync the changes back.

It is hostile to people that want to gain anything from their software, but it is not against the license.

aprilnya• 3 years ago

I’m pretty sure that is EEE

andrewstuart• 3 years ago

One of the things that led me to lose interest in AWS was its attitude to open source.

pydry10• 3 years ago

Though it was always incorrect to lambast AWS for “strip-mining” open source

It really wasn't. That was their whole approach to Elastic.

The tantrum they threw at Elastic when they relicensed was particularly cringeworthy.

prepend• 3 years ago

They didn’t throw a tantrum. They just forked and continued. A perfectly reasonable approach.

I think it’s important to describe things accurately. Disliking something isn’t a “tantrum.” I think amazons blog posts were pretty normal speech and tl:dr’d to “we don’t like it, so now we forked it and will maintain under Apache.”

And they did that, so it’s actually kind of nice as I can’t use ElasticSearch under its new license but can use open search.

pydry10• 3 years ago

Tantrum is pretty accurate I think. They voiced their extreme displeasure in a war of words with Elastic. They knew that the relicensing was targeted 100% at their gravy train and they condescendingly declared that they would "step up on behalf of open source" while they stepped up exclusively to prevent Elastic getting a cut of their ElasticSearch hosting profits.

The fork is really half hearted with about 1/10th the resources Elastic dedicates to the mainline branch:

https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da...

Meanwhile, the only effective licensing difference is that you can sell hosting with their fork - something that only benefits them.

briffle• 3 years ago

Meanwhile, On Google Cloud, instead of using a service that uses a rebranded elastic, you just install elastic from your GCP console, and the company that develops it gets revenue.

And Mongo, Redis Enterprise, Databriks, neo4j, etc.

Their 'partner solutions' and not then directly competing with their partners is a win/win.

https://www.elastic.co/blog/elastic-and-google-cloud-announc...

helen___keller• 3 years ago

Makes sense if you think about incentives. AWS has a lot of financial incentive to build on open source projects used internally (at a massive scale, I assume). Meanwhile, not a lot of incentive to maintain internal forks on a big project like PostgreSQL

albert_e• 3 years ago

Is there a good lookup list for all opensource projects and their corresponding AWS branded services?

Presto/Trino : Athena

scientism• 3 years ago

It's Azure but, Apache Atlas : MS Purview

up2isomorphism• 3 years ago

The author seems like a “open source strategist” who does not write code, but if one does, he will likely understand the difficulties of contributing to open source in a cloud service environment and why the observed “free ride”. It will take too many cycles to push an even trivial change upstream and sometimes you can never do that. But your customer does not wait for you, they will happily go to your competitors when they “free ride” more than you do.

EdwardDiego2• 3 years ago

It'd be nice if they open sourced some of their libs. When I was trying to debug a very odd issue with their Python Glue stuff, it was impossible to delve into, because the open source Python just wraps a Java blob you can't see.

https://github.com/awslabs/aws-glue-libs/tree/master/awsglue

robertkoss• 3 years ago

This is one of the reasons we moved away from Glue..

https://medium.com/claimsforce/lakehouse-running-delta-lake-...

JackFr• 3 years ago

just wraps a Java blob you can't see

I ahve to admit no familiarity with Glue, but if that Java blob is bytecodes, shouldn't it be trivially decompilable?

EdwardDiego2• 3 years ago

The blob lives in Glue, so not available to decompile.

tracfren• 3 years ago

I lead client-side application development in AWS IoT and I can tell you we’re investing heavily in open source in AWS IoT. We can build features a lot faster without having to own the infra ourselves and cut out a lot of internal red tape. Customers love features and customization.

Examples: AWS IoT Application - https://github.com/awslabs/iot-application - OOB app for monitoring IoT assets.

AWS IoT App Kit - https://github.com/awslabs/iot-app-kit - Library for building AWS IoT apps.

throwchainaway• 3 years ago

To the commenters making embrace/extinguish overtures, I’ll say that log4shell got a lot of Amazon very spooked re. supply chain security.

There were a number of internal programs spun up with blank checks to mitigate a future log4shell. The Open Source program was one of those

cs702• 3 years ago

Demand for a product increases when the prices of its complements decrease. For example, demand for mobile phones increases as the price of mobile OSs and applications declines.

To maximize demand for a product, make its complements free. For example, to maximize demand for mobile phones, make the operating system free and add a growing number of built-in default apps for free.

Surely, Amazon is thinking along these lines. Which Amazon products/services benefit from all the free code Amazon is contributing/releasing? AWS.

m463• 3 years ago

I don't see software <-> aws as complements like windows <-> pc, because this software doesn't necessarily need aws like windows

myroon5• 3 years ago

Author disclosed he worked at AWS, but worth noting he led open source marketing:

https://www.linkedin.com/in/mjasay

ctvo• 3 years ago

Why is that worth noting other than he probably still has an interest in open source at AWS? He works for a competitor now, which makes this post even more remarkable.

lamontcg• 3 years ago

Though it was always incorrect to lambast AWS for “strip-mining” open source

Amazon absolutely strip-mined open source.

In 2001-2006 Amazon was all built on open source and had ditched HP and Digital Unix for Linux, RedHat, Apache, Xen, etc.

It was abusively hard to contribute back to open source projects, including the Linux kernel, and you had to get extensive signoff and review to ensure that you weren't giving away any IP with your lock order bugfix or whatever. There were also constant reminders and threats to not engage in open source discussion in any way that could leak company IP which created an atmosphere where it was much easier to just not participate at all. It was definitely very easy to pull open source into the company and very, very difficult to participate going the other direction.

_msw_• 3 years ago

Back when you were at Amazon, I was at Red Hat. (disclosure: I am at Amazon now, and have been for a bit over 13 years).

From my perspective when I was at Red Hat, Amazon was very much an IT consumer. Amazon was a _hugely_ important customer for Red Hat, even appearing on stage to give their testimony about the benefits [1]. It was a "lighthouse" customer that really opened some doors to selling Enterprise Software.

Part of what we sold to Amazon was being their connection "upstream". In 2001 it was quite unusual for an IT shop like Amazon to participate directly upstream. One benefit of being a customer of Red Hat's was: they will fix bugs and work upstream on your behalf.

It was a very reasonable approach in those days to use an existing contractual / support relationship with a vendor like Red Hat to deal with all of those details and complexities, which often required strong social connections among open source developers and maintainers working together upstream.

I think that over time as large companies shift to consume open source software directly from upstream, they have to build the capabilities and connections to participate directly rather than going through an intermediary like Red Hat. It can take a fairly long time to do that.

[1] https://www.computerworld.com/article/2577912/linuxworld--am...

Amazon’s quiet open source revolution

Related Stories