gyehuda

Very excited to see this. I used to work at Yahoo. This is a serious team with serious tech.

When this project was first published, the name of the company that published it was AOL. Before that, the company was called America Online. Subsequently it was named Oath, and now it's Verizon Media.

Names change. The common theme: names are not easy, sometimes they are beloved brands, sometimes they fall out of favor. Sometimes they were just bad ideas from the start, but happened anyway.

This gives rise to an interesting challenge for open source projects when you have an open source project in a github org, and the name of your company changes (or your company gets acquired), should you move the project? The problem is real since you don't want to lose your community, but you don't want to be stuck in the past.

They do an amazing job keeping it secret too. Can't even find them on the contributor list https://github.com/aol/moloch/graphs/contributors

Moloch is free. :-)

Preach Brother!

Moloch is a serious open source project, run by serious people who care about network security. They published their code under an open source license, showing off just how confident they are that it is solid. You can use this project to inspect packets on your network, you can learn how they built it and become a valuable network security engineer with a job somewhere finding people trying to hack in to your site. You can propose modifications to make this even better (and if your code is good enough, it will get accepted and used by security teams around the world). Or you could focus on the name they chose and the name of the company at the time this was published.

I'd seize the opportunity to talk tech and focus on network security. Infosec jobs pay better than brand marketing jobs.

You can see https://www.verizonmedia.com/our-brands to see the collection of online brands in the family. You might use lots of these brands today without really noticing. There's a lot of internet content you get via https://www.verizondigitalmedia.com/ which is also part of the same company. Aol is still a thing, people do use it. Many people use lots of these brands as part of their internet experience.

If you are here, you probably care about internet security. Verizon Media runs a pretty significant Bug Bounty program https://hackerone.com/verizonmedia which you can read about here https://www.protocol.com/hackerone-bug-bounty-virtual-verizo... and their blog here https://www.verizonmedia.com/technology/security#/bug-bounty

You might even be looking for a job as an information security professional. You can join "The Paranoids" team (now that's a good name, don't you think!) by checking out some of their jobs. https://www.verizonmedia.com/careers/search.html?q=paranoids

I fixed that for you https://github.com/aol

According to my sources, DARPA wasn't involved in any of the child sacrifices, but they might use Moloch to help secure their networks.

Naming open source projects is a challenging task. The team hears this all the time, but hey, open source is about code you can use, a community you can join to make the code better, and the pride that people around the world want to use your code. If you are on a blue-team, you'll want to look at this project. If you are making lists of names that distract attention this goes on it.

Sorta like https://www.wireshark.org/. But Moloch is a very active project, used by many, and used internally at Verizon Media. Aol is part of Verizon Media (which brought AOL and Yahoo together). Open source is very active here. ;-)

Compare also to https://enterprise.verizon.com/products/security/advanced-th... (formerly known as Protectwise), which is not open source.

oops, thanks. I meant to disclose my affiliation, not disclaim it.

FWIW, there was a research team at Yahoo who would actually get insights about the way people used language (relevant for contextual search) from Yahoo Answers. They were more active during the earlier years when the content quality was much higher. I don't know if they used Vespa in their mining process, but it would not surprise me if they did. Vespa is used in many projects because it's just that good. So if you are thinking of using it -- for Yahoo Answers, for Quora, whatever, go for it. I know a cancer researcher/engineer who wants to use Vespa for serving clinical reports and trial outcomes.

As for the current Answers site, well, we'll see what happens. I know the PM, a delightful person. I don't know the plan. But if there is a plan to make something useful from it, apply. It apparently makes money (otherwise it would have been killed long ago), and that means there's something to work with. When I started at Yahoo I was hoping to get onto the Groups team for the same reason -- a huge challenge to fix something that could be made cool again.

"Vespa is the single greatest piece of software Yahoo ever built. It's like ElasticSearch but a hundred times better. I am so happy." Laurie Voss Co-founder/COO of @npmjs https://twitter.com/seldo/status/912876700542787585

My job is to manage the open source process for Oath (which is essentially Yahoo + AOL). That includes helping ensure we can publish code like this and the hundreds of other projects we publish too. I'm the one who cares about open source licenses, patent clauses, github permissions, etc. Many large tech companies have someone in a comparable role and some of us work together in the todogroup to help manage the way we do opensource. I'm beginning to meet the people in Verizon who do the same. I hope their open source legacy grows too. Heck I celebrate when Google, Amazon, and Comcast publish great code too. It's good for us all. But Vespa is a real treat. It's really really special to Yahoo and we are very hopeful that the Big Data community sees how many things they can do with this, at scale, the way we have.

Some companies make money by licensing software. They are less willing to publish code since, to them, code is revenue and they don't want to give it away. Internet-media companies view code more like a required expense. Giving code away 1. helps reduce carrying costs 2. attracts developers 3. forces out of dependency debt, 4. encourages developers to make their code better 5. builds skills that transfer to other companies 6. make it easier for us to find people who already know the tech we use. etc. etc. There are TONS of upsides. Case in point: we can hire a Hadoop developer. Had we never open sourced the code, we'd have to keep an army of developers in house for a decade. Instead, they can leave to form startup companies (if that's what they want to do) and we still get the benefit of their creative effort. We also invested in one of those companies, so when they make money, we do too. Developers want to work at Yahoo (really) since it will help them build skills they can take elsewhere, or they can stick around and use those skills internally. Either way, why would a developer work on proprietary code when they can work on open source code which will give them more options.

There's a good give and take in the tech world. Sure, our code has helped Facebook and Google eat our lunch. But we don't blame the tech sharing for that -- since they've contributed quite a bit too. We work rather closely with them on a bunch of projects -- which help us all.

Sure, there are some projects that we'd consider the "secret sauce" that really differentiates us from others. We won't open source those. But a lot of code is there 'cuz we need to move bits around quickly. Sharing that code is not going to make or break a multibillion$ enterprise. It's actually going to help make it better in the long run.

To so make money: our sales people to do that, not the tech people. The sales people are given amazing products, and huge audiences to sell to the advertisers. Whereas a podcast might have a million subscribers, a popular radio program have 10 millions listeners, a TV show getting 50 million viewers, or a wireless company have 150 million subscribers, we have over a billion users -- and the advertisers LOVE that. So we all sell ads, and who ever does that better wins. But as tech folks, we're collaborative. It's not really a new thing, it's very much part of the fabric that has helped the internet evolve.

Note: TFoS is also a Yahoo open source project. The teams work together. https://github.com/yahoo/TensorFlowOnSpark

Proprietary code is expensive to maintain. Even though there's a dedicated team of a few dozen people that has been working on Vespa over the years there are thousands of developers who have been contributing to ElasticSearch, Lucene, and other projects in the open source world that are in a similar space. Getting contributions to Vespa will help it grow and evolve to make it better -- much like Yahoo did when it evolved Hadoop and scaled it out, much like Yahoo did when it help caffe, druid, hive, oozie, openstack, pig, storm, shark, spark, and tensorflow and tons of other projects (that it either created, co-created, or took from others and contributed improvements back to help make better for all).

Sharing code is fine since we use lots of shared code too. We don't sell code. So if someone wants to use Vespa to make an amazing product and make tons of money, we hope they do. We know that sharing Hadoop helped our competitors, but we also know that the revenue stream comes from ads. So we're glad to share code that makes tech better for everyone. As it turns out, many tech companies feel the same way and openly share code with the industry to help us all get to better tech platforms. In the tech space, it's not about grabbing more of the pie, it's about making a bigger pie. The internet revolution is young and the more we built it in the open, the better it will be for all of us.

The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on...

"Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source.

Arguably Yahoo!’s greatest legacy once it is a division of Verizon will be big data, after one of its engineers – Doug Cutting – wrote an open-source implementation of Google’s MapReduce that became Hadoop. What followed was an entire ecosystem of startups and projects crunching data at scale – Cloudera, Hortonworks, MapR to name three in a market some calculate will be worth $50bn by 2020."

Thanks, I came to a similar conclusion and shared it here (in case anyone wishes to see or comment). I'm not a lawyer, btw. https://www.quora.com/Are-GitHubs-new-terms-of-services-prob...

The ToDo Group has stopped working on the Open Code of Conduct and suggests to use an existing code of conduct.

Thanks. It's hard to measure what I can't see. Many companies do participate in open source, but it's hard to notice all the activity since there is just way too much going on in so many communities. What does surprise me is when senior tech people tell me their companies don't participate in Open Source since it is almost always the case that they do, but do it "under the radar." IMO it's important for senior tech folks to recognize and support the activity. After all, but embracing it, they can ensure that it is operating in ways that advance the company's goals.

I see this as one of the fundamental problems with the AGPL/Apache licensing patterns. The company can create an Apache driver, but it's not clear when a member of the community creates a driver if they can also use the Apache license. If all they do is present the driver, then presumably yes, but if they publish a service using their driver calling the AGPL licensed code, their driver should be subject to AGPL's terms which would apply the AGPL to it.

Alternatively they could assign the copyright of their code to a company? But why would they be motivated to do so?

This creates an asymmetry of rights where the company can do things that community members cannot. This seems to the very opposite of what the Open Source movement has been all about.

So far, AGPL has not been tested in court. GPL has. The scope of Derivative Works (reach) has as well (but the case law in this area is more complicated than most open source developers are aware of since copyright law is understood differently based on which federal circuit court hears the case). The more challenging consideration for me is the AGPL/Apache (DB/Driver) method that these companies are using -- and especially when you have Apache licensed community contributions (these apparently violates AGPL's own terms). I see this as an area that can backfire against the open source community -- and indeed I'd also like to know if FSF, SFLC, or GPL-violations takes a position on the use of Apache drivers to AGPL DBs. If so, it would be best for the open source community to know about this soon rather than after many have placed themselves in a position where their best option is to pay the vendors that have somehow managed to achieve lock-in. Ironically, this is the opposite of the goals that many in the open source community endorse. So if FSF takes a strict position on AGPL, it could accidentally result in the opposite of the freedoms we have been working for.

I'm also fascinated when companies use a license and then state in their FAQ that they didn't really mean all the terms of the license. If this comes up in court, the judge might take the FAQ into account and interpret the intent of the licensor, or might just look at the license text itself. It'll be interesting to see what happens. My advice for now is to assume the text of the license to be what the licensor intended and my hope is that companies that use open source licenses simply use the licenses that match what they mean. If they really mean that they want you to pay, they should just be clear about it and people can decide if they want to pay.

Dear jmillikin Let me make a helpful suggestion that is purely intended to be for your benefit (and if you work for a company, it would significantly help them too): contact an attorney who is familiar with Open Source legalities to get clarity on the obligations associated with distributing derivative works that use GPL. I just checked to make sure that you don't work in my company, 'cuz if you did I'd be very motivated to meet with you ASAP and help you understand some of the issues that GPL poses.

I don't mean to be condescending in my tone. Seriously. But Open Source licensing confuses many people -- and many form their opinions based on blogs (which are probably not very admissible in court), and not on the license text. Many people want and wish that the licenses work in the way they think it ought to. But that's not the way the legal system works.

Much like lawyers can't grok why a misplaced semicolon can cause a computer program to crash, developers don't seem to understand how contract language is interpreted by the legal system. In both cases though, the "interpreter" is very literal and does not heed to the authors intent as much as we might hope.