Google should open source what actually matters: their search ranking algorithm

http://cdixon.org/2009/12/22/google-should-open-source-what-actually-matters-their-search-ranking-algorithm/

by ivankirigin • 17 years ago

57 65 17 years ago

showerst2• 17 years ago

The notion that being 'committed' to open source means that a company should give away everything of value is ludicrous. Google came up with great software that makes millions of peoples' lives easier, and they have every right to keep it secret and profit off of it.

The zealous idea that open source and trade secrets cannot coexist is detrimental to the open source movement as a whole, in my opinion.

rriepe• 17 years ago

Well put. Almost any good idea taken to its extreme is a bad idea.

electromagnetic• 17 years ago

High taxation eventually reaches communism, if there's benefits like healthcare 'refunded' to the citizen and if there isn't then that ends up being a kleptocracy. Low taxation eventually reaches anarchy as the government is no longer in control.

Increasing birth rates enables economies to grow without becoming stagnant and old, placing undue demand due to pensions. An exceptionally high birth rate creates food production problems, housing problems, and generally the 'sardine can' effect. Low birthrates create a stagnant civilization that will eventually reach a tipping point when taxes get so high to pay for pensions that the younger generations will protest.

Market freedom to the extreme leads to monopolies taking advantage, 'gentlemen's agreements' between companies to price fix and many other problems that rules and regulations try to control. Equally a controlled market develops problems in and of itself as it's unable to respond naturally to market fluctuations, and quickly becomes unruly and unstable, and as many of the countries who have held long-term controlled markets are testaments to, they develop massive black-market economies to deal with these.

The zealous burning of fossil fuels, aside from global warming concerns, can expel vast quantities of radioactive materials, sulfurous emissions and a long list of things we've accepted in western society as being more harmful than smoking to the wide populations health, but rules and regulations on air quality increasingly get more lax (at least they have been where I live). With zero fossil fuel production, however, our civilization would collapse (at least with an immediate stop to production), and I'm not just talking the halting of transportation vehicles, I'm talking the stop in production of the advanced polymers used in virtually all aspects of our lives. The majority of people wouldn't have clothes on their back . . . my wardrobe would certainly be a lot cheaper if my clothes were all genuine cotton and wool. Oh and farewell cheap elasticated sweatpants.

Every good idea taken to its extreme is a bad idea and will create monumental problems in its wake. Electric vehicles are amazing, however a pure-electric is a ridiculous prospect in winter when the capacity hits below 60% of room-temperature, suddenly your vehicle can't get you to work and back on a single charge. Zero integration of hybrids is, as everyone knows, damaging to the environment but also an extreme waste of entropy. A 1 tonne vehicle at 100km/h is a massive store of energy, and allowing that to be wasted is criminal, the amount of energy humanity has wasted on braking at highway speeds would likely run the entire human civilization for over a year.

mattmanser• 17 years ago

I think you're missing the point, the article is pretty much a direct response to:

http://googleblog.blogspot.com/2009/12/meaning-of-open.html (submitted here: http://news.ycombinator.com/item?id=1008908)

In it a Google open source zealot rants wildly about openess being ultimate, apart from search algorithms which should be closed. It also fails to mention all the times open has failed (communism being one of the more spectacular failures of open).

A cynic might surmise what they actually mean is content should be open so they can profit from their closed search algorithm.

RyanMcGreal2• 17 years ago

> communism being one of the more spectacular failures of open

Wait, what? Communism is a spectacular failure of centralized planning over the entire economy. It has nothing to do with open or closed in principle, and in every implementation it has been aggressively closed in terms of information flows and decision making.

idlewords2• 17 years ago

How on earth was communism an example of "open"?

jedc• 17 years ago

It was written by Jonathan Rosenberg, who's a Senior Vice President for Product Management, and on the top management committee at Google. That doesn't quite meet my personal description of an "open source zealot", though he's clearly quite passionate about it.

showerst2• 17 years ago

Thanks for the link, that helps put the post into a bit more context.

That said, I think the author's point is still wrong. As a community, we have to understand that these open systems are an enabler of progress, some of which will undoubtedly be proprietary. Part of openness is understanding that you're not locking people into using this stuff, and they're not locked in to how they use it.

I don't see it as hypocritical that a business can talk about being open and how critical it is to give back, while still keeping trade secrets to pay the bills.

I recently saw the CEO of the Economist group speak about journalism, and one thing he said (paraphrasing) was that the best defense of journalistic impartiality in the modern age is to turn a healthy (and diversified) profit. The subtext was that you can then risk offending any given party by doing good reporting without the appearance of favoritism, and without having a corporate (or nonprofit) master to please.

In the same sense I think that the best defense of free software is to allow companies to use it on their own terms, and give back when they can/will. The most useful open source projects are used (and likely abused) ad-nauseum, and trying to limit any company, even one that preaches hypocritically about open source is a recipe for them to tell the community to shove it and go hand money to <insert big vendor here>.

While it's fairly easy to preach the open-source gospel to fellow programmers and nerds that appreciate the value of this stuff, it's business people and non-tech-literates that oftentimes allow the checks to be signed, and the open source community needs to look to them like a group of enablers, not inflexible idealogues.

allenp2• 17 years ago

Additionally it may be the case that a large part of the value of the algorithm is created from its secrecy. If the algorithm was open it would be trivial to create sites with artificially high rankings which would reduce the value of the search results.

chollida1• 17 years ago

> If the algorithm was open it would be trivial to create sites with artificially high rankings

I don't necessarily think this is true. Sites that have high page rank, do so because other sites of high value link to them.

Even if Google open sourced their search engine, spam sites would still have a hard time getting high quality inbound links. The reputable and high ranking sites wouldn't have any increased incentive to link to low quality spam sites.

kvs• 17 years ago

Agreed.

After reading the comments at cdixson.org by the author and reading openness article at Google, it seems like author is simply saying: "I agree they won't open it. But then they should stop talking sactimoniously about openness." which I agree with as well.

Here is an interesting perspective: "And millions of websites will have to continue blindly relying on a small group of anonymous engineers in charge of the secret algorithm that determines their fate." Have you ever been delisted by Google? Is there a transparent process to get back on the list? Perhaps transparency is what Google needs the most at the moment.

jacquesm5• 17 years ago

> Perhaps transparency is what Google needs the most at the moment.

Absolutely, and not only on the search engine part of it, Adwords/Adsense are extremely opaque and as a publisher continuously give you the feeling you're being screwed without any chance to prove that for lack of information.

iamelgringo2• 17 years ago

The notion that being 'committed' to open source means that a company should give away everything of value is ludicrous.

I disagree with the statement that Google would be giving away everything by opening up page rank. Google's "secret sauce" is hardly page rank alone. Google is really secretive about this stuff, but they arguably have the world's larges corpus of multilingual text in the form of their html index, coupled with the world's largest fault tolerant compute cluster, the largest repository of scanned text,the largest repository of digital video, and probably the largest repository of digital audio in the world. I'm sure that they also have the world's largest repository of people's surfing and searching habits, along with the one of the largest sets of analytics data.

Even if page rank, map reduce, GFS, etc.. were open sourced I'm sure they'd still find a way to make billions off of the data that they have.

But, I think the larger point that the author makes, is that unlike many giant technology companies (Microsoft, HP, Apple, etc..), Google has made billions off of using open source software. And, while they are a very large supporter of open source, what total percentage of their profits go back to OSS? What percentage of their code gets re-released as open source? These are good questions to ask, and I'd love to hear an open discussion about it.

buster• 17 years ago

opening this algorithm would lead to A LOT of new spammers and SEO-crap.

Google, please keep it a secret, PLEASE!

mustpax• 17 years ago

The analogy to "security through obscurity" is flawed. Open security works when there's an obvious way a secure system should behave, say, make it infeasible to open a lock/decrypt data without the correct key.

In Google’s, case the correct relevancy ordering of the web is a huge point of contention. So they cannot simply open it up hoping to close out all the holes in the open. It is also hard to distinguish acceptable behavior (linking) from unacceptable behavior (selling links) except in the aggregate through heuristics that assume ignorance on the part of the attacker.

In this sense, Google’s system is a lot closer to an intrusion detection system than a lock or a cipher suite. You want to establish some ground rules but there’s no added security in declaring what you’re monitoring on your servers and how.

swombat• 17 years ago

That is a completely ridiculous point to make. Come on, be serious here. What would be the benefit to anyone except for spammers, if Google open-sourced their algorithms? It's a lose-lose option. Google loses some money, we lose a lot of search engine accuracy, and spammers gain new ways to screw around with search results.

And does anyone gain from it? Not really. At this point, Google's supremacy is due just as much to their hardware know-how, to their market leader position, and to their excellent brand, as it is to their search algorithms. It won't help any competing search engines - just spammers.

roc2• 17 years ago

The stated goal isn't to help other search engines; it's to get the community to help Google improve, for the good of all its users.

The theory goes: if you let more people look at the algorithm, they can suggest better answers than Google alone, to thwart the spammers who are gaming the system either way.

But I suspect the real goal has more to do with the deep suspicion that Google can or does do a significant bit of hand-tweaking. And these people want to force the algorithm into the open to either end or curtail that.

jacquesm5• 17 years ago

I'll believe that when I see open source achieve a conclusive victory against email spam.

Spam:email = 90% or worse

in search results it is closer to the opposite.

I'm sure once the open source community actually manages to do this and prove the 'many eyes' doctrine applies to more than just security and bug fixes that many doors will open.

bbsabelli• 17 years ago

tl;dr? Author confuses source code with revenue. Welcome to the less-than-free world.

jlees• 17 years ago

An interesting thing to think about even though the basic reaction is "wait, no, that's their industry secret/revenue basis/whatever". Google is so dominant right now that maybe it's going to get to the point where, in the interests of the open web, they get pressured and lobbied to release their secret sauce; after all, as swombat points out, it's not just the algorithm by a long chalk.

It's especially interesting if you consider the origins of PageRank in academia, freely published and peer-reviewed, and still not that well understood by laypeople; would the open-sourced magic of Google even be understandable by that many folk? Those that do understand it could make a fortune as consultants; at the moment, shady SEO snake-oil salesmen are so prevalent and 99% of their advice is outdated or plain wrong, something that definitively calls them out on it could be helpful.

amix• 17 years ago

I would love if they open-sourced their infrastructure tools such as GFS, BigTable and Map-Reduce tools. Most of these tools are built using open-source technology such as Linux.

stanleydrew2• 17 years ago

Apache's Hadoop (http://hadoop.apache.org/) has open-sourced implementations of those tools.

iamelgringo2• 17 years ago

Yahoo was the company that opensourced hadoop, not google.

mbreese5• 17 years ago

I'd be interested to see some benchmarks (unofficial-probably) between Hadoop and Google's C++ versions (at least, I assume Google's is C++). I've always wondered if there was any extra overhead in using Hadoop. I know Java is almost if not faster than C++ in some areas, and long running servers (like Hadoop) should benefit the most. But I wonder if there are some extra limitations Java places on things like BigTable.

stuff4ben• 17 years ago

I don't get it, is the author talking about PageRank? http://en.wikipedia.org/wiki/PageRank What more does he want?

minsight• 17 years ago

The algorithm used by Google today is light-years away from that described in the PageRank paper. It uses a form of PageRank, but it also uses many other parameters.

spf• 17 years ago

If it's secret, then how can you be so sure?

My impression is that PageRank, while initially extremely successful at filtering signal-from-noise, is very susceptible to manipulation. For example links farms which make unimportant pages seem important. As a result of such manipulation, Google started modifying their crawling, in order to reduce the amount of garbage going into PageRank (less garbage in, less garbage out). For example, they might look at the age and ownership in DNS records, and exclude domains that look fishy based on some heuristics. The spammers fought back with more tricks, and the resulting arms race has come to have an acronym: SEO. But the underlying idea of using an iterative approximation of the likelihood of encountering a page via a random walk of the web is preserved. (Some details here: http://www.miislita.com/information-retrieval-tutorial/matri...)

If that's right (and I could well be wrong... would like to be convinced...), then I don't see it as a "light-years away" kind of issue, but rather a series of hacks to stay one step ahead of the spammers. I don't see any particular value in open sourcing those.

mooism2• 17 years ago

Google rankings are determined by much more than PageRank.

Mystalic• 17 years ago

Google is a business. Their job, legally defined, is to create value (aka profit) for its shareholders. A more open web has helped Google, but opening up its search algorithms, a cornerstone to Google's growth, is just ludicrous and bad business.

Compared to a lot of other companies (Apple, Microsoft, nearly every other tech giant), they are an open company. It's already great that they realize there's value in an Open-source browser, and open-source OS, and multiple open platforms.

tome• 17 years ago

Their job, legally defined, is to create value (aka profit) for its shareholders.

This is a meme I see a lot, but never with any evidence. That the board has a fiduciary duty to the shareholders I can understand. That it would have a legal obligation to create value, well, I think that would be hard to define, and I have never seen any evidence for the existence of this obligation.

mbreese5• 17 years ago

In the context of a publicly traded company this is the definition of fiduciary. The company has sold stock to investors with the sole purpose of increasing the value of the stock. Therefore, it is up to the board to make decisions that reflect this duty.

You can argue about the timeline (short versus long-term growth), but you can't say that there isn't a legal obligation to attempt to increase value.

selven2• 17 years ago

It may be true that it is a company's legal responsibility to serve its shareholders first, but that's not the way it should be. With this mentality, corporations will minimally follow the law (if the fine is less than the profits, the corporation may be legally required to break the law!) and do whatever they can in their own interests. Society as a whole would benefit massively if caring for others and society in general was regarded as a good thing.

buro9• 17 years ago

You need to take a look at Dodge v. Ford to see the evidence: http://en.wikipedia.org/wiki/Dodge_v._Ford_Motor_Company

Not to say it's interpreted correctly, but that is where the 'meme' stems from. If people quoted primary sources this might be more widely understood.

mlLK• 17 years ago

http://www.google.com/support/webmasters/ easily brings down this bogus claim. I honestly don't think much would change even if they did open-source their PageRank algorithm because the people willing and able enough to exploit it already do, and regardless of how much their PageRank increases Google will send your domain deep into an internet abyss and out of their index.

allenp2• 17 years ago

I thought open source was about software, not about logic (business or otherwise) - am I missing something here?

lacker• 17 years ago

(I used to work in the Search Quality division at Google.)

Security by obscurity is a bad analogy here. The problem is that Google's algorithms are not mathematically perfect, in the sense of cryptography. Instead, they merely statistically tend to find good results. You can't find bugs just by looking at source code. You have to statistically analyze the effects of any proposed changes.

btilly• 17 years ago

I upvoted the story not because I agree with it, but because it explains a very successful open sourcing strategy that I wish more people understood.

In general people are willing to spend a certain amount on a solution. Therefore if you can things that make the complement to what you make money on cheaper, that increases how much you can charge for your piece of the solution. Open sourcing is a way to make those pieces cheaper.

Google does this. They give away tools and knowledge about how to make websites. They give away analytics and an A/B testing platform and educate people about how that makes it easier to produce good websites. This makes building good websites cheaper, and therefore increases how much people are willing to spend to advertise those websites.

This isn't a new strategy. In the late 90s when Oracle and IBM got into open source, this was their strategy. Oracle wanted to get people to stop paying for Sun hardware so Oracle could negotiate higher prices. (I know more than a few people who were surprised to find that licenses for Oracle on Linux are higher than they are on Sun.) IBM was looking to reduce the price for complements to IBM websphere, and was looking for opportunities to avoid having customers pay the Microsoft tax.

If you run a company and wish to strategically open source something, you should think in the same terms. What is the complement to what you are charging for, and how can you make that cheaper?

selven2• 17 years ago

And what is wrong with commoditizing the complement? Google's working hard to make operating systems free and we'll probably see Microsoft working hard to open up search engines. End result: both operating systems and search engines will be open and the consumer benefits.

xcombinator• 17 years ago

Well, I can't understand this article, there is not "secret sauce". The google pagerank algorithm is open for all to see and understand. I myself made my own simple implementation.

Everybody could do it. This man is not asking for code, is asking for DATA only google has. It's like an atheist saying to a catholic what he must do because he believes. When this man gives something of value to the world,like google did(I remember the Altavista-Ad-flickering in your eyes days) then he could demand to be given by others.

eli3• 17 years ago

The documented PageRank algorithm doesn't have very much to do with how results are actually returned from Google today.

oujheush• 17 years ago

I am disturbed by the lack of faith in open source among the readers here. I mean, I understand some skepticism but this, to me, is pretty extreme IP-fetishism.

It's perfectly understandable. Lots of money is made in the existing paradigm and so many of you have incentive to believe it's the most rational system. Besides, hard to imagine Google's doing something wrong, right?

But the argument that "the only people this will help is spammers"? Really? Are the only people helped by open cryptographic protocols those who look to decipher traffic?

"They have every right to keep it secret and profit off of it." Let's allow this for a moment. We've still got a false dichotomy. Making it public will not deprive them of their profit. First, incredible marketing coup. Does anyone else remember how much of Google's initial strength came from its rabid fans? This would do a lot of re-energize that base.

Second, are there going to be new startups which can effectively compete with Google, even given its algorithm? How are they going to provide a better experience, or serve the volume of traffic, or index a larger amount of pages with the same algorithm with fewer resources?

Third, who says they have to authorize competitors to use it? This is I think one of the more interesting points. They could release the algorithm but license it only for non-commercial use for anyone not themselves. Ridiculous? Because Microsoft would violate such a deal anyhow? Because the algorithm could simply be reimplemented? Perhaps. But it would be another roadblock.

Fourth, why do we continue to believe that Google's strength comes from proprietary code? Why do we not recognize that its strength comes from mindshare, user experience, and quality of execution? None of which would be negatively affected by releasing the source.

Perhaps Google would face heightened competition as a result of releasing its algorithm. Perhaps this would outweigh the benefits of doing so. But there would be significant good which would result, including public feedback which would make it significantly harder for spammers to be successful.

But I'm just a foaming-at-the-mouth open source radical. And I'm sure history is on your collective side. If there's anything we've learnt from a hundred years or so of Computer Science (overestimate if you only count actual programming, underestimate if you allow Babbage as you should), it's surely that secrecy is what drives innovation, right?

ez772• 17 years ago

As a matter of fact, I think Google represents the ideal business model: one which should work even in a patents-free, copyright-free world.

ez772• 17 years ago

Let me elaborate. Not even Stallman would call for this... What can of totalitarian regime is desired, where businesses (even individuals?) are forced to publish all of their code? Wouldn't it be good enough to do away with patents and copyright? As long as I manage to keep my stuff private, let me (and Google) face the advantages and disadvantages.

Some balance in the force, for Yoda's sake.

rpdillon• 17 years ago

This post manages to completely miss the point of the Google letter on openness.

As some have incorrectly asserted, Google's latest post is not about "everything open", it is an internal memo to product managers that it occurred to them would be fit for public consumption.

The central thesis of the letter is that "open" is hard to define (and defined differently by different people) and yet crucially important. Because it is hard to define, many people misuse it (OpenDNS comes to mind), and no one can call them on it because it is a vague term.

Internally, Google employees argue about what open means, and the purpose of this letter was to clarify what open means to Google. They lay it out in extremely clear terms, and it is well structured.

This response essay by Dixon misses all that completely - he unilaterally defines "open" to mean "open source", and then asks for Google to give away their software, claiming that if they don't, they're hypocrites.

Going back to Google's definition of "open" (which is complex), one significant aspect they emphasize is the notion of "lock-in", i.e. locking in users, and locking out competitors. This is one area where Google puts their money where their mouth is; they let users leave. You can get at your email, export your Google Reader content, download all your Google Docs, archive your Picasa pictures, and export all your contacts. That means if a competitor comes along with something better, you can start using it. Try that with Facebook.

Dixon acting like that is worthless because it isn't the "open-source" form of "open" is patently absurd. If this is confusing, I highly recommend reading Simon Phipps essays on open source to understand what kind of software makes a good candidate for a business to open source (operating systems, browsers, compilers...the basics that we all use to compete) and what does not (grid/cloud computing algorithms, specialized software, and other types of software that are differentiators). Essentially, the business model is to pool our collective resources in areas in which we don't want to compete (web server software, operating systems) and to spend time innovating in areas we do wish to compete (search algorithms, voice recognition, image analysis, etc.) Over time, as those specialized areas become more mainstream, they become better candidates for open-sourcing.

eli3• 17 years ago

Yeah, and where's my pony?

vicaya• 17 years ago

I'm an open source advocate (even for search algorithms in general), but ranking algo/model is inherently adversarial and gameable until true search AI (always correctly identify good and original content without false positives) is feasible.

I think it would be a good revenue strategy, if they charge $dollars to display top 10 positive features and top 10 negative features of any given query-document pair without any real weights/scores (maybe re-squashed with some logistic function). The $dollars amount is computed dynamically to offset the negative impact of the partial reveal of the ranking algo/model. This can be a tremendously useful tool for troubleshooting search results by site owners.

russell• 17 years ago

A more reasonable proposal would be for someone to pull a netflix; $1 million or even a $1o million prize for a significant advance in search relevancy. $10 million would be cheap for MS, Yahoo, ASK, a VC, or even DARPA.

Google should open source what actually matters: their search ranking algorithm

Related Stories