The Science of Crawl: Improving on PageRank

http://blog.urx.com/urx-blog/2015/3/31/the-science-of-crawl-part-3-prioritization

by jisaacso • 11 years ago

35 7 11 years ago

blog.urx.com

Relying on PageRank (or the graph structure of the pages) alone is no longer enough. Now you need to follow users and see who's clicking on what, and what trails are being blazed across the web. This is where Facebook's "Like button" comes in, or Google Analytics (or AdSense). Google can follow people around and see what's hot right now, and surface those links. WikiPedia may be a great hub, but if people are making a beeline for a Brad Pitt article on TMZ, then that Brad Pitt article may be more relevant right now than his Wikipedia page.

TL;DR: Search is hard. Google and others have taken ranking to levels far higher than PageRank, which is nearly 20 years old now.

Animats• 11 years ago

Following users works for popular culture and not much else. The head of Google search says "as the search engine gets smarter, the queries get dumber." It's not that useful even for product or service search - that just drives more traffic to sites that are already getting traffic. For a short period, Google tried using Google Trends to affect search ratings. That backfired badly; if someone could generate a lot of searches leading to their site for an hour, suddenly they went to the top of results. There were some very funny search results during that period.

As for URX, a company whose pitch is "Get paid to link into apps. It's not an ad, it's an action" probably isn't interested in improving search quality.

There's a big opportunity for Apple to move into search and offer a higher-quality experience for Apple customers. Apple makes their money selling to their users, not selling ads.

(Our pitch, of course, is to automatically do some due diligence on the company and use business legitimacy as an input to ranking. This works reasonably well, but for an ad-supported search engine, does not maximized profits. Adsense (ads on third party sites) is 30% of Google's revenue.)

jisaacso• 11 years ago

Author here,

The article discusses applying PageRank to help decide how a web crawler should discover new content. While this article does not touch on how URX evaluates search quality, I can promise it's a metric we care deeply about. One of the slogans on our home page "Page content + user context = relevance".

Happy to answer any questions.

solve2• 11 years ago

PageRank as a secret sauce that solves search -- this was actually always an overblown myth created in the early days of Google for PR purposes. It fits their PR mission perfectly because they want everyone to think that super smart algorithms are the secret to their success.

If PageRank was really as useful as they say, Google would have had tons of high-quality competitors from day 1.

pc2g4d• 11 years ago

Except for patents: http://en.wikipedia.org/wiki/PageRank#Relevant_patents

discardorama2• 11 years ago

PageRank (and other such systems like Kleinberg's HITS) was a big step up from the traditional IR approach people were taking in the late 90s. But Google quickly moved beyond it and incorporated a whole slew of other signals.

The Science of Crawl: Improving on PageRank

Related Stories