PG's essays, ranked "Google-style"

http://www.solipsys.co.uk/new/PaulGrahamEssaysRanking.html?HN

by RiderOfGiraffes • 16 years ago

85 27 16 years ago

solipsys.co.uk

RiderOfGiraffes9• 16 years ago

In case anyone is interested, here's a clickable chart of all PG's essays showing explicit links between them:

http://www.solipsys.co.uk/new/PaulGrahamEssays.html?HN

I was prompted to provide these links by this item:

http://news.ycombinator.com/item?id=2011897

I do regularly re-visit PG's essays, and I use these charts/graphs to see what catches my eye on any particular occasion.

sudont2• 16 years ago

A little context may be required. I'm assuming that a bigger number means that it's higher linked? More important?

RiderOfGiraffes9• 16 years ago

I kind of assumed everyone would know how the underlying page-rank algorithm works.

Put people on pages at random, and have them click on an outgoing link at random. Then, 20% of the time, teleport them to a new, random page. What is their distribution in the long run?

The higher the number, the more likely you are to end up there. This means you have more incoming links, but they are incoming links from pages with higher rankings. Incoming links from lower ranked pages - pages with few incoming links - don't give you much "Google Juice".

I'm curious, do people really not know this? Should I write it up? It's a standard piece of linear algebra to find the eigenvector of the appropriate matrix. I thought pretty much everyone would know it.

raganwald• 16 years ago

I get the algorithm, but what tool(s) did you use generate the page itself?

RiderOfGiraffes9• 16 years ago

OK, here's the script that drives the process.

    echo Retrieving index
        curl http://www.paulgraham.com/articles.html > data0

    echo Extracting URLs
        grep -ho "<a href[^>]*>[^<]*<.a>" data0 | grep -v http | grep -v RSS > data1

ComputeRankings takes the list of essays, extracts outgoing links to create a digraph, and then computes the Page-Rank of each page.

CreatePGER takes the results and an HTML template and glues them together to create the rankings page.

    echo Creating rankings page
        ./ComputeRankings.py > ComputeRankings.out
        ./Create_PGER.py > PaulGrahamEssaysRanking

MakeGraph outputs dot files for the giant component and the "other nodes" graph.

    echo Creating DOT file
        ./MakeGraph.py > links0.dot

Then I use neato to layout the graphs. In each case I run it a few hundred times and pick the result that has the largest boxes. That means it's the most compact output, and seems to be a good heuristic.

    echo Create Giant.png
        ./LayoutGraph Giant 11,40

    echo Create Other.png
        ./LayoutGraph Other   8,5

Finally I create the HTML you see using an HTML template.

    echo Create page with map
        ./Create_PGE.py > PaulGrahamEssays

There's a small lie in this. The web site is actually a statically generated "wiki". When it was first devised I didn't have the facility to run scripts on my host. I generate the pages as plain text with some mark-down, then off-line generate the entire site. Then I upload the parts that have changed.

If you try to edit then your suggested new version gets emailed to me, where it goes through an aggressive spam filter. Then it sits in my inbox for me to decide if it's a good change. If so, I trigger a refresh. Some people have passwords that trigger the refresh automatically, without my intervention, so it really does work as a limited access wiki.

Er, right. Is that what you wanted to know?

pclark• 16 years ago

What does "Google-style" mean? How are these numbers calculated and what do they mean?

nickolai• 16 years ago

He said he used PageRank algorithm

http://en.wikipedia.org/wiki/PageRank

It is what google ranking is said ot be based upon. though they keep their exact algorithm secret afaik.

In a nutshell, you give each page a base score, and have it give away part of its score to the pages it links to. Assuming that the page author wouldn't link to a page with poor/irrelevant content, you get some sort of quality/popularity score for each page.

evo_9• 16 years ago

Very cool.

Suggestion - please use a san-serif font.

sushi• 16 years ago

Interestingly there's no font-styling (or any css for that matter, he is using tables for christ's sake) on that webpage so the reason you are seeing Serif is because of your web browser more than anything else.

RiderOfGiraffes9• 16 years ago

He might mean within the graphs. I haven't played with asking neato to change the font being used, and I'm not sure what fonts I have available on my very, very elderly setup.

wahnfrieden• 16 years ago

It's too old school to expect people to have their own personal default style settings for their browsers. There are barely any sites these days that don't define their own styles.

It's actually a neat idea of the original web, but it's of little practical value today.

RiderOfGiraffes9• 16 years ago

Consider:

http://www.solipsys.co.uk/Giant.png

http://www.solipsys.co.uk/Other.png

These will change as I experiment. Comments welcome.

jrockway2• 16 years ago

I like. You see pg's popular essays all the time and get a little tired of them. Beating the averages, yeah whatever... hackers and painters, uh huh. But the less popular ones are really good, I especially like "The Submarine".

(Maybe because it is my hobby to push back against dress codes. One time, I was visiting my company's Singapore office. "You can't wear tennis shoes here, it's against the dress code!" "Do you know who I am?" "Nope." "Can you fire me?" "Nope." "Then what are you going to do about it?" :)

burgerbrain• 16 years ago

Aren't you mister social.

Protip: pissing off random people in your company for childish reasons is a piss-poor career move.

jrockway2• 16 years ago

Have you ever watched "Curb your enthusiasm"? I aspire to be Larry David.

PG's essays, ranked "Google-style"

Related Stories