Links between Paul Graham's essays - revisited

http://www.solipsys.co.uk/new/PaulGrahamEssays.html?YC_News

48 33 18 years ago

solipsys.co.uk

Eliezer2• 18 years ago

Is there a standard tool that does this? Especially for, shall we say, highly interconnected webpages? I need it for Overcoming Bias.

alexwg• 18 years ago

Yup, Graphviz: http://www.graphviz.org

jyothi3• 18 years ago

Graphviz only renders a graph which is represented as per dot format. There are other programs: dot, neato etc which can produce jpeg,png,gif... formats through command line tools.

I guess what he meant was tools which can do this for a particular site.

I can't think of any but as a previous poster said - a crawl of desired depth for all local pages, then feed the access log to "statviz" to generate a neat dot file along with the links.

jjs2• 18 years ago

Yes, it's called python.

Write a crawler that examines all local links, and builds a graph (it needn't be anything fancy; a collection of nodes and edges is enough, but you'll have to detect cycles when building the graph), and then print it out in graphviz .dot language.

Then you can feed it into neato [either through a command-line pipe, or construct one using Python's subprocess module], et voilà, instant pagemap! :)

You can also use some of the rendering modes and scrape position data for the rendered nodes, if you want to generate a clickable imagemap.

jey• 18 years ago

Here's a first cut implementation in ruby. Requires the "hpricot" gem. Outputs a dot file to stdout. http://gist.github.com/35841

Output from the above is at: http://jey.kottalam.net/tmp/obgraph.out

I tried rendering this with "dot -Tcmapx -oob.map -Tgif -oob.gif" but dot segfaulted after 70 minutes. The code as posted at gist only outputs nodes for articles written by Eliezer in an attempt to make the dot file a manageable size. Tips/fixes appreciated.

zack• 18 years ago

I am home for winter break and would be willing to write this for you in the name of science. Plus, it seems that you know Marcello Herreshoff, with whom I took Computer Science in high school. Email me the requirements for your tool to <zackster>:at:<gmail.com> please.

markessien• 18 years ago

A tool like this would be ideal in locating the moment when a person stops producing original thought, and starts repacking his old thoughts in new structures.

jyothi3• 18 years ago

One or two links to an old post would not deteriorate the quality or freshness of content. In fact a lot of times one might be interested to get broader idea of the author stream of thoughts, for the ones not interested thye just have to blind to hyperlinks.

May be it would become restructured content if there are too many outward references like the... "Undergraduate" post. But even that when you look into this post the hyperlinks are for keywords like "essays", "hacker", "computer science", what you "love", hack a "blub" on windows ..

jyothi3• 18 years ago

In your words what does this graph imply ? I would be great if PG himself can see something through this and respond.

My take: - The most referenced (high in-links) posts are more dear to PG or most popular/strong in his line of thought.

- The ones with high outgoing links at times might mean they are restructured thought as 'markessien' noted.

- For a personal blog like PG's does higher interlinking reduce readability or make the content richer ? I guess the reader chooses this.

pg3• 18 years ago

It measures some combination of how general an essay is, how early it was written (I usually add these links when I first write something, so most links are only backward in time), how much I like it, and sheer randomness (at least half the time I don't bother to add crosslinks).

Larger numbers of outgoing links probably don't imply recycling. If anything it's a sign I think an essay is good and will get lots of readers, so it's worth trying to spread that traffic around. (These predictions are often wrong, though. I find it practically impossible to guess which essays will get lots of traffic.)

RiderOfGiraffes7• 18 years ago

Is it not worth considering putting "forward" links into older essays, so that when you write an essay it gets referenced by other essays, even when they're older?

Perhaps you don't have time, but would you consider adding links that other people suggest?

It would be interesting to do to this what I do with my SiteMap, and colour nodes darker if they have more traffic.

http://www.solipsys.co.uk/new/SiteMapExpt.html?YC_News

gibsonf1• 18 years ago

I like the idea, but it would be more meaningful if you could link for semantic reasons - that would be a very nice tool, could even be used by authors who want to put a book together from various essays they've written, etc. Or help you focus on an area of writing you're interested in by the author. I mention this because looking at the graph you can see that things that are related are not necessarily linked.

RiderOfGiraffes7• 18 years ago

This would be really cool. Can you suggest a way, given two essays, to decide if, semantically, the first should point to the second? If an author could use a tool like that it would mean that links could be discovered rather than inserted by hand.

That would be valuable ... you've made me think ... I may have a way of doing something close enough.

Hmm.

zupatol2• 18 years ago

Latent semantic analysis can be used to measure the similarity of two documents.

http://en.wikipedia.org/wiki/Latent_semantic_analysis

I found examples in ruby and python in this blog, which is unfortunately down just at the moment http://blog.josephwilk.net/ruby/latent-semantic-analysis-in-...

aswanson• 18 years ago

In a way, there is a gross semantic linking in a Q/A type of way with the titles, e.g., Why do Startups Condense in America --> The Power of the Marginal. What Can Business Learn from Open Source -->How To Start A Startup + Inequality and Risk.

shaunxcode• 18 years ago

This is cool - led me to http://www.paulgraham.com/javacover.html in which I read "Historically, languages designed for other people to use have been bad" which makes me feel a little less insane for spending the morning working on my "pet language".

kylec• 18 years ago

I wonder if someone can use the PageRank algorithm or something similar to determine which essays are the most prominent. It's hard to tell with this graph which is the most connected essay - is it "Taste for Makers", "What You Can't Say", "Great Hackers", or something else? It would also be interesting to correlate this with the Google PageRank for these pages, and traffic data (if it's available).

RiderOfGiraffes7• 18 years ago

OK - done that ...

http://www.solipsys.co.uk/new/PaulGrahamEssaysRanking.html?Y...

michael_nielsen• 18 years ago

URLs are malformed. Otherwise, nice. When you say it's "similar" to PageRank on your webpage, what do you mean?

pg3• 18 years ago

Here are the ones that get the most traffic, with daily page views:

     1760 Stuff
      770 What You'll Wish You'd Known
      403 How to Start a Startup
      365 Why to Start a Startup in a Bad Economy
      349 Why Nerds are Unpopular
      294 How to Do what You Love
      274 Web 2.0
      247 Great Hackers
      207 How to Make Wealth
      187 Lies We Tell Kids

ph0rque• 18 years ago

"Stuff" is my sending the given link to my female relatives to dissuade them from, or to reduce their practice of, being packrats. I'm pretty sure there are other users out there that do the same...

DougBTX• 18 years ago

In PageRank order: http://www.google.com/search?q=site%3awww.paulgraham.com

Google has a good PageRank implementation, though it isn't open source.

RiderOfGiraffes7• 18 years ago

> In PageRank order: > http://www.google.com/search?q=site%3awww.paulgraham.com > Google has a good PageRank implementation, though it > isn't open source.

Yes, but doing what you've done here gives the entire site, whereas implementing it from the definition allows you to compute the actual values, as I did.

Of course, Google have twoke their algorithm so it's no longer exactly as they originally published, and they're not telling anyone what it actually does. Using the "clean" version is at least transparent.

paulgb• 18 years ago

Awesome!

One small thing, it looks like you included the RSS link from the bottom of the page in the list of essays.

RiderOfGiraffes7• 18 years ago

Re RSS: Hmm. Oh well, Wabi Sabi.

RiderOfGiraffes7• 18 years ago

Updated again to include new essays. Also, the RSS feed has been removed, and a couple of other spurious links, just 'cos I was there and could do it.

I'm still interested in additional links based on semantics.

Links between Paul Graham's essays - revisited

Related Stories