I made a ruby library for accessing hacker news - check it out!

http://github.com/bolthar/ruby-hackernews

by random_guy • 16 years ago

10 3 16 years ago

It might be fun to scrape and index all HN stories; and perhaps create a way to download links/comments to a local machine automatically . There is a lot of really great information provided via this place.

random_guy• 16 years ago

The idea is interesting indeed - and easily doable as well: Entry.all gives you all the links on the front page, you'd just have fetch the main page between regular intervals of time (say, once every hour) and then save the entry information to a database.

Could you elaborate on this? If you can provide better "specifications" I'll implement this and add it to the library.

lukehasnoname2• 16 years ago

I imagine this:

-DOWNLOAD HN pages locally OR to another site

This is to make a permanent copy of the site and its discussions locally, so you'll have the knowledge of the articles and discussion forever.

Either download the linked site itself, or just the link, folowed by comments. Automatically grab comment updates once a day for a week, or whenever discussion statistically tapers off.

In other words, here are the options:

-Submission titles, dates, and links to HN page

This is to provide a searchable database for HN links, and perhaps (if this were done on a separate website) you could implement a "favorites" system for people to login and bookmark those topics of discussion they liked most, ala Server Fault.

Downside: I don't know how long HN articles persist. If they disappear, this is largely useless.

-Linked sites themselves, URL to HN page, and comments

Much more permanent; a complete mirror of source sites and HN discussions, searchable. Similar login/favorites iface could be put in place.

Downside: More information is cached, more complicated scraping (source sites and HN).

I made a ruby library for accessing hacker news - check it out!

Related Stories