Hacking Hacker News

http://joelgrus.com/2012/02/16/hacking-hacker-news/

by stickhandle • 12 years ago

132 50 12 years ago

minimaxir4• 12 years ago

FYI, the new Hacker News API allows easy programmatic access of story/comments and infinite chronological paging. You could download every Hacker News story in less than 2 hours without breaking the API request limit.

https://hn.algolia.com/api

jrussino• 12 years ago

"new Hacker News API" -> do you mean there's now an official API? I must have missed this news. Can you provide a link / additional info?

sillysaurus34• 12 years ago

the new Hacker News API allows easy programmatic access of story/comments and infinite chronological paging. You could download every Hacker News story in less than 2 hours without breaking the API request limit.

Which API? https://www.google.com/search?q=hacker+news+api

jared314• 12 years ago

With hnsearch.com being shutdown, I believe he is referring to: http://hn.algolia.com/api

deft• 12 years ago

This is old so what you're suggesting wouldn't be possible unless he updated it (which I don't think he has any reason to do).

e15ctr0n• 12 years ago

There's a list of all the apps that have been built based on this API: http://hn.algolia.com/cool_apps

pak• 12 years ago

Yup, I've switched over my Chrome extension [1] to use Algolia's API instead of HNSearch (which is shutting down), and so far, it seems to be working peachy.

[1] https://chrome.google.com/webstore/detail/hacker-news-sideba...

ehsanu1• 12 years ago

Was this there when you posted?:

    RATE LIMITS
    We are limiting the number of API requests from a single IP to 1000 per hour.
    If you or your application has been blacklisted and you think there has been
    an error, please contact us.

minimaxir4• 12 years ago

Yes, that was there, and the math is still correct.

You can query 1,000 stories per request, and do 1,000 requests per hour. That's 1M requests per hour. There are 1.26M Hacker News stories indexed by the API. :)

EDIT: Finished downloading all the entries (and can confirm that 1.26M is indeed all of them). Took 3 hours due to a conservative wait period between each request to make sure I stayed within the limits.

karangoeluw• 12 years ago

I have written an API for HN:

- Python module: https://github.com/karan/HackerNewsAPI

- REST API: https://github.com/karan/HNify

himal• 12 years ago

Github link: https://github.com/joelgrus/hackernews

AznHisoka• 12 years ago

It seems you prefer to read articles from:

- WashingtonPost

- BusinessWeek

- MarginalRevolution

- NY Times

[1] Based on BuzzSumo's social data:

http://app.buzzsumo.com/#/influencers?q=@joelgrus&type=influ... (Press View Links Shared, Analyze Links Tab]

joelgrus4• 12 years ago

Oh jeez, who submitted this again? I learned my lesson a couple of years ago, everyone hates this. :)

joelgrus4• 12 years ago

Also, FYI, I don't even use this anymore, these days I just read the HN frontpage. :)

tlarkworthy• 12 years ago

Its exactly the kind of thing I would build and abandon. So the maybe more interesting things is why you don't use it? I presume its a UI thing, or classifier is unreliable, or something?

I would love to hear why vanilla HN is better now.

gus_massa• 12 years ago

I found the previous discussion: https://news.ycombinator.com/item?id=3602407 (498 points, 737 days ago, 82 comments)

(Note: I usually prefer the "unfiltered" version, unless there is a very special new that covers the 90% of the front page.)

stickhandle• 12 years ago

Sorry. I stumbled on it and thought it was interesting.

joelthelion• 12 years ago

I implemented the same thing for reddit a while back, and learned the same lesson :)

However, I haven't completely given up on the idea. The point is that while limiting news to things you've liked in the past is a terrible idea, helping you find good articles isn't.

The question is, what constitutes a good news item? I think the key is that there are many answers to this question:

1. Something very specific to your interests (your classifier should be good at detecting this)

2. Something big that everyone should hear about (the HN frontpage is good for this)

3. Something about a new concept you've never heard about. I've implemented this by keeping track of popular words on reddit and giving a boost to words not seen in the past, with some success.

4. Probably others I haven't thought about.

I'd love to hear your thoughts on this.

rickdale• 12 years ago

This is totally cool and kudos to you. But be aware there are limits of personalized hacker news. I think Bill Maher put it best when ranting just last night about facebooks customized news feeds:

Newspapers may be old-fashioned, but here's what we're losing if you never see one. They are trying to tell you what's actually important, not just what's important to you. You may not read the whole paper, but you at least see headlines, making you aware that something's going on outside of your microtargeted world of fashion or music or Wiccans or zombies or whatever you're into.

Replace 'newspapers' with hacker news and you get the point.

https://www.youtube.com/watch?v=WohtmZDZCGM

ThomPete• 12 years ago

Newspapers are trying to show you what they think is actually important. But of course omnibus papers will never really tell you what is important only what is current.

sosborn• 12 years ago

Never?

hnriot• 12 years ago

total nonsense, the classifier is just pushed upstream, from what I want to see, to what some ad-motivated editor wants me to see.

newspapers reported stories to sell you something, be it papers, ads or someone's agenda, not because they believed we'd all be more rounded citizens.

ketralnis• 12 years ago

But in the case of filtering Hacker News you're taking that pre-filtered list (filtered by geeks and startup folk and whathaveyou) and filtering it even more.

Whether that bubble is a subset of some other bubble, it's still a bubble.

hayksaakian• 12 years ago

ideally you could train the data set from stories i've upvoted on HN

https://news.ycombinator.com/saved?id={{username}}

pflanze• 12 years ago

Previous discussion: https://news.ycombinator.com/item?id=3602407

piracyde25• 12 years ago

Wait, this is 2 years ago?

siculars• 12 years ago

Ideas this the first time around. Cool, but still have the same problem I had then. Confirmation bias.

seizethecheese• 12 years ago

From the first paragraph: "people vote [links] up or down." Um... can't links only be voted up?

sethaurus• 12 years ago

Past a certain karmic threshold, both are allowed.

ColinWright• 12 years ago

Are you sure? I have over 60k karma and still can't downvote links.

benaiah• 12 years ago

Nope, only comments. The only actions you can take on a link are upvoting or flagging it.

arnorhs• 12 years ago

If this is a solution to the "not enough links that I personally like" - kudos to the author. Nice to find a fun project to work on that will also solve a problem for them.

I personally despise recommendation / personalization algorithms of any kind. I still have never found one that's actually better than myself at distinguishing articles that I'd like to read, music that I want to listen to, tweets I'd like to see, etc.

When reading HN, I'm constantly surprised by links that would not normally be on my radar for things I'm interested in. I think personalization algos, in general, are good at filtering those away.

Since the author mentioned HN being too much of a firehose and this then also being a solution to the "too many links to keep up to date on" problem, the solution might be a bit simpler than the author suggested.

HN already has the "best" links at https://news.ycombinator.com/best

It's hard to find - it's in the 'Lists' section in the footer, but it's still there and I use it all the time, when I haven't been actively reading HN for a while.

ingend88• 12 years ago

Interesting. This will go into today's top5HN Newsletter. Signup at top5hn.launchrock.co

smoyer• 12 years ago

"The model can only get better with more training data, which requires me to judge whether I like stories or not. I do this occasionally when there’s nothing interesting on Facebook. Right now this is just the above command-line tool, but maybe I’ll come up with something better in the future."

If you let your program log into HN using your account, it should be able to tell which of the stories you've up-voted there. If you use that as the input to your classifier, as you read stories on HN, simply mark those that interest you by up-voting them.

I'm also curious to know whether the stories are weighted by age to account for changes in what you find interesting.

ninjakeyboard• 12 years ago

Cool story bro. You may want to check out digitalocean for hosting - their cheapest option is only $5 a month - about equivalent to the smaller $40/month aws option. It's very simple as well.

j2kun• 12 years ago

It seems there is a small but very strong subculture of HackerNews readers who enjoy reading and discussing mathematical things. I would love to have a separate feed of those stories (and then after I'm done I could browse the HN front page), and I have often thought about the possibility of writing a program to do that.

DataTau (the HN for data mining) seems to have failed, so I imagine a filter is the way to go rather than make a new website.

jt2190• 12 years ago

I think one of the challenges of hosting a "sub-HN" is that the hosting costs are hard to justify.

This raises the question: How does YC justify hosting costs? My completely-off-the-cuff-assumption-take-this-with-a-huge-grain-of-salt is that YC benefits by having a huge audience to make announcements to, like job postings at YC funded companies, various pg essays, or just investing in overall goodwill from the HN audience. Probably the most likely reason is to increase deal-flow to YCombinator itself, though.

icebraining2• 12 years ago

Why does it need to have some justification beyond being a fun hobby? HN is hosted on a single server, and it probably uses less than 1TB/month, so it's not that expensive for someone with a Bay Area tech salary, let alone the whole YC.

kra• 12 years ago

I just use the 50 or 100 point minimum feed in my reader, and skip articles that don't look interesting based on how much time I want to spend. Sometimes I only read articles if they're a day old and the first comment makes them look interesting.

Hacking Hacker News

Related Stories