Show HN: Hacker News Comments

http://hncomments.nathancahill.com/

by nathancahill • 14 years ago

189 69 14 years ago

hncomments.nathancahill.com

Great contribution Nathan, looking forward to using this to embed comments after my next blog post. Can you let me know how the comment system works? Are you scraping HN?

nathancahill6• 14 years ago

Thanks! I'm respectfully scraping. I've rate limited all calls to YC's servers and am caching all comments on my side.

nickwoodhams6• 14 years ago

Seems respectable. Out of curiosity, why did none of the existing 3rd party API's suit your purpose?

mootothemax• 14 years ago

Wow, I really like this, great work!

I've previously thought that comment threads on blogs tend to die down fairly quickly, whereas discussions on HN tend to last a bit longer, so this is doubly cool.

I guess the only downside is the near-impossibility of letting people comment directly from the page without having to log in again.

nathancahill6• 14 years ago

Not a downside. I'm not allowing posting comments or upvotes through this system to maintain the integrity of the discussion on HN.

sgdesign• 14 years ago

I'd definitely use this on my blog (http://sachagreif.com) if it was officially supported (or at least tolerated) by HN somehow.

And the fact that people can only comment if they have an HN account makes it even better!

jacobr• 14 years ago

> Although many websites and blogs have a comment system, the quality of the discussion on Hacker News tends to be much better.

Wouldn't this risk changing that?

danmaz742• 14 years ago

Probably not, unless your blog has an organic audience that is bigger (or at least comparable) than that of HN homepage...

nathancahill6• 14 years ago

Blogs this big are unlikely to implement a third-party (and unsupported) discussion feed.

ooobo• 14 years ago

I would see that eventuating if this allowed commenting via the embedding blog, thankfully it doesn't. I don't have the same issue with this that others do, but I remember TheNextWeb getting a bit of slack from commenters for embedding HN comments (although that was mixed amongst other comment platforms, and is a little different)

API-wise, http://api.ihackernews.com/ , http://hndroidapi.appspot.com/ or http://www.hnsearch.com/api didn't suit? [edit] what am I thinking, an API doesn't need an API...

Oh, and simplicity of this seems pretty nice!

danmaz742• 14 years ago

Very, very interesting! So you already calculated that the cost of this will be insignificant for you even with millions of pageviews? Blogs have a long tail of visits... ;)

irrationalfab• 14 years ago

Why isn't this open source?

nickwoodhams6• 14 years ago

Why would you want open source for this? Do you want to spend the time to deploy your own instance just to have the exact same result?

sp3322• 14 years ago

To learn from it; so that more than one person can work on improving it; just in case I don't trust the submitter to serve code to my visitors.

muellerwolfram2• 14 years ago

yes i would. to implement the whole thing, instead of only this little jquery snippet shouldn't take much longer. but then i'm not dependent on someone elses server being up/fast.

progrock4• 14 years ago

You'd hope it was optimally done client side, and then hosting would be moot.

unimpressive2• 14 years ago

In my preferred model of Internet content, stuff like this would be available for every element without scraping.

Good job.

jerf• 14 years ago

The word for that is http://en.wikipedia.org/wiki/Transclusion , and it's interesting that it was a core component of Xanadu (see the Wikipedia link there).

unimpressive2• 14 years ago

Yeah. I know.

We also need a micropayment processor that can do sub-penny transactions.

EDIT: Actually, I would be fine with penny transactions considering that the current state of things with say, paypal; is 30+ cents per transaction.

beshrkayali• 14 years ago

This is very interesting! Great work.

swah3• 14 years ago

This comment could be an upvote to the original story :)

nilved• 14 years ago

Couldn't your comment be a downvote? ;)

sp3322• 14 years ago

Do you think people are going to wait for someone to submit each post to HN? I think this will encourage writers to submit every single story they write to HN, just to get a comments page.

nickwoodhams6• 14 years ago

That's a good question, but it's probably a very small percentage. And they can always add it later.

samjc• 14 years ago

Even if someone put every single story they write on HN, that's not a guarantee that it will get comments, since it might not get up-voted.

I think if I were to implement this on my own page, I would probably have two tabs that expand if clicked. Something like this:

Show [your site's name] comments | Show HN comments

And then for the HN comment tab, you would use the OP's awesome submission. Thanks, btw!

sktrdie• 14 years ago

This would make more sense if it was entirely client-side code towards an official JSONP endpoint, such as the official hacker news API.

mattacular• 14 years ago

Not saying this is necessarily true for Hacker News, but unfortunately a lot of sites don't have the time/resources to develop and then maintain a public API for the majority of their content. And mostly see an RSS or XML feed of their news/articles/stories as good enough.

Davertron• 14 years ago

How hard would it be to have your script look for the submission on HN by url instead of having to manually put in the id? Maybe it could use the one with the highest score if there's more than one?

nathancahill6• 14 years ago

Yes, that's the plan! A new API endpoint will use the referring page to capture the URL, then link it with the appropriate discussion on HN.

progrock4• 14 years ago

This idea is kind of alluring. But you can't really include discussions on your page that might be happening all over the Internet. Reddit etc.

This is where the browser should really be smart enough to do a search engine lookup, and propose smart links alongside the page you're reading. Or even summarise and merge discussions found elsewhere.

personlurking• 14 years ago

I'm going off topic a little but...

An element I really like from random blog posts I've seen off of HN is using the sidebars for additional, short commentary/details. Your comment reminded me of that, but sadly this might not work well if an article were all chopped up with tons of commentary (I'm thinking of used books where 'important' parts are highlighted and comments put beside them by a random reader, which I dislike). Well, then again, perhaps for people who like to dive into subjects and really digest an article, there could be a straight text version and a feature to see it marked up w/ smart commentary.

This also reminds me of a small 'movement' a year or two ago for citing in-article links at the bottom instead of linking to them throughout the article itself.

progrock4• 14 years ago

Does it really matter where the links are listed - in the article or at the end?

Opera has a side panel, one of which is the Links panel. That pulls links out of the page and lists them alongside. But this alone isn't that much use. The browser however could do something more interesting with them.

I think a lot of pages seem dreadfully wasteful by placing in the same old accessory content on their pages. Buttonitis etc. I don't think this necessarily should live 'inside' the 'page' at all. The content should remain dumb. The inbound links, and chat around the content augments it. And this could be summised somehow by browser tools / other services.

I understand the author's want for embedding HN comments on a page. There's a simple and elegant solution to the problem. Simply open up another browser window/tab and point it to the relevant HN comment page. To do so, the author could include a simple link in the page, leave it to the browser to infer, or just let the user discover it for themselves (pointers do help though.)

Why the obsession of stuffing our pages with content?

It also feels a little snooty doing something like this (embedding HN comments on your page that is.)

progrock4• 14 years ago

> but sadly this might not work well if an article were all chopped up with tons of commentary

I think this should be left to the domain of the browser. Create a tool for filtering and showing certain aspects.

quesera• 14 years ago

Scraping comments on HN hits the robots.txt wall rather quickly.

At a maximum 2 scrapes per minute, your cache will be 60 minutes old when you're watching 120 (single-page) comment threads. That might be tolerable. Now try 1200 or 12,000 comment threads.

You can do smart cache refreshes, but you will miss bursty activity. You can ignore articles older than X days, but you'll miss the "dupe, comments here:" resurgences.

HN gets about 150 posts per day that hit the top 60, at least briefly, as a point of reference.

This is a neat thing, but if it gets popular you will have a problem. On the plus side, it's self-limiting. As the usage gets higher, the experience will degrade and usage will decline. No intervention required by you. :)

I've wished for a policy-compliant way to cache comments in a timely manner, but there's just no way to stay current under the existing rules.

rcavezza• 14 years ago

What if you're scraping with ajax? Wouldn't each individual user's IP take the hit and not the domain's IP?

dchuk• 14 years ago

The ajax request looks to be getting proxied through this guy's server. You have to do something like that because of cross domain ajax issues

NathanKP• 14 years ago

You can't scrape with AJAX because of cross domain security restrictions.

One potential solution to obey robots.txt might be to spawn multiple small EC2 instances with different IP's and have them coordinate with each other to share the crawling without individually running over the limits. (This is also useful for scraping from sites that have rate limits)

jeromegn• 14 years ago

Perhaps an iframe would work best. With some injected styles and javascript (some that would make the iframe expand automatically à la Disqus), I think this would work nicely.

jere2• 14 years ago

>Although many websites and blogs have a comment system, the quality of the discussion on Hacker News tends to be much better.

Probably because your random internet stranger has never heard of HN.

It's a shame, for instance, that I was ever allowed to find this place.

seagreen3• 14 years ago

If that ever changes and HN is overrun (insert dramatic and sad Hans Zimmer score) what about starting a new HN just for programmers? I wouldn't be able to get in at the moment, but someday I'd hope to be able to.[1]

[1] I'm assuming you'd have some kind of rotating or individualized Project Euler style problems that you have to solve to get in, and that they would be pretty tough. It occurs to me that you could also have temporary student accounts for people who are just getting into the software world as well though.

jere2• 14 years ago

Clever. It's like CAPTCHA for programmers. It wouldn't be much more difficult than the current registration process.

mapleoin• 14 years ago

Passing a programming test is no guarantee for having anything interesting to contribute to an online community, just look at your company's internal IT mailing lists.

pdeuchler• 14 years ago

An invitation based system, while not perfect, would probably be the most effective method. Plus, the owner of the site could restrict or encourage growth by the amount of invitations allocated... allowing for easy management of unwanted lulls or spikes in user activity.

Invitation based communities are also very self-policing. If you invite someone and they happen to be an idiot then there is egg on your face as well.

sigkill2• 14 years ago

Iirc there's a certain site that requires you to solve a differential equation as captcha. The name escapes my mind.

mcav• 14 years ago

This has already happened. Someone should start something new, ideally a smaller, more focused crowd.

ertdfgcb• 14 years ago

By "get in" do you mean to comment or to simply read the comments? I'm all for a intelligence test (I personally would like a "actually RTFA test"), but I'm pretty sure that making that test required to even read the comments is a terrible idea.

morenoh149• 14 years ago

could try what metafilter does. It seems like a nice community.

joshaidan• 14 years ago

Here's a question, is it possible to put a Hacker News upvote button on your blog post? Or is that against the HN rules, or something that's frowned upon in the community?

sbashyal• 14 years ago

I created hnlike.com about a year ago that helps you to setup an upvote button for your blog. I have disabled the upvote feature as the community seemed worried. See the HN discussion on my story for details.

nathancahill6• 14 years ago

I'm not supporting upvoting or posting comments. I like HN how it is now (or more so a year ago).

danjessen• 14 years ago

looks good :)

swah3• 14 years ago

Upvoting gives the same result as "nice job!" comments like yours :)

mattjball• 14 years ago

Just a heads up, I tried to test out your inline "Give it a try below!" but it's not working right now. In the browser console: "GET http://hncomments.nathancahill.com/comments/4285019/html/ 500 (Internal Server Error)"

nathancahill6• 14 years ago

Thanks, there were a couple hiccups handling the large amount of traffic. Should be in working order now.

sirbrad• 14 years ago

Nice work, this is pretty cool! I've been working on a similar approach but using Github (mainly because I host my blog there using Jekyll). I'll try and get it released tonight :)

biot• 14 years ago

How do people opt-in to allow their comments to be shared on sites other than HN? Or is this strictly opt-out and, if so, what is the opt-out mechanism?

bherms• 14 years ago

I hope this doesn't further impact the quality of discussion on hacker news. It seems that every day I notice more and more people posting completely non-contributing crap in the comments that you'd expect from Reddit or something (one-liners, stupid jokes, etc). I go to Reddit for that, I come here for insight (which, granted, you can find a lot on Reddit also, but HN is supposed to be a bastion of intellectual thought and discussion).

Show HN: Hacker News Comments

Related Stories