Reinstate hystry's Hacker News

https://news.ycombinator.com/item?id=99372

by pchristensen • 19 years ago

18 33 19 years ago

news.ycombinator.com

hystry.com lets me read Hacker News comments in chronological order instead of having to go back to each story (http://hystry.com/newsyc/follow/). I found it invaluable and I was a lot more involved in the community when I could follow the conversations easier. It died a couple days ago, apparently its IP address was banned from crawling Hacker News.

pg, could you please let them back in? Please, pretty please?

EDIT: or at least change the Comments page to include link and title for the story each comment pertains to?

ojbyrne• 19 years ago

How is that different from the "comments" link? Other than you don't have to login.

pchristensen7• 19 years ago

The comments page shows all of the comments in order, but hystry shows the link to the article next to each comment, lets you filter out comments from specific stories, and view all the parents of a comment.

It's hard to get a feel for it now since the feed got shut down, but I could browse through the last 100 comments in just a few minutes, and quickly get a feel for which stories people were talking about.

pg7• 19 years ago

It was really pounding the server. Crawling for a search index is ok, but retrieving pages to dynamically update stuff generates a lot of requests.

If you can tell me what you think is missing from News, I can try to add it.

pchristensen7• 19 years ago

Until ojbyrne mentioned it, I had never really looked at the comments page. It's almost right, but it would be much easier to follow the conversation if each comment was tagged with the story it refers to. Right now the comment threads are disjointed from each other (that's the price of a chronological list) but they're also disjointed from their source article.

On hystry, it's 2 columns, with the left column being the title and link and the right being the comment. This makes it easier to mentally sort the comments into conversations. Like I said, most of it is already in the Comments page, which I now use instead of hystry. You might think of another way to do it that also works well.

Thanks, pg!

imsteve• 19 years ago

What would be great for kartik's app is a live "push" style feed of all the new content on hacker news (possibly including votes too). This way, the load on the site would be minimal.

This would also let a lot of other people scratch their own itch to try out outlandish new ideas. Few of us newsyc readers are even going to bother thinking up ideas for your site because we realize that there's a low chance of them being implemented. You just couldn't implement them all if you wanted to. If you provide this feed though, it would allow people to get the instant gratification of coding up their own ideas and, I think, a lot more innovation would occur.

I don't think you're trying to get rich by selling this site so there's not much to lose. Brand dilution likewise seems unlikely in this case. Any other downsides?

chengmi• 19 years ago

Why don't URLs in Ask YC posts appear as links? e.g. http://hystry.com/newsyc/follow/ in pchristensen's submission?

bootload2• 19 years ago

"... It was really pounding the server. Crawling for a search index is ok, but retrieving pages to dynamically update stuff generates a lot of requests. ..."

I hit the RSS feed every 15m and have so for months. I call the RSS file, extract the necessary data from this file, then re-hit user pages once [0] to get their contact info. Some of the things I looked at to try and reduce hits is to check some basic HTTP options [1]:

- 'Last-Modified/If-Modified-Since' to only take, if updated

- 'ETag/If-None-Match' checking admin defined time to reload

- 'gzip' to reduce size of downloads

But none of the above are implemented (probably for good reason - complicated & not really required) checking the HTTP spec implemented is pretty bare. I only found server status. I could also check the the robot.txt file, but like I said I only download the RSS file and user pages. A quick check for "robot.txt" reveals none.

So you could automate/enforce crawling rules using a combination of HTTP and/or robot.txt files. But this does require the client to correctly implement them. Having said that increasing the RSS feed from 25 to 30/35 entries to cover the front page [2] is about all I can really think of.

[0] Have to check this. Mainly get contact info & cache if found.

[1] The basics are found here: http://www.diveintopython.org/http_web_services/http_feature...

[2] Having only 25 entries means you might be tempted to grab the front page (and maybe the second page) instead of RSS because you miss the bottom 5 entries.

dpapathanasiou• 19 years ago

Are you sure YC is blocking hystry?

The hystry blog makes no mention of anything like that.

It might be that Kartik just stopped running it, for whatever reason.

Note that the last blog entry was back in November, so maybe he's just moved on to something else, e.g.

pchristensen7• 19 years ago

I found this out by emailing him yesterday. I thought it would be better if I made the case for reinstating it by telling pg the value it creates for me while using his site. Kartik had nothing to do with this public request.

henning• 19 years ago

Did hystry get permission from PG to crawl the server? Did they agree upon a certain rate at which hystry would access hacker news?

pchristensen7• 19 years ago

I'm not saying hystry has any right, I'm just saying it was valuable to me. Like I mentioned in my other comments, the Comments page does almost everything hystry does, it's just missing a little story context.

cratuki• 19 years ago

Should they need to? They should be free to scrape and represent, and ycom should be free to block them if they choose. When you pursue anti-rights relating to the ownership of information distribution you come to disgusting solutions like current copyright models.

akkartik4• 19 years ago

Wow, this is awesome. I've managed to make something at least somebody wants. It's a start.

I didn't see this until now because I can't seem to go back to reading Hacker News without the ability to bounce between my interface and the original.

I'm sure hystry'll be back once PG's given us a timeout :) And it's a well-deserved timeout. My crawler process seems to have gone rogue and I wasn't keeping an eye on it. Definitely my mistake.

Reinstate hystry's Hacker News

Related Stories