Show HN: Search HN for interesting comment sections

https://www.searchhacker.news

by isoprophlex • 2 years ago

60 11 2 years ago

I built this tool to help me find interesting discussions on Hacker News. I love reading HN discussions almost more than the articles themselves. However, I found that full text search, although highly performant, is not always good at surfacing interesting discussions on a certain topic -- especially if you don't know what to search for exactly.

I built this by scraping the most recent ~6 million posts (that's about 2 years of history) and putting the resulting posts and their vector embeddings into Postgres.

Let me know what could be improved, and if you'd like a more detailed writeup of how this was built :)

humbleferret2• 2 years ago

Good work!

The sample searches were particularly strong for uncovering discussions on specific subjects, i.e. 'Dedicated Vector Databases'.

With less focused searches, such as 'Deep Work', I notice some results feel only loosely related. Is this an aim – to broaden the scope of discovery that may not seem directly related?

isoprophlex4• 2 years ago

Thanks for testing it out! So, because I've found (in purely vibe-based testing, of course) that with queries of 1, 2 or 3 words, a vector embedding doesn't always make a lot of sense. What's "Apple"? The fruit, the company? So, here it defaults to priming the fuzzy search with a full text search on 'apple', or 'flow state'. If you want to know something about deep work, try "tricks for performing deep work", or "help me get into a flow state".

It's all very serendipity-driven, and this is mostly a tool for wasting time, but you might discover some interesting (and hopefully on point) conversations this way.

Anyway, valuable feedback. Maybe in a next interation, I'll always do both searches and merge the resulting ranking.

humbleferret2• 2 years ago

Thank you for the detailed reply – I appreciate the insight into how the shorter queries are handled.

gus_massa• 2 years ago

Feature request: I expected that clicking on the number of comments would send me to the HN page like in the HN search. Just make all the second line clicky too.

isoprophlex4• 2 years ago

Consider it done :)

RGamma• 2 years ago

Ah so this is what we wrote about recently. Good job. Would be interested how you did it for sure :) It was really quick too!

Also, somehow this has a relatively big horizontal margin on mobile.

isoprophlex4• 2 years ago

Yes, exactly! Your feedback prompted me to do a quick iteration into something good enough for a Show HN, so thanks :)

I see what you mean about the margin on mobile... note that I'm one of the most unqualified front end people I know and this was mostly done using github copilot, I might not be brainy enough to understand the intricacies of css responsive layouting ;)

I'll post a writeup sometime too!

ryandrake• 2 years ago

Minor nit-pick: The browser back button doesn't seem able to navigate back to the landing page.

Interesting project, though. As a shortcut, I usually just use the hidden /active URL to find the interesting stories. Generally these are the ones with the most active commentary and often they get pushed off the main HN screen quickly by the "flamewar detector". So you have to be quick to find them, or just use /active.

isoprophlex4• 2 years ago

Yeah, good observation, that really annoyed me too. I tried to get HTMX to mess with the browser history, but that only seemed to make things worse...

Interesting aside on flamewar detection: you can visualize the comment embeddings in a 2d scatterplot, color coded by username. As you'd expect, a great deal of dang's comments occupy a distinct position in comment-space.

paulgerhardt• 2 years ago

I’d love to see the default examples seeded from the “best comments” section [1].

Three of examples on the current home page surface some toxic threads (“llm waifus”, “internet of shit”, and “AI Doomers”)… which while controversial aren’t as rich or insightful as say the comments from people that built Rust, the whole earth catalog, or the first x86 chip.

[1] https://news.ycombinator.com/bestcomments

HackRover• 2 years ago

Nice work. I also enjoy reading comments. Diverse comments can provide more information.

Show HN: Search HN for interesting comment sections

Related Stories