History of Hacker News Search from 2007 to 2024

https://trieve.ai/history-of-hnsearch/

by skeptrune • 2 years ago

120 72 2 years ago

trieve.ai

cactusplant73742• 2 years ago

When is the launch?

skeptrune7• 2 years ago

Not sure yet! Fingers crossed it'll be this week or next.

cactusplant73742• 2 years ago

Cool!

swyx2• 2 years ago

i want to publicly thank Algolia for providing an excellent HN search for so long. i was on a call with Linus Lee and we both were referencing something on HN and i started pulling it up and he said "i know exactly what website you're on" without seeing my screen and it was of course the Algolia HN search. unbelievable mind sync.

Idk if it can be replaced (i guess i could do with semantic search + content crawling to start?), but even if it is replaced, Algolia will always have a special place in my heart for doing such a great job for free. thank you whoever worked on it (Algolians - is there a behind the scenes writeup somewhere?)

squeaky-clean• 2 years ago

It is free, but Algolia is a YCombinator backed company (YC W14) so for them it's probably very useful as a sort of low-stakes phase-1-prod environment. Basically a win-win-win.

davisr• 2 years ago

Now, if only it could be used without necessitating JavaScript...

MrVandemar• 2 years ago

Yeah, I always have to turn it on to search for something. It's a pain. It's friction.

Javascript is not required for search. It has never been required for search. Google. Yahoo. Alta Vista. They worked fine without it back in the day.

dankwizard• 2 years ago

Referencing something on HN and then being amazed youre both thinking about the HN search engine... Unbelievable mind sync.

spasibot4• 2 years ago

I wonder if they'll let us search flagged and dead posts and comments.

I remember reading some insightful exchanges back in the day that got flagged because of being a controversial topic that other users didn't like.

No way to find them now, even knowing some keywords and approximate month and year.

yorwba• 2 years ago

They're using the firebase API, https://github.com/devflowinc/trieve-hn-discovery/tree/main/... so no showdead, I think.

greenchair• 2 years ago

contributes to the echo chamber

dang6• 2 years ago

We had to exclude [dead] and eventually even just [flagged] posts from the public API because many third-party clients and sites were displaying them as if they were regular posts. For the ever-fragile HN ecosystem, that is catastrophic. We would get angry emails saying "how can your expletive site possibly condone such expletive expletive comments as <link>" ... and then it would turn out that <link> was a post by some account that had been banned for years.

It's fine if users turn 'showdead' on in their profile to read everything—just please remember that by doing that, you're subscribing to various bottoms of various barrels. But it's definitely not ok when people browse HN with some app we have nothing to do with, run into horrible things, understandably are outraged and then forever have their view of HN imprinted.

IMO this issue is existential for HN. We've spent years and so much energy trying to find a balance between internet openness and human decency, a task which oscillates between barely-possible and simply-doomed, so the idea that anybody anywhere sees anything labeled "Hacker News" that pours all the toxic waste back into the commons is physically painful to me. Much as I dislike the idea of restricting anyone's curiosity about the entire corpus of what gets posted, I don't see what choice we have.

spasibot4• 2 years ago

Thank you dang, I very much appreciate you taking the time to explain.

I was going to suggest maybe the public API could have a "showdead" flag too but I guess that too easily enables the problem you're trying to prevent? As in an enterprising app developer could turn the "showdead" tap to "yes" with every request and then the waste gushes out once more.

dredmorbius15• 2 years ago

I can appreciate that concern and see it even with flagging / dead / killed posts and submissions.

I've had my own concerns about HN's moderation, both excesses and insufficiencies. When I've done occasional polls about what people's issues are about HN I'm very often pointed to comments which now show as flagged. I'd found a few which hadn't been flagged and forwarded those to dang, who (admittedly long after the fact) flagged them. As dang's said many time, moderators don't see everything, most moderation is by members, and mods step in relatively rarely.

Based on Whaly's 2021 analysis and looking at dang's own comment post history (via Algolia), HN nets roughly 4 million comments/year and 400k submissions, with about 150k active members. Over his ten years as moderator dang's averaged about 20 comments per day, though there's a great deal more moderation occurring (some automated, some member-based, some manual but not noted with comments, which tend to be reserved for established accounts).

My read is that HN mostly tends toward its stated goals and, frankly, good-netizen behaviour. It does have a pronounced status-quo bias, though it seems to be self-aware on this point. I've a few further concerns I'm still thinking through.

The problem with an overly-open archive is that this makes possible misconstrued assertions about what HN does or doesn't tolerate. An open-access archive and third-party apps which don't reflect moderation actions, say, a third-party app which explicitly only showed flagged, killed, and/or dead posts, comments, and users, would paint a distinctly different picture of HN, and one which would greatly harm the reputation of the site.

There are some ... possible ways around this. HN uses sequentially-numbered IDs for posts and comments (both are treated the same so far as I can tell). UserIDs seem to have an internal representation which is similar (I've seen, for example, names which change over time), but the internal representation doesn't seem to be publicly exposed. If you want to find my own content you'd do it with "UserID=dredmorbius" and not by some numeric identifier.

But the numeric content ID means that a determined scraper could walk (sequentially or randomly) through the entire database, pull out every post and comment, and then glue those back together. That's somewhat north of 40 million items presently.... (There are benefits to using sparse, random / arbitrary UUIDs for systems.)

saagarjha• 2 years ago

I figured as such. Can I ask that you change the website itself to make them visible to logged out accounts? I understand exactly why you did this but I feel like if you showed them collapsed by default on the single-comment page and you have to actively click on a "banned" to expand them you really are out of line when you complain about how Hacker News hosts horrible content or whatever.

dredmorbius15• 2 years ago

It's possible to favourite such items. They'll not be visible unless you're logged in, but at least you can go through your fave list and find them.

Or, of course, bookmark them yourself for later reference.

I've run into this issue myself, though dang's reply (still being edited as I write this) does hit on some valid points.

zX41ZdbW2• 2 years ago

Here is the list of recent flagged/dead stories, ordered by rating, that they managed to accumulate: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a...

Here is the list of recent flagged/dead stories: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a...

yamrzou• 2 years ago

Is that playground open source?

olalonde• 2 years ago

I was looking at Algolia's website recently and it seems they really went all in with the "AI" marketing/SEO.

swyx2• 2 years ago

"we put the AI in AIgolia"

re• 2 years ago

I'm a little confused by the context and comments here. Is Trieve associated with HN at all or is this an independent/third-party offering? Is the Algolia search going anywhere? Will the search field on HN still take me to Algolia search or is that changing?

dang6• 2 years ago

It's independent/thirdparty, not officially associated with HN, though Trieve is YC-funded (https://www.ycombinator.com/companies/trieve) and in that sense there's an affiliation.

There's no current plan to change HN Search though I wouldn't rule it out. PG often used to integrate recent YC startups into HN in various ways (search as detailed by the OP, but also an SSO startup at one point, a carbon-reduction startup, I forget what else) as a way of giving them a boost and I could imagine us doing that again. (side note: I guess the mental model in my head is that earlier-stage startups are more closely bonded to YC and that as they succeed and expand, there's certainly still a friendly connection but the attachment becomes a bit weaker. For startups that have been around 10+ years, for example, I'm not sure it still makes sense to have frontpage job ads on HN.)

The change I'd really like to make to HN Search is to bring the front-end part of it into the HN codebase so the search results can be 'real' HN pages*. It's never been a priority to implement that though, and the Algolia system has been fabulous for a long time.

* Funnily enough, I made almost the same point using my pre-dang account back when Algolia was just getting started: https://news.ycombinator.com/item?id=7126635. Credit to pvg, who misses nothing, for spotting this. The reply was from Algolia cofounder ndessaigne who is now a group partner at YC! He was good on his word, btw.

neom• 2 years ago

It's YC-aside, anyway. Just from a general founder perspective: there comes a point where repeatability far outpaces any growth hacks you put in place perviously. In that vain, I think it shows exceedingly good sportsmanship towards founders to switch someone who has long established repeatable business to someone who has just found it. If you give someone a chance right as they hit escape velocity PMF, I suspect you provide a considerable tailwind, certainly a very kind thing to do for any founder if you're able.

ChrisArchitect• 2 years ago

Vote on Algolia vs. Trieve HN Dataset Blind Search Relevance Poll?

https://news.ycombinator.com/item?id=41172033

NKosmatos2• 2 years ago

Can we get an option to search for users/usernames? Or even better, searching for users based on their karma? ;-)

dredmorbius15• 2 years ago

User-specific search is possible in Algolia using "by:<username> <query>". So "by:dredmorbius privacy" will find my posts or comments on privacy.

dredmorbius15• 2 years ago

There's also the Leaders list which shows the 100 highest-karma members:

<https://news.ycombinator.com/leaders>

That's ... pretty constant over time, more so amongst the highest-ranked members than, say, positions 75--100.

Whaly did an analysis of the most-active HN participants in 2022, looking at 2021 data. That's also fairly constant.

<https://whaly.io/posts/top-10k-commenters-of-hacker-news-in-...>

Discussion: <https://news.ycombinator.com/item?id=29778994>

NKosmatos2• 2 years ago

I know, I’ve seen them both :-) As you correctly mentioned, the top100 rarely changes and thus has limited practical usage. I’d like to see something more dynamic and relevant for us plebeians, i.e. being able to sort all users by karma, comments, submissions, creation date.

dredmorbius15• 2 years ago

There are a few capabiliti es lacking from Algolia which I'd really like to see in a replacement:

- Negative search / exclusion: the ability to exclude terms from a search, as in "procfs -linux", which would look for any references to "procfs" which did not also reference "linux".

Edit: This exists, see dang's reply below.

- Replies to a specific user, e.g., "by:dredmorbius inreplyto:skeptrune <search terms>". I'm often looking for a specific context of my own previous comments.

- An improved date-bounding interface. If there's one thing that frustrates me about Algolia's interface, it's the GUI (and syntax) for defining dates. It's cumbersome, and at least on my browser, the dates are generally hard to read or invisible. Going back years is especially cumbersome.

I'll add: Algolia has been massively useful, and the fact that I can search HN, especially for my own content, has been a huge part of the value of the site, and is worlds ahead of other online platforms. (Mastodon / the Fediverse is catching up here, Diaspora*'s lack of search was among my main frustrations with the site and explains my absence there after more than a decade of participation.)

dang6• 2 years ago

Algolia does do search term exclusion. Compare https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... and https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu....

On the second point - HN has an undocumented endpoint https://news.ycombinator.com/replies?id=skeptrune&by=dredmor... but that doesn't give you search of course.

tptacek2• 2 years ago

whoah

dredmorbius15• 2 years ago

Search term exclusion: TIL! I swear I've tried that w/o luck before. Possibly was confounding that with OR pairing, usually given as "(termA|termB)". I'm pretty sure that doesn't work as expected (and just tested it). It's also annoyingly absent from DDG search, which is out of scope for here.

I'd been made aware of the undocumented endpoint but didn't want to spill the beans ;-) It's apparently expensive. OK to run manually, but don't script it.

niutech5• 2 years ago

Is it possible to add the alternative ("OR") operator? https://github.com/algolia/hn-search/issues/169

skeptrune7• 2 years ago

Replies to a specific user

Will definitely make sure to add that to our search before launch! That is a really good idea.

Improved date-bounding interface

Note taken :)

We also have negated terms which will work the same way.

dredmorbius15• 2 years ago

While you're taking notes ...

Comments to a posts from a specific domain might also be useful. HN exposes that in its story syntax and I'm pretty sure that's in the API as well.

tillulen• 2 years ago

I would like to sort comments by the level of the author’s expertise in whatever they are discussing. HN is a goldmine, but finding valuable knowledge within heated or elaborate discussions requires too much commitment to read through everything.

A weighted number of a comment’s upvotes is one signal. However, I can often tell when an author has deep knowledge or comprehensive experience with a subject just by reading their comment.

Do you think it might be possible to automate that kind of judgment?

skeaker• 2 years ago

As long as people are throwing suggestions your way, it might cool if we could sort by either Votes or Comments rather than just "popularity" which is seemingly just sorting by votes. A lot of posts will only have 30-50 votes but over a hundred comments which makes them hard to find even by searching.

simonw2• 2 years ago

On the topic of Hacker News search… a useful trick that not enough people know is that you can take the ID of a story and use search to return all comments ordered by most recent first - great for keeping up with what’s new in a specific conversation.

Eg for this thread the most recent comments can be found here: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I built an Observable notebook to save me from having to manually construct those searches here: https://observablehq.com/@simonw/hacker-news-homepage

pogue• 2 years ago

HN could really use some client software. If nothing else for choosing how to read thread replies (ie: most recent, most upvoted/popular, most downvoted, by replies from the OP, etc) + a more advanced built in search.

skeptrune7• 2 years ago

more advanced built-in search

Do you have any specific feature requests? I would love more suggestions and ideas!

dang6• 2 years ago

Since it's mention-undocumented-endpoints day, https://news.ycombinator.com/latest?id=$ID does that, where $ID can either be a story (in which case you get the entire thread sorted reverse-chronologically) or a comment (in which case you get the comments in that subthread).

So, for the current thread: https://news.ycombinator.com/latest?id=41228935

or for a subthread: https://news.ycombinator.com/latest?id=41229914.

simonw2• 2 years ago

Wow, I had no idea! Thanks for that.

skeptrune7• 2 years ago

Updated the blog at the link to include PG's HN post and the archive-available ycombinator.com post documenting the Octopart/ThriftDB search launch.

Commit here - https://github.com/devflowinc/trieve-website/commit/ab563475...

Links: - https://news.ycombinator.com/item?id=2619736

- https://web.archive.org/web/20110618105517/http://ycombinato...

krackers• 2 years ago

One thing I'd like is ability to search your (or others') favorites.

skeptrune7• 2 years ago

That's a really cool idea and also likely doable. Probably won't ship it before we release, but will add it to our backlog.

andirk• 2 years ago

The more features, the lamer the product. Keep HN clean!

niutech5• 2 years ago

I disagree. The more features the more useful the search engine. It doesn't make HN dirty, search engine is a separate project.

wisemang• 2 years ago

Putting this here with zero expectations but kinda hopeful there’s a workaround.

I like to keep my hands on my keyboard and I can `command-L` `hn` `return` my way to Algolia quickly from an open browser.

But why oh why doesn’t the search input have focus by default. And since it doesn’t why can’t I type `/` to get focus on the search input. I guess by now the three tab presses should be muscle memory for that but I’m so annoyed by that fact I refuse to internalize it.

Apologies for the random rant

walterbell• 2 years ago

Create a greasemonkey/userscript for existing browser extension?

Possibly useful example: https://github.com/imdj/HNRelevant

niutech5• 2 years ago

In Firefox, just add a bookmark for https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so..., edit it and add a keyword like "hn". Then type in the address bar: "hn whatever" and it will show you SERP for whatever.

Another way is to add the OpenSearch engine (https://mycroftproject.com/install.html?id=84751&basename=ha...) to Firefox and assign a shortcut key to it: https://support.mozilla.org/en-US/kb/assign-shortcuts-search...

downrightmike• 2 years ago

There's a search feature?

niutech5• 2 years ago

What I am missing from Algolia's Hacker News Search is the "OR" (alternative) operator, so that I can search for e.g. "WebKit OR Gecko OR Blink" in a single query. I hope Trieve will have such functionality!

History of Hacker News Search from 2007 to 2024

Related Stories