Ask PG: Is HN Expired Link Eventual Fix Planned?

https://news.ycombinator.com/item?id=7048774

by baldajan • 13 years ago

96 118 13 years ago

news.ycombinator.com

I believe the expired link issue came from using a new lisp feature, but I'm curious if there is any plans to fix it?

It's a little annoying since I constantly pickup tabs that are a day old, and have to start from the beginning. Hopefully I'm not the only one on HN with that issue.

11thEarlOfMar• 13 years ago

No, you are not. It is annoying.

SamReidHughes• 13 years ago

What sort of data structure do you recommend they use to efficiently reconstruct older story orderings?

baldajan• 13 years ago

Well, I would trade off older story ordering for consistency. Or better yet, rather than return Expired Link, return the "current" story ordering.

recursive• 13 years ago

No one's asking for older story orderings. I'd prefer newer story orderings.

dsuth• 13 years ago

Much like every other paginated site with constantly updating content in existence. Just link to the next page with whatever the current content for that page is. People can figure out that items they're looking for get bumped from one page to another semi-frequently.

vl3• 13 years ago

You know, like return entire page instead of an arbitrary cut off?

michaelmior2• 13 years ago

What does "entire page" mean? The front page of HN could be nearly infinite in theory if you include all stories every posted. It has to be cut off at some point.

slashnull• 13 years ago

I was about to post about it.

I would definitely like to see this one sorted out.

makecheck• 13 years ago

Although this is a significant problem, it’s probably better to use the bottom-of-page links:

- “Bugs” (in this case, already seems to be known and talked about; see [1]).

- “Feature Requests” for anything that isn’t a bug.

[1] https://github.com/HackerNews/HN/issues/11

pchristensen• 13 years ago

They are continuations stored on the server (not a Lisp-specific feature). There is a limit on the number of continuations stored, which is why older ones come up as unknown or expired.

This has been there since Day 1 and will probably never be changed.

pg9• 13 years ago

Closures, not continuations.

elwell3• 13 years ago

Could you give a quick explanation or a link that explains how that is set up? (closures stored on server)

Just curious.

diminoten2• 13 years ago

This site isn't actively developed, and hasn't been, as far as I know, since it was established in 2007.

This is intentional. I forget the specific rationale, however.

So no, to answer your question (though I'm not pg), it likely won't be fixed.

vezzy-fnord• 13 years ago

I'm assuming the minimalism and hackish nature was deliberately chosen to weed out the lowest common denominator and cater only to the niche HN targets.

supersystem• 13 years ago

That can't be right? Last time I heard they were all about inclusion by being a gateway.

qbrass• 13 years ago

The rationale is that the value provided by fixing the problems is less than the cost of doing the work.

This particular problem isn't going to cause tons of people to leave, and fixing it isn't going to cause tons of people to join.

Unless someone does the work for free, or manages to get a large portion of the current users to quit over the issue, there's no reason to expect it to be changed.

pg9• 13 years ago

Actually we change things almost daily.

umanwizard• 13 years ago

Any recent examples? (Just curious, not trying to debate you)

diminoten2• 13 years ago

Can you explain where I got this impression, or am I completely out in left field?

ilaksh6• 13 years ago

They are not going to make improvements to Hacker News. They have proven that.

They don't have to fix it or make any improvements. They can run a very poorly performing site with no features and a horrible appearance, because they are entrenched, and they literally own quite a large portion of their userbase.

Unfortunately no matter how many people invent software that does the same thing as Hacker News but works much better and offer it up for free, nothing will be done.

The owner of the site has too much invested in its technology stack (Lisp) and its particular implementation and appearance. The improvements necessary probably require going back on some of those fundamentals, which is never going to happen, because they are too proud.

kranner3• 13 years ago

they literally own quite a large portion of their userbase.

Not literally.

ilaksh6• 13 years ago

That's debatable, but we are talking about a matter of degrees. Wage-based employment is a moderate improvement upon slavery by direct force. The relationship between investors and startup founders is a moderate improvement upon that, but still has quite a few similarities with traditional slavery.

consz• 13 years ago

Seriously, this bug has been around for years and people have constantly complained about it. The only reason they would have for not having fixed it at this point is they simply do not care.

read2• 13 years ago

It's dangerous to claim anything is the only reason something happened because it increases the odds you could be wrong.

they simply do not care

If it's negligence that you're implying, consider a more valuable alternative. The expiration link bug is starting to look like the HN test of one's ability to tell what's important to care for. Any hacker would want to develop this ability.

asveikau3• 13 years ago

I upvoted you because I like seeing people who go against the grain here, who hold combative viewpoints and are willing to be blunt about faults, regardless of whether or not I agree.

However, after seeing this:

... because they are entrenched, and they literally own quite a large portion of their userbase.
Unfortunately no matter how many people invent software that does the same thing as Hacker News but works much better and offer it up for free, nothing will be done.

... I have to say, if you are disappointed, have you thought about testing this theory? This website has not been around forever. It isn't the first of its kind either. I'm probably not the only one here who came to this site after having frequented a bunch of others that were essentially the same thing. So I don't expect it to be the last one, either. If someone has the interest and the skill set, I say, please do make this. Worst case you waste a few weekends or something.

ilaksh6• 13 years ago

There are a number of Hacker News clones, i.e. people got fed up and even went ahead and figured out how to mirror the Hacker News content. Far as I know very few if any gained any popularity, but its not because they weren't better.

kyro2• 13 years ago

people who go against the grain here, who hold combative viewpoints and are willing to be blunt

ie assholes

mercurial• 13 years ago

Going against the grain ( eg, the prevalent PG / Musk worship) is fine. Feelings of entitlement, less so.

grrowl• 13 years ago

I even suspect Hacker News isn't that fussed if most of its community moves elsewhere. The core community (that it was built for) will stay, and the postings will become more focussed on what the core community finds, reads, and is interested in.

arikrak2• 13 years ago

Its kinda like Craigslist, though not nearly as bad, since Hacker News gets the job done well enough.

(Also, they let people scrape them and create improved versions!)

Crito2• 13 years ago

I don't understand the appearance complaints. Must every site look like a unicorn vomited it out?

kbenson• 13 years ago

I never minded the appearance too much, but the usability bothered me. I month or two ago I tried some of the chrome extensions (specifically HackerNew[1]), and it's made a big difference. In-line replies, hover to see user stats, collapsible threads, etc. I'm not enamored with the infinite scroll, but it's not horrible.

It makes it look a little slicker too, but I truly couldn't care less about that.

[1] https://chrome.google.com/webstore/detail/hackernew/lgoghlnd...

panacea• 13 years ago

It's not a good mobile phone experience. It's fixed width <table> based.

jfoster• 13 years ago

Those are some very absolute claims. If you worded that as observations and not so absolutely (or provided some quotes to back up the claims), it would sound a lot more credible.

pg9• 13 years ago

The improvements necessary probably require going back on some of those fundamentals

Not at all. It's pretty trivial to replace code that uses dynamically generated links with code that doesn't.

chrisdotcode• 13 years ago

If it's that trivial, a lot of us would be happy for such a fix; is there any particular limitation as to why it hasn't been done?

ilaksh6• 13 years ago

https://github.com/arclanguage/anarki/blob/ab138be626728fbe1...

Can they change the 'lasts' parameter so that things don't expire as quickly?

Another idea.

     To be more sophisticated, instead of killing fnids, could first
    ; replace them with fns that tell the server it's harvesting too
    ; aggressively if they start to get called. But the right thing to
    ; do is estimate what the max no of fnids can be and set the harvest
    ; limit there-- beyond that the only solution is to buy more memory.

    (def harvest-fnids ((o n 50000)) ; was 20000

Can they change that number? Maybe that would help.

Another idea. Could you change it so that when they are harvested the functions are serialized to disk (maybe temporarily), and make fns a function or something so if it had been removed from memory it could be deserialized? It wouldn't need to be a serialization solution that would work for arbitrary closures, just something that would work for those cases, like timed-aform or whatever.

I have barely ever tried to use Lisp and I don't know much about this system so this may not make any sense.

Another idea. Use one of the Hacker News clones that people have built over the years that doesn't have this problem.

michaelmior2• 13 years ago

Part of this is that HN is much more than the technology, it's the community. Sure there may be some scaling issues, but building a HN is something many people here could do in an afternoon. Building the community is a much more complicated problem.

acchow• 13 years ago

Unfortunately no matter how many people invent software that does the same thing as Hacker News but works much better and offer it up for free, nothing will be done.

We should name this - how about "The Craigslist Problem"?

thekevan• 13 years ago

The word "fix" assumes it is a problem. You load a page and after a certain amount of time, the next page is expired.

I am not saying, "it's not a bug, it's a feature!" I'm just saying that's how it works.

jacquesm• 13 years ago

It is a problem.

vl3• 13 years ago

For practical solution: Install "Hacker News Enhancement Suite" Chrome extension, it solves this and godzillion other problems.

belluchan• 13 years ago

Wow that is so much nicer. Thank you for suggesting this!

sikhnerd• 13 years ago

I loved this extension, and is really the only thing I'm missing now that I've moved back to Firefox :(

pg9• 13 years ago

It's not a single thing. There are multiple places that use dynamically-generated links. They make the source very simple. I've gradually replaced them in the most common situations. Replacing more is not in my top 10 priorities for things to fix. The reason is that it's not really what users want. What people come here for is good stories and comments. I doubt we have ever lost a single user over expired links. Whereas stupid or uncivil comment threads have probably cost us lots of users, and indeed, some of the ones I'd least like to lose. So that's the sort of issue I spend time thinking about.

Incidentally, for those of you working on startups, this is a good example of why making something people want is not a simple matter of making what they say they want.

credo• 13 years ago

>The reason is that it's not really what users want. What people come here for is good stories and comments. I doubt we have ever lost a single user over expired links.

Yes, you probably haven't lost any users over expired links

However, I think you have inconvenienced a lot of users through the expired links. Sometimes, I think it is good to spend a little bit of effort on helping a large number of users.

I'm assuming among readers (who attempt to go past page-1), an overwhelming majority has hit this problem multiple times (and lost time because of this problem)

ghshephard• 13 years ago

I've been a user for 5+ years, for more hours a day than I would care to admit, and have never run into an expired-link problem that couldn't be resolved with a BackButton, copy+paste.

Honestly, it gives HN a bit of charm, and it's never ever really bothered me.

SiVal2• 13 years ago

Amen. What a nuisance. You can keep clicking the next page button to see what other stories there are until you actually read one, after which the next page button will fail, and you'll have to start over again. What's the point of even having more than about three pages? The only way to reach those pages is to keep clicking next while carefully avoiding the temptation to read anything along the way, lest you get sent back to Level 1.

I'm not buying the argument that if HN has good stories and comments people will keep coming back and if they come back, it means they want a next page button that fails. A doesn't imply B.

If frequent expiration is to remain a feature, would it be possible to have a preference setting allowing us to put the first four pages or so on our front page? Then we could read whatever looks most interesting among the first 120 items instead of the first 30 before we get that expired next button we supposedly don't mind.

001sky• 13 years ago

However, I think you have inconvenienced a lot of users...

'Inconvenienced' is broad word. Plenty of people eventually get the hang of driving a motorcycle with the front-brake in the "wrong" spot, and are no longer inconvenienced by that. But it is not truly dangerous or troublesome in the way that having the control pedals in a Car were to be put in the wrong place (eg swap the throttle and brake pedal location). So, in this case its more an inconvenience of having some learning curve; it's not the more dangerous or nefarious inconvenience of something that is truly troublesome, random, or wasteful.

zimmern• 13 years ago

The solution is ctrl-left click (in chrome) so articles open in a new tab in the background, queue up everything you want to read before reading the articles, simple.

pjungwir2• 13 years ago

I think you have inconvenienced a lot of users through the expired links.

I suspect that fixing expired links would improve HN's front-page story quality. I seem to recall patio11 or tptacek once saying if you really want to help out HN, you should sit on the New page and upvote stories. That's too much of a pain for me, but if I could get to pages 2-4 without friction, I'd see more non-front-page stories I could upvote. This would help good stories make it to the front page or stay there longer. So perhaps making things easier for users would also help HN. It may even make it easier for content-rich stories to compete against link-baity ones. What if fixing expired links kept HN from becoming like Reddit?

wpeterson• 13 years ago

I'm impressed at the closure based pagination as a novel way to solve a problem.

Unfortunately, the user experience is quite unpleasant. I hit expired pagination links several times a day. You guys can do better.

olivier1664• 13 years ago

I use http://hckrnews.com/ only to avoid this problem.

codex3• 13 years ago

Is there value to maintaining this site yourself? I agree that you shouldn't spend the time to fix this issue yourself, but couldn't a contractor fix it for pocket change, at a low cognitive burden to you? Hasn't this issue been surfaced repeatedly by users, wasting the time of a good many people (including you)? Doesn't it hurt your reputation as a thoughtful and detail oriented organization?

pg9• 13 years ago

I'm not the only person working on it anymore. For the past year or so we've also had Nick Sivo (kogir), who handles all the systems stuff. And we also have a moderator who is a hacker and has written a bunch of code for identifying voting rings, sockpuppets, and so forth.

alok-g2• 13 years ago

I would say that the site should actually be opened up to the whole community to add to it. Hackers may be able to help in ways never imagined before. Some examples:

1. Auto assign upvotes to new stories using machine learning algorithms that use older, already upvoted, stories for learning. This can fix one of the issues often encountered where good stories go ignored because too few people have time to screen new stories.

2. Instead of using ad hoc rules for ordering stories and comments, use machine learning engines again.

Some stories get flagged. These would also provide good data for supervised learning.

I understand some control would need to be exercised to prevent gaming of the system.

CamperBob23• 13 years ago

The reason is that it's not really what users want

Yeah, actually, it is. Seriously.

Nothing makes "hackers" look worse than a hack that doesn't work.

pg9• 13 years ago

The true hacker solves the right problem. Expired links are just the visible problem. The real problems are more subtle: users voting up fluff posts, nastiness in comment threads...

mproud• 13 years ago

Count me in as this being the most annoying issue for me. (Maybe that means you’re doing a lot of things right, but it’s still plenty irritating.) So much so as I have been using sites which mirror or display HN submissions because they don’t have this problem.

elwell3• 13 years ago

What considerations have been taken into account to dissuade making HN open source? (I know there are some copies of older versions of the source on GitHub) (Arc code could be rewritten in Clojure (don't kill me!)) (I'm assuming there is some concern that transparency would increase ability for spam or 'gaming' the system)

pg9• 13 years ago

The main reason I haven't published a new version lately is just that I've been too busy working on YC itself. It would take a day of work to strain out the bits of code we don't want to publish, and I don't have any spare days now.

bfe• 13 years ago

Every comment expressing a preference between lisps should be made with nested parens. Well played.

WWKong• 13 years ago

Even if the site had blinking text with glitter background the regulars would still come. Those very regulars click on the pagination and get inconvenienced frequently (also typing and comment and losing it). Since this seems like a highly visible problem and probably the only bug I encounter, I'm a little surprised that it is not in top 10.

chacham15• 13 years ago

While I agree with the overall sentiment, it is like optimizing a program. You profile the program and see that foo (lets relate this to the expired link problem) takes 5% of the cpu and bar (the upvoting problem) which takes 50% of the cpu. Naturally, youd want to optimize bar. However, if it takes 50 hours to optimize bar and 1 to optimize foo you're better off going with foo. This too is said with the caveat that 5% better is worthwhile: nobody cares if a 1 second operation is shortened to 950 milliseconds whereas shortened to .5 seconds is something more noteworthy.

alok-g2• 13 years ago

Here's a simple thing that has helped me avoid the issue (which otherwise is quite nagging):

Starting from the HN main page, I first open the discussion pages of all the stories I potentially want to read in new tabs (till I start encountering older stories that I have already gone through). This process is fast enough to not encounter expired links.

For each story now, I click the link leading to the story from the HN discussion page, and get back to the discussion page simply using the back button. When done, I simply close the tab.

bruceboughton• 13 years ago

Ironically, the only time I really run into this is when reading through bestcomments.

I'm often more interested in interesting discussions rather than the original links and bestcomments is a good way to jump right into these discussions.

It'd be great if bestcomments could me migrated away from the fnid mechanism, but I appreciate this is a side project and this is low priority.

turingbook• 13 years ago

Why not just integrating HNSearch? That is very convenient for all users.

btbuildem• 13 years ago

I think it falls under "Not important enough to fix even though everyone complains about it all the time" -- luckily there exist browser extensions that fix what authors of this site couldn't. (CTRL-F for said extensions)

codex3• 13 years ago

Visible signs of entropy considered harmful: http://en.m.wikipedia.org/wiki/Broken_windows_theory

jijojv• 13 years ago

From a UI perspective is it too much to ask that instead of showing a FU expired message, just client side redirect the user back to the homepage - should be trivial to implement.

On a desktop it doesn't save much due to bookmarks bar but on mobile it saves a lot of effort on starting over.

IvyMike• 13 years ago

I'd just like it if the "expire links page" said "Expired link, sorry. Here is a link to a fresh top page"

dserban• 13 years ago

The issue at hand never annoyed me too much. It can probably be fixed with a browser extension (I'm using HN Enhancement Suite, maybe contact the author and start a discussion?)

DanBC3• 13 years ago

It is a useful feature, not a bug.

People need to refresh the thread before they respond to ot, especially if it is a day old. This ensures they have an uptodate version of the thread and allows them ro see if anyone has already made their point.

People clicking next to read past the first page is good. Open any comment pages you want to read in new tabs, and refresh those tabs just before you comment.

I strongly agree that there are other problems with HN that need fixing.

dded• 13 years ago

One of the worst things about expiring links is that they punish you for taking the time to read articles and long discussions. If I'm just skimming through, there's no problem with expiring links.

wglb• 13 years ago

expired link issue came from using a new lisp feature

I don't think this is the case.

This fix would have a low priority on my list. I would rather have work done on improving the quality of comments and unfortunate high ranking of middlebrow comments.

Ask PG: Is HN Expired Link Eventual Fix Planned?

Related Stories