Ask PG: Would it be possible to do an A/B test on HN?

https://news.ycombinator.com/item?id=740983

by jacquesm • 17 years ago

46 42 17 years ago

news.ycombinator.com

One week (or even a day) with comment and article points hidden from view to see what that does for the quality of the site ?

I'm very curious about this because I suspect that the visibility of the points is what is starting to create a negative undercurrent.

Or maybe leave them on articles but drop them on comments...

prakash• 17 years ago

Sounds like an interesting experiment, but how do you measure quality?

pavel_lishin• 17 years ago

Subjectively, I'd wager.

rms• 17 years ago

You can't isolate the quality variable. The quality of the site ebbs and flows; there are good days and bad days. The quality on the days when points are off is more likely to be random than connected to whether or not the points are displayed.

mkuhn3• 17 years ago

Interesting idea, but:

You have to worry about measurability. How would you measure the quality of the site? And how would you do it objectively?

paraschopra3• 17 years ago

Total number of comments and clicks on the articles would be the metric. Time spent on site would be a great measure too.

mkuhn3• 17 years ago

I think that both those KPIs are quite a good idea, but they can alos lead to quite wrong measurements: What if this leads to people just clicking on every comment link to find out if there is a discussion or if there is a new post... etc.

It doesn't meassure the quality of the discussion and the submissions.

req26• 17 years ago

I don't think either of those are terribly good metrics.

Using two of my own submissions...

Compare Google's blog post on robots crawling news articles: http://news.ycombinator.com/item?id=708417 to an inflammatory post about Techcrunch: http://news.ycombinator.com/item?id=658308

One of those is 'quality', and the other has comments (and 'dead' status).

oldgregg• 17 years ago

What if you could view the down votes made by any particular user from their profile page? A little passive accountability...

Personally I think the comment voting system is working alright. The overall s/n is more of an issue for me-- it would be nice to see some kind of karma based throttle for posting new articles.

jacquesm9• 17 years ago

that's a good one!

There is a downside to that though. Plenty of people use the 'up' as "I agree" and 'down' as "I disagree". I know that's not the way they are intended to be used, but it opens the door to 'I don't like you' and 'I like you' votes as well.

Without having them split out like that you will then only be able to view those votes through your own perception of what was voted on.

This will eventually lead to meta-moderation (not necessarily a bad thing), that's a lot more work than a simple switch, which is one of the reasons I asked this, it's dead simple, takes 2 minutes to implement and we'll have results in a couple of days.

euroclydon• 17 years ago

onclick=confirm('Are you up-modding because you feel the comment added value to the discussion?')

tokenadult2• 17 years ago

Plenty of people use the 'up' as "I agree" and 'down' as "I disagree". I know that's not the way they are intended to be used

Darn, I wish I would have bookmarked the comment in which pg said it was okay to vote to indicate agreement or disagreement. SearchYC is my friend:

http://news.ycombinator.com/item?id=117171

(And gojomo was my friend too, as I first found his comment linking to pg's comment when I did my SearchYC search.)

rarestblog• 17 years ago

Actually, one of the biggest Russian IT social news sites (habrahabr) has this for quite a while - you can't see a rating of article until you vote (up, down or "zero", but you can't upvote or downvote if you vote "zero", you just get to see the rating). It's working pretty well ever since it was implemented.

On the contrary note - bash.org.ru, which is kind of a clone of bash.org, tested this idea, but then they got back to showing the rating beforehand.

raquo• 17 years ago

Yep, but bash.org.ru did not give you the option to just see the score without voting. Given how high the trash / not trash ratio is there, I think it's only reasonable that many people wanted to see a quote's score before reading it (and upvoted the quote to see it).

chanux• 17 years ago

This sounds really good. Like to see what kind of change this make on our voting habits.

anamax2• 17 years ago

Is it time to start treating posts about points like political posts?

jacquesm9• 17 years ago

Apologies for crossing the first rule of HN (The first rule of HN: do not talk about HN on HN) but there seems to be no other good spot to have a discussion about an idea like this.

What would you suggest I do instead ?

stonemetal• 17 years ago

I believe the feature request link at the bottom goes to an appropriate place to have such conversations.

youngian• 17 years ago

This is the first meta-discussion that hasn't started off on a sour note for me, because the OP is not just saying "Heyyyyy I hate it when people downvote boo hoo etc." He's offered his thoughts on the causes of downmodding tension and has put forward his idea of how to approach the problem. Sure, it's still meta-discussion, but at least it's productive.

anamax2• 17 years ago

I didn't criticise meta-discussion in general; I only commented on points discussions.

Hiding points is not a new proposal. It came up yesterday (under http://news.ycombinator.com/item?id=739969 ) and has probably come up before.

My success criteria would be whether hiding points results in fewer points discussions than exposing them.

kashif2• 17 years ago

http://news.ycombinator.com/item?id=169479

kashif2• 17 years ago

http://news.ycombinator.com/item?id=169662

req26• 17 years ago

Any 'solution' to the 'quality problem' that fails to acknowledge that some people are obviously voting up the (politics|techcrunch|zed|etc.) articles you perceive as low quality is going to fail.

You need to account that some people love the hyperbole that you always find in (politics|techcrunch|zed|etc.) articles, and some people don't. This suggests implementing 'cliques'. If you upvote a submission, you grow one vote closer to the clique of people who also upvoted that submission. People whose vote history match yours very well will very easily influence your front page, and the articles of a kind you rarely upvote will be less likely to cloud your front page.

(More feasibly, you could just add some 'coolfinders' manually, whose upvoted articles automatically jump to your page.)

seanlinmt• 17 years ago

How about having different categories?

req26• 17 years ago

Categories just invite meta posts of the 'not <category>' type. Articles that fall into multiple categories are another problem, and tags don't quite solve that one either. They also inorganically split the community and add an extra step to the voting or submission process.

jacquesm9• 17 years ago

I've sent you some mail about this. Very interesting indeed!

req26• 17 years ago

I haven't had time to look it over in full yet, but I will respond when I can.

Ask PG: Would it be possible to do an A/B test on HN?

Related Stories