Ask PG: Karma Data Dump

https://news.ycombinator.com/item?id=822333

by chasingsparks • 17 years ago

17 28 17 years ago

news.ycombinator.com

This might be outside the scope of the Ask HN tag (although it is not really a feature suggestion either), but can you provide an SQL dump of all users Karma (i.e. url, vote sums, and dates). I think there are many people who might be curious about the statistical properties.

yan• 17 years ago

I would be one of those people.

noodle• 17 years ago

this would probably be better titled "ask pg". he's probably the only one who can [reasonably] help you.

chasingsparks4• 17 years ago

edited.

jacquesm4• 17 years ago

For the life of me I can't find the sql connection in arc, would you help me find it ;) ?

http://ycombinator.com/arc/arc3.tar

Personally I think releasing that data is a bad idea. It might help people to game the system and it might inadvertently release more than intended, see:

http://arstechnica.com/tech-policy/news/2009/09/your-secrets...

chasingsparks4• 17 years ago

I am not sure the game-the-system argument is strong. I could write a scraper to get the same data I am requesting in about 20 minutes. I cannot speak for other members, but I personally would want a karma dump to see if I can improve the algorithm. It is already pretty robust against abuses, but I think it could be improved as far as ranking goes.

jacquesm4• 17 years ago

HN has pretty good bot blocking measures, a little while ago someone posted a less-then-optimal script to retrieve the users karma on the fly and almost all of the users of that script got blocked.

Datamining a body of data like that is likely to turn up all kinds of interesting tidbits, amongst others you could use it to identify the 30 or so 'secret' power users on HN, figure out when they're sleeping and so maximize your take on spammed links.

Even now it can take more than 2 hours before a spam link gets deleted, no need to make it worse.

Not everybody uses HN with good intentions, there are plenty of people that see this as just another place to dump their links.

kogir• 17 years ago

Why assume that SQL is involved at all? Hint: it's not.

jacquesm4• 17 years ago

That was the point.

chasingsparks4• 17 years ago

TSV, JSON, or whatever then -- it's the less relevant detail. I just want a dump and I consider it uncouth to scrape if someone is willing to freely provide a dump -- or at least, I would prefer permission.

adrinavarro• 17 years ago

    I think there are many people who might be curious about the statistical properties

I am also :)

deutronium• 17 years ago

I think this information would be really cool, maybe you could cluster people together depending on the content of their submissions, to find people whose submissions you could monitor.

chasingsparks4• 17 years ago

That was part of my reasoning. There are many karma algorithms that could be deployed in different contexts. If the HN readers found some good alternatives, maybe PG -- or whoever maintains the sites code -- could integrate it as a user preference. For example, I would prefer to read the comments of people who have a few very strong comments rather than people who have many average comments.

(Note: I am completely ignorant about HN's code -- I don't know how they cache or calculate anything).

zck3• 17 years ago

There's no database for HN. Everything is saved as text files. The save-votes function (line 133 of news.arc), which writes out the hash table of a user's votes to a file, is as follows:

(def save-votes (u) (save-table (votes* u) (+ votedir* u)))

So the data could be gotten as a zip, but not a SQL dump.

kqr2• 17 years ago

Interesting. If HN ever decides to go the database route, sqlite might be a good fit since it's relatively simple and everything is stored as a single file.

benhoyt• 17 years ago

PostgreSQL stores everything in flat files too (on Windows, in a directory like "C:\Program Files\PostgreSQL\8.1\data\base\100929"), and I'm guessing MySQL would too.

mahmud• 17 years ago

Are you nutts!? sqlite LOCKS the entire database for every update!

ShabbyDoo• 17 years ago

I am interested in the Parato-ization of up/downvotes. If one recorded all users who visited each news item and whether or not that user voted at least once, what would the distribution of voting percentages look like?

In the US, old people are a powerful voting block because they vote a lot. Do a small fraction of HN readers have a highly disproportionate effect on karma?

zkz• 17 years ago

By the way, what would happen if all karma was reset?

blasdel• 17 years ago

The front page would be empty until people started upvoting from the new page.

Comments would be ranked purely reverse-chronological until people started voting.

It'd be less disruptive to the front of the site than another revisiting of 'Erlang sure is interesting'. Past threads would slowly be mined for re-karma as they were re-submitted and found in google searches.

Ask PG: Karma Data Dump

Related Stories