The US Library of Congress has put 25M items free online

http://www.sciencealert.com/the-us-library-of-congress-just-put-25-million-records-online-free-of-charge

by leephillips • 9 years ago

210 40 9 years ago

What's the copyright? Would it be legal to unzip those and serve them directly, so archive.org or anyone else can make them more inviting for access?

I know you shouldn't look a gift horse in the mouth but there's not even an index or a rough idea what something like "Name Authorities" might mean. That's not what I call wide open doors, that more seems like doing some legally required minimum.

an27• 9 years ago

I suppose you could do that and then wait for the C&D notice?

I can't imagine you'd get sued for putting online a copy of this if you comply with C&D notices, then again I'm very narrow-minded.

Regarding Name Authorities, this article should clarify it somewhat: https://en.wikipedia.org/wiki/Authority_control

It's basically an authentication provider maintained by the Library of Congress, which serves to define cannonical identifiers for library-catalogued entities, like books and public figures.

The Library of Congress uses the MARC standard (developed internally) and that is the format of the Name Authorities files: https://en.wikipedia.org/wiki/MARC_standards

wonderous2• 9 years ago

Files are 25 million bibliographic index files which were produced by US Federal employees, so yes, they're likely in the public domain as a result.

https://en.m.wikipedia.org/wiki/Copyright_status_of_work_by_...

wordupmaking2• 9 years ago

Awesome, thanks!

pmoriarty2• 9 years ago

I wonder how much more extensive the release could have been were copyright laws not in the way.

Then there's the old question of whether the works under copyright today will ever go in to the public domain, or if their copyright will be extended forever by future changes in copyright law.

wonderous2• 9 years ago

Release is 25 million bibliographic index files and has nothing to do with copyright, since none of the data was ever covered by copyright protection.

brudgers• 9 years ago

https://loc.gov/collections

The Library of Congress has put materials online for a number of years. American Memory was the first I became aware of:

https://memory.loc.gov/ammem/index.html

According to Wikipedia, it began in 1994.

Mathnerd314• 9 years ago

Can someone change the title to match the article? s/items/records/

dogruck• 9 years ago

Another nit -- paid for via taxes.

alphonsegaston• 9 years ago

For those who haven't ever worked with MARC record data before, there's a python library that's a pretty easy interface called pymarc:

https://github.com/edsu/pymarc

lgierth5• 9 years ago

I added the raw .gz files to IPFS when Library of Congress announced this last week: https://github.com/ipfs/archives/issues/152

It's slightly more than 100GB and here it is: https://ipfs.io/ipfs/QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtH...

(Note that the filesizes in the directory listing are all wrong -- that's the original index.html from loc.gov/cds/downloads/MDSConnect/)

This makes it a lot easier to use this dataset at e.g. hackathons, where a lot of people would simultaneously pester that LoC server, which already seemed pretty bandwidth-limited on its own when I downloaded the files.

lgierth5• 9 years ago

You can pin it: `ipfs pin add QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

Or list it: `ipfs ls QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

Or copy it into your local filesystem `ipfs get QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

toomuchtodo5• 9 years ago

Did you push them into the Internet Archive yet? If not, going to grab a beer and start iterating through your IPFS objects.

lgierth5• 9 years ago

Go for it! :) I've never pushed anything to IA so far to be honest.

lgierth5• 9 years ago

Oops, it turns out I somehow ended up with decompressed files, that's why the filesizes are so far off. I'm creating a new fixed dump.

lgierth5• 9 years ago

The correct dataset (with gzip compression) is here: https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1...

themodelplumber• 9 years ago

Just in case anyone else is wondering: This is, as I understand it, 25M pieces of metadata, not 25M books, songs, movies, and treasures from the past.

gt_• 9 years ago

It's something along those lines. It looks like the music is all "cover songs" :( one of the heavier metadata types.

xvilka2• 9 years ago

Meanwhile Elsevier, who is widely known for inhibiting science progress by setting incredible high prices even for government funded research papers, makes a move against SciHub [1] and LibGen [2] again [3].

[1] https://sci-hub.cc/

[2] http://libgen.io/

[3] https://torrentfreak.com/elsevier-wants-15-million-piracy-da...

agumonkey• 9 years ago

I find scihub/libgen a very important project actually. Mirrors should be done (not an order, just a plan).

petepete• 9 years ago

I've never heard of libgen but it's blocked by my ISP (in the UK)

The US Library of Congress has put 25M items free online

Related Stories