Using wget and grep to explore inconveniently organized federal data

https://gist.github.com/dannguyen/26e5922614dc22053745

by danso • 11 years ago

55 20 11 years ago

gist.github.com

adricnet2• 11 years ago

ProPublica has some great material on techniques for data journalism, including scraping and transforming unfriendly data formatting:

https://www.propublica.org/nerds

baldfat3• 11 years ago

The URL is nerds??? I look at the page and there was nothing on there that seems to show it as a acronym either. Just the title of the blog. Not sure I like that.

sp332• 11 years ago

It's what their News Apps team calls themselves. https://twitter.com/propubnerds

minikites• 11 years ago

I'm skeptical of a complaint like this from someone who voluntarily chose the handle "baldfat".

adricnet2• 11 years ago

I was looking for these articles when I posted earlier, very nice tutorials for EG that data you want that is only presented as a Flash applet:

https://www.propublica.org/nerds/item/doc-dollars-guides-col...

spectre2562• 11 years ago

This feels like a data set that might already be on the amazing https://enigma.io

otoburb• 11 years ago

Enigma's secure URL isn't working. The working address is http://enigma.io

spectre2562• 11 years ago

That's disappointing. I actually didn't even check that they used HTTPS, I figured it was safe to assume they do. Guess not.

danso• 11 years ago

Enigma is a great service and they might have a list of the Section 333 exemptions but even with the sophistication of their discovery algorithms, I'd be amazed if they had this as structured data...mostly because right now, it exists officially as a list of PDFs with some basic information, e.g. operator name and stated mission. Extracting location of operator probably wouldn't be too hard...detect if PDF follows standard letter format and then run address extraction algorithm on the first (or 5) last lines...

Obviously, someone from enigma.io could give some manual direction (such as deciding that city/state are actually important data points to extract), but it probably wouldn't be as detailed as dataset as what The Verge and Bard College have manually curated [1]

[1] https://github.com/voxmedia/data-projects/tree/master/verge-...

baldfat3• 11 years ago

http://openrefine.org

If you are not going to learn AWK, R, Python or Julia the next best thing is Open Refine.

OpenRefine (formerly Google Refine) is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; extending it with web services; and linking it to databases like Freebase. Please note that since October 2nd, 2012, Google is not actively supporting this project, which has now been rebranded to OpenRefine. Project development, documentation and promotion is now fully supported by volunteers. Find out more about the history of OpenRefine and how you can help the community.

kazinator• 11 years ago

TXR: http://nongnu.org/txr

ape4• 11 years ago

Its so sad there isn't a site with everything in plain sensible formats. data.gov seems like a good start. But a plain text UI (like gopher or ftp) seems better. A subfolder for each state. And a subfolder for each department. Maybe something like Linux's /proc for laws and federal data.

conductor• 11 years ago

This reminds me of Manning's case.

"That Manning was convicted of computer fraud seems to suggest that using wget on a U.S. government computer to download large numbers of files can be considered the digital equivalent of trespassing – even if it's on turf you're otherwise allowed to access."

http://www.washingtonpost.com/news/worldviews/wp/2013/07/30/...

morninj• 11 years ago

Well, sort of. In the Manning case, the government argued that wget was unauthorized on government-owned client machines. Their argument in that case wouldn't apply to the public's use of wget to send requests to government-owned servers.

samstave• 11 years ago

Then we need a scan of all government machines to see if there was a precedent of it being installed across many thousands of machines to suggest that this was not an actual individual breach of policy. and show that the government was remiss in its ability to enforce the policy at all.

engi_nerd• 11 years ago

Government information security and technology people, in my experience, are not necessarily forgiving or understanding in any way. And I mean "understanding" in both senses of that word. They neither care to understand your needs nor necessarily understand much about technology. I see so much extra work done just because the security person in charge of writing a procedure had no understanding of what the phrase "volatile memory" means. And that's just one instance from my work; I wish I could share more but it all makes me sad.

All this is a very long way of saying that some of these people may not have even understood what wget is used for, or believed it was unauthorized for use even though it is a standard tool.

csense• 11 years ago

One interesting thing the author mentions -- interesting despite being somewhat irrelevant to the topic -- is the astonishing fact that the vast majority of people don't know how to use Ctrl+F.

I submitted it here: https://news.ycombinator.com/item?id=10321439

melling• 11 years ago

Anyone know of a quick trick to extract the English text from html files?

dmckeon• 11 years ago

Try:

  lynx -nolist -dump

-dump dumps the formatted output of the default document or those specified on the command line to standard output.

-nolist disable the link list feature in dumps.

Using wget and grep to explore inconveniently organized federal data

Related Stories