The Mueller Report, Searchable and Accessible on the Archive

https://blog.archive.org/2019/04/26/the-mueller-report-searchable-and-accessible-on-the-archive/

by sohkamyung • 7 years ago

156 83 7 years ago

blog.archive.org

The search function seems to be broken. I searched for instances of "Russian collusion" and waited for two years for the results and it came back with zero instances of Russian collusion found...

hyperdunc• 7 years ago

You're searching the wrong archive. Try the CNN or MSNBC archives.

rdtsc3• 7 years ago

Actually that's an interesting research idea. Calculate the total hours of airtime and/or news articles written. I bet it would be close to a continuous month if not more.

valine3• 7 years ago

The word collusion has no meaning in a legal sense. Try searching for obstruction of justice.

s_y_n_t_a_x4• 7 years ago

[self-censored]

siidooloo2• 7 years ago

Yeah,

A bunch of people in the White House deleted evidence and used encrypted communications. Not hypocritical at all.

The President probably committed obstruction multiple times, but Mueller refuses to Make a final judgment on that because they believe that is the job of congress.

And the campaign did try to benefit from the Russian influence campaign, and did try to obtain the stolen emails. They wanted Russias help. But there was no evidence of direct coordination (remember the deleted evidence).

This doesn’t seem like much of an ‘exoneration.’

colordrops4• 7 years ago

This Twitter talking point is ignoring the fact that the whole point of this effort was to investigate collusion with the Russians, regardless of any particular crime the investigated are accused of. The report's title is "Report on the Investigation into Russian Interference in the 2016 Presidential Election" no less.

dredmorbius• 7 years ago

'Russia" and "collusion" are both specifically, though independently, addressed.

badcede• 7 years ago

it came back with zero instances of Russian collusion found

Just like the Mueller Report itself!

muxator• 7 years ago

The government initially released the document in a PDF format which renders it like an image, impossible to search. When PDF files are uploaded to the Archive, we automatically run them through an Optical Character Recognition (OCR) process. This turns those images into text, making it much easier to move between sections and search for specific words or phrases. This allows journalists and the public to more easily parse through volumes of information contained within these massive documents.

Shouldn't an accessible, searchable and digital version of government documents be the default form in which they are released?

Why do citizens have to resort to the Internet Archive to see such a basic right fulfilled?

In my country this happens frequently, as well. Too often, documents that should be accessible and widely publicized are hidden somewhere on the internet with practically no incoming links and in a form which is the digital equivalent of a grainy fax.

Sometimes the intention of superficially fulfilling a transparency obligation, without actually communicating anything is palpable.

sohkamyung• 7 years ago

One possibly is because they want to make sure the redacted sections are really redacted. You can do a web search and discover that others have tried to redact sections of PDF documents, only to later discover that people can uncover the redacted sections via software tools.

Scanning a redacted document is the simplest way of making sure redacted sections stay redacted as there is no underlying information to reveal.

rossenberg792• 7 years ago

You nailed it. Aside from that anyone who is serious about search is going to run it through an OCR and get the raw text anyway. Why bother doing more work?

ALittleLight• 7 years ago

Releasing the plain text is all that's needed. I fail to see how redacting by overwriting the relevant text is at all reversible - e.g. replace every redacted word with five X's. Replace multiple redactions with ten X's. That actually seems more secure than what's currently done, where you can guess at the words by their length.

maxlybbert• 7 years ago

The government is legally required to create an accessible version of the report for disabled people, and the Justice Department already promised to do so. While I can understand not wanting to delay the report’s release because that version wasn’t available, (1) I don’t know what the law says about eventually fulfilling the requirement, and (2) the Justice Department actually has the original documents and doesn’t need OCR to create the accessible PDF, so I’m surprised they haven’t done so yet.

gizmo686• 7 years ago

Much of the policy around classified materials is designed to be idiot proof. I imagine the process for releasing the document was something along the lines of: go into secure room; print out redacted document; review redacted document; take redacted document to non-secure scanner; release scanned version.

Sending digital files across a security boundary is just a much more dangerous operation in general than sending paper. Given the prevelence and quality of OCR (particuarly when run on a good quality print, as was in this case), there is just not enough of a reason to take the risks of sending files across the boundary.

EDIT: Idiot proof is probably not a fair characterization either. The other part of security policies is to be genius proof. What if someone involved is trying to exfiltrate data. There is a lot more places for them to hide it if you give them a digital pathway out then if you make them take it out on pieces of paper that will be reviewed.

mbrumlow• 7 years ago

Shouldn't an accessible, searchable and digital version of government documents be the default form in which they are released?

The way it was done was the most sane way from a security standpoint. In my opinion it is the first time I have thought that somebody who knows what they are doing is actually on the case.

I say this because it is just too easy for you to leak out the redacted parts. People have tried, and failed. We don't know if what is under those black blots are really important or not, but if they were, and the leaking of such information could cause somebody harm -- then this is the way to do it.

blihp• 7 years ago

I suspect the reason they released a scanned version of the redacted document is probably a policy decision to avoid serious screw-ups.[1] Also, organizations of all sizes have known how to photocopy/scan documents for decades so it's a path of least resistance thing. The other thing is that those in charge of releasing this report had no interest in doing anything, including a simple OCR pass[2], on that scanned document that would help disseminate the information in this report faster... it's against their interests to do so.[3] So you're right, a more useful document should have been released. But for the previously mentioned reasons, it wasn't. As long as no critical mass of citizens demands digital competence from their government, their government will not offer it.

[1] There was a story in the last month or two about how someone screwed up redaction in an electronic document in a court case where they effectively only painted black bars over the text but the underlying text was in the release document.

[2] It wouldn't surprise me one bit if lawyers viewed an image-only PDF document dump as a digital form of the 'paper blizzard'. It really isn't (not even close) as there was already an OCR'd version of the document available within the hour of its release and some NLP analysis within the day.

[3] In the event that objections are raised, they can always throw up the 'we were moving as fast as we could to get it released and this was the best we could do given time constraints' defense. Most politicians and judges would likely buy that story since they don't tend to be technologically literate themselves.

rayiner2• 7 years ago

[2] It wouldn't surprise me one bit if lawyers viewed an image-only PDF document dump as a digital form of the 'paper blizzard'.

I highly doubt any federal litigator would think of having to OCR a document as being some sort of road block. Acrobat Pro (which has excellent OCR just a couple of clicks away) is universal in the field. Many federal courts will reject electronic filings if they’re not fully text searchable. It’s almost certain that someone just forgot to run the OCR after redaction.

rayiner2• 7 years ago

The government releases documents as PDFs, which is the gold standard for document exchange. Here, all that happened is that the OCR text was stripped when the underlying Word document was redacted, and someone forgot to re-run OCR on the post-redaction version. But it’s trivial to fix.

(The article’a belly-aching about the fact someone scanned in the document from paper is misplaced. I have definitely done the exact same thing, because a purely digital conversion keeps all sorts of metadata. Yeah there are ways to strip various layers, but printing to paper and scanning it back in is a sure fire way to strip the metadata and make sure that redactions “stick.”)

plebio• 7 years ago

The report on the Department of Justice website is already searchable. No one's been relying on the Archive to search it.

fxfan• 7 years ago

At this point the whole thing feels like Iraq 2.0 and I wonder if I'm getting suckered by media.

What only worries me as a progressively-disenchanted but firmly and forever a supporter of Bernie Sanders is that even sanders seems to have been wrong on this one unlike Iraq, which I would shout from rooftops that he was the only politician right about. I would (and still do) quote his 'What are you afraid of' video to all who would listen...

The question is just why?

rdtsc3• 7 years ago

I'd be interested in what Chomsky would have to say about it. Manufacturing Consent seems more relevant than ever.

colordrops4• 7 years ago

He's said several things about it, and has trashed the Russiagate conspiracy. He emphasizes both that the US interferes constantly in other elections, and also points out that Israeli collusion and interference in the US is orders of magnitude worse than Russian influence:

https://www.realclearpolitics.com/video/2019/04/01/noam_chom...

spaceflunky• 7 years ago

Chomsky already said "Russian collusion" was/is a complete joke

JeremyBanks• 7 years ago

What are you even trying to say

zaroth8• 7 years ago

For better or worse, you just can’t have this conversation on HN.

akhilcacharya• 7 years ago

It's not Iraq 2.0 if we saw repeated interference, all deeply researched and shown in the report!

duxup• 7 years ago

What was Sanders or anyone else wrong about?

ericd2• 7 years ago

Matt Taibbi wrote an interesting article that agrees with you on the Iraq 2.0 front: https://taibbi.substack.com/p/russiagate-is-wmd-times-a-mill...

I haven't read this one, but it seems to be in the same vein: https://www.rollingstone.com/politics/politics-features/russ...

asciibinary• 7 years ago

The second article is full of errors and false interpretations: https://twitter.com/emptywheel/status/1121229047760027648

You should read the report

yingw787• 7 years ago

I really like this OCR version! I did a search for "Manafort" and it not only tells you where in the docs the keyword pops up with nice location markers, but also a snippet of surrounding text to give the keyword context. I do wish there was a low-level cache of commonly used keywords so that some searches would be faster, but I don't know what archive.org's position is on tracking user behavior (probably not a fan).

justinjlynn• 7 years ago

One can't be forced to divulge data on users one does not record in the first place.

sm4rk0• 7 years ago

Only 7 views since April 22?

WheelsAtLarge• 7 years ago

Here are few other options:

Web based and searchable

https://www.nytimes.com/interactive/2019/04/18/us/politics/m...

Original document has been updated so it can be searched

https://qz.com/1601873/the-pdf-of-the-mueller-report-has-bee...

rayrrr• 7 years ago

Thanks to this OCR'ed version, I was able to put an NLP text summarizer of the document into production yesterday. Check it! https://news.ycombinator.com/item?id=19815506

The Mueller Report, Searchable and Accessible on the Archive

Related Stories