U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events

http://online.wsj.com/articles/u-s-intelligence-community-explores-more-rigorous-ways-to-forecast-events-1409937859

by petethomas • 12 years ago

39 16 12 years ago

online.wsj.com

grizzles• 12 years ago

I guess they haven't did their homework on how to detect precrime.

All they need to do is hire some Asberger-y women, put them in vats of liquid, and give them some psychedelics. Ask them to predict the future.

How hard is that? You might argue that my approach is unscientific, but I would argue that is at least as scientific as their forecasting approach.

rrggrr2• 12 years ago

Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Research, think tanks, etc. Often the difference between good analysis and bad is simply how successful the analyst is at getting their projections read by the 'right' people. Along the same lines, if policy makers could stop shopping for analysis that confirms their biases, much progress could finally be made in international and public policy.

pmorici• 12 years ago

I was thinking the same thing. It sounds like they just did a really good job of rooting out the people that were bad at the work and keeping the ones that were good around. In government I would be skeptical that they actually promote or fire people based on how good they are at the job.

jrapdx3• 12 years ago

It seems logical that keeping the people who did well at the task and dismissing those who didn't would be the way to find people with forecasting talent in a specific domain.

It's the same principle as evolutionary selection. If I want to have a stand of only unusual purple flowers whereas the usual ones are red, then save seeds of the purple ones and plant those seeds and keep selecting. Maybe it could be called a purple flower meritocracy.

It sure seems to be the case that many employers aren't distilling the most capable employees among those employed so an effective workforce doesn't evolve.

One thing that bugged me a little was the hantavirus prediction. Seems the incidence hadn't changed much over time, I wonder how hard it was to predict accurately?

jacques_chester3• 12 years ago

Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Research, think tanks, etc.

Actually, we do. Philip Tetlock, who is mentioned in the article, spent more than a decade collecting tens of thousands of predictions from geopolitical experts. He found that humans are terrible at making predictions about complex phenomena; typically we are stomped by simple linear models or probablistic random walks.

Some humans were slightly better at predicting outcomes, but compared to simple models, they still sucked.

It didn't matter what other variables were involved. Education, access to classified information, experience, seniority, nationality, profession ... none of them mattered. A Russian politics expert is about as likely to pick Putin's next move as a financial economist who reads the New York Times.

Tetlock's book, Expert Political Judgement, is comprehensive. I reviewed it here: http://chester.id.au/2012/07/29/review-expert-political-judg...

rrggrr2• 12 years ago

You don't understand how DC works. On a given issue analysis and projections are produced by numerous agencies. For example a Country desk at State may deliver an accurate estimate, but another group at state may water it down, or it may be stuffed by a competing estimate from CIA. Since its politically safer to bury your head in the sand than it is to fight for an estimate, the process is ripe for manipulation. All this concept does is pass the buck without - apparently - any comparison to all the analysis produced.

Tl;dr - This solution speaks to the failings of the intel process and not the people.

dminor• 12 years ago

Hah, I worked on this the first time around 10 years ago when it was called FutureMAP. Didn't exactly end well.

Bud• 12 years ago

I suggest that they hire Hari Seldon.

vdaniuk• 12 years ago

We've still a long way from Gaia, but internet + biological computing may take us to there faster than expected.

shawkinaw• 12 years ago

Reminds me a bit of Asimov's psychohistory [0].

[0] https://en.wikipedia.org/wiki/Psychohistory_(fictional)

empressplay• 12 years ago

Simply keeping those who successfully predict events over time will do nothing but end with one incredibly lucky person, who will then proceed to get their future predictions wrong most of the time, or at least as much as the group you started with.

jacques_chester3• 12 years ago

I think the competitions are structured to show the difference between distinction and calibration. Submitting outlier predictions constantly eventually gives you the former but not the latter.

The easiest prediction in the short term is "Yesterday's Weather". Most of the time, most things don't change much.

Teapot• 12 years ago

It becomes more complicated as they then act on those predictions. Thus mucking up said predictions. And by doing so requires new predictions (expensive guesses).

(I just watched DaysOfFuturePast)

U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events

Related Stories