Inside the Secret World of the Data Crunchers Who Helped Obama Win

http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/print/

by sek • 14 years ago

285 133 14 years ago

swampland.time.com

plinkplonk• 14 years ago

Does anyone have any info about the tech stack?

spamizbad• 14 years ago

I don't know about their data scientists, but I've heard from people who know engineers on the campaign that they're essentially a Python shop. Not sure about the rest of the stack...

abecedarius• 14 years ago

A more interesting question, perhaps: what data were they mining?

patrickgzill3• 14 years ago

Hilarious!

What gave PA to Obama was the blocking by the courts of the Voter ID act that the legislature passed.

Note that in all the praise about individual contibutors in the article, there is no mention of "bundling" and the routine use of cutouts, both rather unsavory practices that both sides use.

EDIT: this site may prove a useful antidote to the above article: http://www.washingtonpost.com/wp-srv/special/politics/campai...

Romney raised more than Obama in total $$$. Even without the SuperPAC contributions that gave Romney more money, the two campaigns were within about 10% of each other in terms of what they raised.

sethg3• 14 years ago

Obama took PA by five percentage points, which is consistent with his margin in the state-level polls that were taken throughout October. In absolute numbers rather than percentages, his margin is about 300,000 votes. Are you really trying to argue that a voter ID law would have taken enough people out of the electorate to flip the state?

patrickgzill3• 14 years ago

Yes. Fraud in Philly voting is legendary and almost on par with Chicago.

I can't find the registered voter turnout for 2012 yet, but in past years Philly RV turnout has been as high as 108%.

brown9-25• 14 years ago

I'm confused on how the point about fundraising is relevant to your other points.

patrickgzill3• 14 years ago

Compare the hyperbole in the article, breathlessly talking about the geniuses on the campaign - then compare with the hard numbers compiled in the link I posted.

downandout2• 14 years ago

The gist of this article is that Obama was able to use a combination of data mining and creative marketing to pinpoint impressionable voters in crucial areas with a view toward manipulating the electoral college system to his advantage. This is yet another reason to toss the electoral system in the garbage and set it on fire. In no event should a relatively small subset of the population in a handful of states be responsible for choosing the President of the United States.

andrewcooke• 14 years ago

the electoral college system only constrains the location of targets. without it, you might make the problem harder, but you would not make it impossible.

btilly• 14 years ago

Not even that constrained. They had a lot of fundraising targeting as well, and that was nationwide.

jivatmanx4• 14 years ago

Had we used a popular voting system rather than electoral college system, this system would have been expanded to every state in the nation, and Obama would likely have won by a far wider margin than he did.

Again, this system was about identifying impressionable voters and marketing to them properly. The location of swing states is simple and public information, not proprietary.

downandout2• 14 years ago

"Had we used a popular voting system rather than electoral college system....Obama would likely have won by a far wider margin than he did."

That is quite debatable. There is no way of knowing how many people simply don't show up to the polls because they live in a state that leans heavily toward the opposition and therefore know their vote will have exactly zero effect on the election.

ihsw• 14 years ago

Even more important is that the system is in place -- already in production, tested, and battle-hardened -- and it is in the hands of Democrats.

The smart Republicans will take this to heart and turn their attention towards the 'quants and coders,' and the smart quants and coders will be looking to be employed by them.

brown9-25• 14 years ago

Why would you need a different voter targeting system/strategy for a popular vote? The goal is the same - convince people that would support you anyway but are not probable to go vote to actually get to the polls.

In fact, seems like this system worked just fine anyway - they won the popular vote as well.

apendleton• 14 years ago

Whether or not you would need a different strategy is debatable, but you would almost definitely see one. Voters in less populous states are over-represented in the EC, and urban voters are a key Democratic constituency, so you would definitely see messaging shifting away from things that would appeal to farmers in Iowa, since it would cease to matter (think things like mentions of ethanol as an energy source in stump speeches), and towards things that would drive up Latino turnout in Los Angeles: heavier discussion of immigration reform, for example, which very rarely came up in this cycle -- the foreign policy debate didn't mention drug violence in Mexico at all, this time, I don't think, and instead focused entirely on the middle east.

I think lots of "urban" issues would suddenly be much more relevant to the campaign, particularly on the left; gun control would be back on the table, marijuana legalization would start to be taken seriously as a social justice issue, etc. I'm partly talking out of my ass, but things would almost definitely change.

anigbrowl• 14 years ago

1. He would have won on the popular vote anyway. People who were excited early in the evening by Romney's apparent lead forgot to figure the west coast #s into their calculations.

2. You should throw your support behind one of the initiatives to link electoral college representation to the popular vote at the state level, because you're not going to get rid of the college itself for the foreseeable future.

3. Mitt Romney had exactly the same opportunities Obama did and knows exactly how the electoral college works. He just didn't use them. For example, the Obama campaign gambled and bought a lot of TV time early on when it was cheaper, so they got more bang for their buck. It's no good whining about one candidate manipulating the electoral college system, as if it had somehow been kept a secret from the other candidate. Romney was able to figure out the GOP primary system, which is at least as complex. Every student of American politics understands how the electoral college functions.

kemiller• 14 years ago

This presents elections as a system to be gamed, rather than an earnest attempt to learn the will of the people.

hackinthebochs3• 14 years ago

> In no event should a relatively small subset of the population in a handful of states be responsible for choosing the President of the United States.

I don't know why people love to say this; its simply not true. Just because we can predict which way certain regions will lean before the vote comes in does not imply that those regions don't matter. Does Nate Silver predicting 100% of the outcome of the election mean none of our votes mattered? Of course not.

The purpose of voting is not to "change the outcome election". The purpose is to participate in the process, thereby giving the process legitimacy.

snowwrestler3• 14 years ago

Obama won the popular vote, so what are you talking about?

soupboy• 14 years ago

http://www.slate.com/articles/news_and_politics/victory_lab/... - Slate article from Jan 2012 "Project Dreamcatcher How cutting-edge text analytics can help the Obama campaign determine voters’ hopes and fears."

brown9-25• 14 years ago

A similar profile from Mother Jones pre-election: http://www.motherjones.com/politics/2012/10/harper-reed-obam...

chockablock• 14 years ago

Thanks! This is much better than the article in the OP. It also links to a number of nice other pieces on the topic, including 2 Slate articles:

http://www.slate.com/articles/news_and_politics/victory_lab/...

ryanwaggoner8• 14 years ago

I find this really interesting, but I'm unclear on the proof that this made a big difference for Obama's campaign success. I'm sure Romney had people crunching the numbers as well, and he actually raised more money than Obama.

This was a pretty narrow victory, there are a lot of structural, ideological, and demographic biases at work here, and Obama did worse among most groups than he did in 2008.

Put another way, is there any evidence that Obama would have lost or even done much worse without this data crunching effort?

d23• 14 years ago

It may have been a narrow victory in the popular vote, but it was an absolute stomp in the electoral college. Obviously we can't go back in time and try it a different way so you're never going to have any "proof", but there was clearly a strategic advantage by the Obama camp in the battleground states.

ajross7• 14 years ago

There's at least some evidence that, in fact, the Romney campaign fooled itself into believing the "unskewed" polling models which simply (and incorrectly) disbelieved the demographic makeup of the polls. So yeah, it's hard to believe the republican campaign could have run a heavily data-driven campaign.

And the evidence that Obama would have actually lost is obviously not going to turn up. But you can look at the relative performance in "swing" states vs. the general electorate for hints. He clearly outperformed among the demographics that ended up deciding the election, as witnessed by the very large electoral victory on the basis of a very narrow (currently <2% I believe) popular vote margin.

ryanwaggoner8• 14 years ago

Sorry, what is that evidence? The fact that they were saying they thought they would win? The fact that they didn't give up? Did you really expect them to? It's important not to confuse the public behavior of a campaign with their internal beliefs. And Romney really could have won this, had things gone a little differently.

enraged_camel• 14 years ago

>>This was a pretty narrow victory

What exactly is "pretty narrow" about a 303-206 electoral split?

protomyth• 14 years ago

I think he meant population-wise which doesn't reflect in the electoral count except in analyzing effects of 3rd parties on results.

jobu• 14 years ago

"I'm sure Romney had people crunching the numbers as well, and he actually raised more money than Obama."

Actually if you look at direct contributions, Obama raised nearly double what Romney brought in: http://www.opensecrets.org/pres12/index.php

SuperPACs allowed Romney to compete financially, but I don't think the number crunchers they talked about in the Time article were involved in the PACs

ZachPruckowski2• 14 years ago

There is no way to disaggregate all the hundreds of factors that went into the 2012 election. So we'll probably never be able to say "X% of Obama's victory came from data mining and Y% came from hiring an addition Z Field Organizers".

But given the disadvantages Obama had going in, he definitely performed towards the top-end of what could be expected.

brown9-25• 14 years ago

I think you assume that all "number crunching" is equal.

ryanwaggoner8• 14 years ago

I absolutely do not, but how do you know which campaign had better number crunchers? You could go by who won, but you don't have a control; how do you know who would have won without the supposed advantage of the supposedly superior number crunchers behind the winning campaign?

sethg3• 14 years ago

The Romney campaign’s equivalent was apparently some system called “Orca”, which involved volunteers on the ground sending information to a central location about the conditions that they saw. It wasn’t as extensive, and now we know that it gave misleading information to the campaign.

Two other things to keep in mind:

(1) The Republican leadership has been really really invested in the “fake it ’till you make it” school of persuasion: if you act like a winner, this will persuade the people who want to get behind a winner, and then you will win.

(2) Democratic operatives have known for decades that the demographic groups more likely to vote for them are also the groups that are less likely to vote at all. The Clinton-era solution to this problem was to pitch a message that was more likely to resonate with the population that was already voting; the Obama-era solution is to get more of those unlikely voters into the voting booths. Republicans don’t have as much headroom to improve their turnout.

snowwrestler3• 14 years ago

There is substantial and consistent reporting that Tuesday night's results were a surprise/shock to many Republican candidates and the associated outside groups. We know that they did a lot of number crunching themselves, so the implication is that the quality of Obama's data operation was superior--in that their predictions more closely resembled the actual results.

lukethomas• 14 years ago

It's interesting that they mention email lists - I tried unsubscribing several times from the President's list, and it never worked. Anyone else notice this?

zeteo2• 14 years ago

I think polling on such a detailed level is fundamentally changing the democratic process. In the past, politicians had to be leaders - present a vision that was quite possibly unpopular to start with, and persuade people to come around to it. (See e.g. JFK and the space program.) Nowadays they're becoming expert hagglers: keep this mini-group barely content, say exactly the right nice things to the other, ignore a third altogether because they're 10% over in enemy territory. I don't know where this is leading to, but it's definitely a big change.

herge• 14 years ago

The oposite opinion can also be argued, that now politicians are a lot more connected to the populace, and that the will of the people is now represented much better than it was in the past.

rmk22• 14 years ago

How does a politician become "more connected" to the populace if said populace is only encountered as an integral function of a wider statistically significant effect?

"Dear Candidate, you will have to see people in that part of the country, not because you care, but because our data has shown that a similar visit before was statistically significant enough to matter."

On top of that, this data-driven approach actively undermines any connection by reducing it to a functional one. There isn't even the pretence anymore that the candidate cares for the given group due to intrinsic values or some sort of connection other than statistics.

makmanalp• 14 years ago

Correction: the will of those who are financially able to represent themselves in front of politicians. Presidents and leaders don't have time to understand their populace on their own accord these days.

ArbitraryLimits• 14 years ago

It seems to me that the whole point of having elected leaders instead of making every choice a referendum is to modulate the will of the people, instead of following it blindly.

cema• 14 years ago

Well, it seems like a big change. In the past, your observation (about haggling) was applicable to local elections; perhaps now it simply escalated to a larger, country-wide, scale?

ajross7• 14 years ago

Agreed; zeteo is looking at the past thorugh some decidedly colored glasses. That's never been the way it worked. Elections have always been won via "haggling", whether that happened in the proverbial smoke-filled rooms of party conventions. Remember that the expectation of a general election being "close" is a fairly recent one. From 1865-1932, for example, the president was effectively picked by the republican convention delegates.

Winning an election through careful attention to the polls simply means you're tailoring your messaging and policies to the opinions of the demographcs that matter most to victor. How is that not "democratic?".

If your complaint is that some demographics matter more than others, then address your criticism to the electoral college, not the strategy picked by the winning candidate.

runjake• 14 years ago

This is called 'science'. They used these techniques back in the days of JFK, too.

The difference is that now, campaigns have access to unprecedented amounts of audience data. But make no mistake, it was all dog and pony back then, too.

zeteo2• 14 years ago

> They used these techniques back in the days of JFK, too.

No, the contemporary approach is based on a series of tools, such as focus groups, which are a very new phenomenon in politics. The first national campaign where these figured prominently was 1992, I believe. (With e.g. "soccer moms" among the archetypal mini-groups.) Their sophistication increased by leaps and bounds up to the level described in this article. And they are quite game-changing for the nature of politics.

dataminer• 14 years ago

I disagree, politics has always been about haggling, you cannot do politics if you don't know how to build a coalition. A politician cannot represent 100% of the people that is why there are political parties and politicians of different ideologies. I think if you look a little deeper all the JFKs, Reagans and Clintons were hagglers and negotiators first. A good politician is an expert negotiator and the very best are exceptional leaders. Its been the same since the first democracy was born. The only change now is that it is very easy to identify who will part of your coalition.

stevesearer2• 14 years ago

The vast majority of the population are influenced by some sort of benefits the government "offers" whether they be tax breaks, healthcare, welfare, gun rights, subsidies, higher pay for teachers, etc...

Because of this I think that this sort of micro-haggling can happen. Instead of offering a broad plan that is ideologically consistent, and then trying to convince the population to go along with that - politicians can turn the knobs of the different constituencies by holding out the carrot that matters to that group of people.

ryguytilidie• 14 years ago

Ronald Reagan's electoral graph would like to disagree.

jbooth8• 14 years ago

Whether you like or dislike Obama, the single biggest difference between him and Romney is leadership the way you described it -- willingness to stand behind something unpopular.

Winning requires all kinds. Sketch out the broad vision, and then do the little data-driven things that can give you a bunch of tiny little edges on the margin. But if it's not close to begin with, no amount of small edges are gonna help you.

A good analogy would be having good architecture and good code quality in a project -- you need both, but no amount of code quality can save a bad architecture.

mseebach4• 14 years ago

> Whether you like or dislike Obama, the single biggest difference between him and Romney is leadership the way you described it -- willingness to stand behind something unpopular.

That doesn't sound completely right to me. Romney's stint at Bain was expected to cost him the presidency (and maybe did) because of the unpopular things he did while there: Fired people and outsourced jobs while making a couple hundred million for himself in the process. Of the many things he's flop-flopped and waffled on, that record isn't one of them. If running for president in a down economy with that around your neck isn't standing behind something unpopular, what is?

firefoxman1• 14 years ago

Just like any new technology when it's introduced, the value of this micro-targeting strategy will be overstated at first. But over time people will realise human nature has stayed the same, and this technology will find its place as just another tool in a politician's utility belt.

Perhaps we've already seen it misused on Romney's part: depending on where in the country he was speaking (say, southwest vs. rustbelt) he would take a different tone on the topic of immigration. In the age of instant news, flipflopping like that in the name of "targeting" your audience can have major consequences.

dkarl• 14 years ago

That's how politics has always worked at the city scale and lower. Now it's happening on the national scale, using polling and computers to gather and analyze the data, instead of using party members to gather data face-to-face and a party organization to collate and understand it.

Groxx• 14 years ago

I think you underestimate the sophistication of techniques of the past. They may have had less raw data, but what makes you think they didn't know precisely who they were talking to?

CrankyPants• 14 years ago

This may be myopic or off-base, but I feel like JFK had two big advantages with the space program that helped make it happen:

1. His constituents were terrified of the Soviet space program dropping something nasty on their heads, and didn't want to be behind the curve in that arena. 2. His assassination effectively canonized the program. (I think he'd argue my terming that an "advantage.") Cutting it after his death would have been political suicide. Where space exploration often has the same inherent challenge as nuclear power–specifically that the length of any serious endeavor will span several presidents, congresses, senates, to say nothing of economic cycles, (see CxP)–the Apollo program, despite being 2.2% of all federal outlays, had broad support.

There were numerous other factors, but I think those two played a big part in its success.

RyanMcGreal• 14 years ago

Interesting.

I somehow got added to the Obama campaign mailing list, and their recent fundraising emails politely noted that their records did not show me having donated any money (I'm not American and don't live in the US). They also said that their records might not reflect my donation if I had made it through a different channel.

However, this article suggests that all the channels go to the same database. I wonder whether one of the following scenarios is correct:

* The database has eventual consistency (maybe my donation through channel X would take some time to be reflected by the campaign team);

* The message was a ruse to throw off independent and Republican analysts to the sophistication of the database;

* It was CYA in case something had gone wrong.

brudgers• 14 years ago

Just for clarity, political contributions directly to a political campaign are a matter of public record in the US. Thus this piece of knowledge was not the result of new data mining techniques, it's been used for a number of years.

snowwrestler3• 14 years ago

They are only matters of public record if the total donation amount exceeds a limit, I think $200.

jychang• 14 years ago

CYA is most likely, although #2 is possible. Changing names (due to marriage, for example), moving, losing (and thus getting a new SSN) are all possible, and might screw up the database. A small CYA phrase helps a lot.

binarray2000• 14 years ago

Fascinating stuff... and something I want to learn more about.

Can you please recommend me books/software applications/online courses for doing the work these guys were doing for Obama campaign?

(Just to clarify: I am a developer and I grok SQL. So that part is covered. Other parts... less so.)

peteforde2• 14 years ago

I recommend Collective Intelligence, published by O'Reilly.

If you look online there's Ruby source code examples to go with the Python in the book.

clarkmoody• 14 years ago

What this article seems to show is that the election was not about political ideology. It was about getting out the vote. The Obama campaign fine-tuned its message to scientifically increase returns in those targeted groups. The Romney campaign played the 'smoke-filled room' game (for all we can tell).

In none of the presidential debates did either man speak in depth about his governing philosophy (or the Fed for that matter). Their talking points rang loud and clear over and over, and finger-pointing abounded, but neither spoke of his core convictions. The campaign speeches were much of the same. For all of the complaints about Romney's lack of details in his economic plan, the President didn't do any better, simply offering a 'stay the course' message (he learned something from Bush).

Therefore, given this article and the lack of ideology involved in the campaigns, I conclude that the voters were not persuaded by philosophy. They were targeted by the campaign so that the powers that be could keep their power.

Looking back to the electoral college results of 1980 and 1984 shows what a strong, clear ideological message can do. Those familiar with Reagan's speeches will know why.

So for those who say that America has embraced any certain ideology, I say that it has not, since ideology hasn't been on the ballot for a long time. People have embraced a man, and the philosophical debate has reached a new low.

hackinthebochs3• 14 years ago

I don't know what campaign you were watching, but I heard a lot about governing philosophy. Abundantly from the Democrats side, but also from the Republicans. Of course, they won't go into minute detail about policies on the national level; most people simply don't want to hear that. But it seems clear to me that Obama's political philosophy resonated with more people than Romney's.

forensic3• 14 years ago

ITT we confuse talking points and identity politics with PHILOSOPHY.

Jesus. You know the intellectual landscape is bleak when even hackers are willing to entertain the notion that democracy/mob rule has anything to do with philosophy, or that politicians operate in line with some sort of rational ethic other than straight forward power broking.

forensic3• 14 years ago

>I conclude that the voters were not persuaded by philosophy.

Very astute, Plato.

Welcome to planet Earth. Hope you enjoy your stay.

harshaw• 14 years ago

I'd sure like to know more about their methods and algorithms. What exactly "is" big data is glossed over so much that I can't tell if they are doing something unique here or simply applying a large degree of automation.

wilfra• 14 years ago

Too bad the single biggest piece of data here is going to be ignored. If they started competing with Silicon Valley for talent and put thousands of these guys to work in a building somewhere to fix this country, they'd quickly become the 'nuclear codes' of the USA. But they wont bother, they'll immediately forget all of this until the next campaign and go back to politics as usual.

f137• 14 years ago

"A politician thinks about the next elections — the statesman thinks about the next generations."

There are no leaders anymore.

azernik• 14 years ago

Nate Silver made an interesting point a while back when he compared the political data revolution to the Sabremetrics revolution in baseball.

In baseball, the journalists and pundits were way ahead of the professional operators in moving from gut calls to data-crunching, while in politics, it's the opposite - what Silver and other public number-based forecasters do is commonplace inside the campaigns.

andrewkkirk• 14 years ago

Want a future in campaign management?

Cut your chops with an e-commerce company. While there learn big data, how to test, and read into behavioral economics, and you're set.

fatbat• 14 years ago

Anyone have additional insights to what is meant here?

"...the campaign’s Quick Donate program, which allowed repeat giving online or via text message without having to re-enter credit-card information, gave about four times as much as other donors. So the program was expanded and incentivized."

- How did the Quick Donate program work? Like Amazon's 1-Click checkout?

- What was the call to action for Quick Donating again?

- How was the program expanded/incentivized?

Inside the Secret World of the Data Crunchers Who Helped Obama Win

Related Stories