Show HN: Beautiful Visualization of Agreement Groups in the US Senate

by friggeri • 14 years ago

95 39 14 years ago

friggeri10• 14 years ago

Op here, this is part of my PhD research. Please bare in mind that I'm French and that my knowledge of U.S politics is limited to what I can read online.

If you find any errors in my analysis or have any tips or other interesting pieces of information I'd be really happy to discuss with you.

toyg3• 14 years ago

> One unexplicable point remains in the data and that is the presence of 18 Democrats in the Republican group of the 108th Congress.

Dude, check the dates. Two words: Iraq invasion.

friggeri10• 14 years ago

I thought about that, but 12 of the 18 Democrats were fiercely opposed to the invasion…

msg• 14 years ago

Which sets of Senators were facing election that year? As a rule, only approximately a third. It would be nice to see if they were correlated to the group "switching" sides.

Which party controlled the Senate that year? It would be interesting to see if switching is correlated to party control. For instance, I see the Republicans controlled the Senate in the 108th (2003-2004).

friggeri10• 14 years ago

Thanks for the idea, I didn't think of the reelection aspects, I'll investigate.

anigbrowl• 14 years ago

It's very good, but suffers from one basic presentational error: You've placed Republicans on the left and Democrats on the right. While I know your algorithm doesn't attempt to plot people on a particular spectrum, it so happens that stereotypically partisan conservative or liberal senators tend to be positioned at the outer margins anyway. since liberals are traditionally thought of as left-leaning and conservatives as right-leaning [1], I would suggest your diagram needs to be flipped horizontally.

1. http://en.wikipedia.org/wiki/Left%E2%80%93right_politics#His...

Cushman2• 14 years ago

The visualization is totally cool, though I don't claim to understand the analysis.

...but if it helps, you made a little English error. Wouldn't even have mentioned if you hadn't said you were French :)

> [Republicans.] Most of which are either moderates or closer to the Democrats than to their own party.

> Olympia Snowe and Susan Collins are the two Republican Senators from Maine which are both regarded to be leading moderates within their party.

Should be "Most of whom..." and "...who are both...".

"Which" is only ever[1] used in relative clauses to describe inanimate objects (as you do correctly five other times in the article). Humans and animals are always either "who" (more personal) or "that" (more impersonal).

Note that's only in relative clauses-- "Which senator is yours?" is fine, but "The senator which represents me" means that I think my senator is a card table.[2]

[1] Outside of the odd dialect, of course.

[2] Not necessarily out of the question.

noahc• 14 years ago

I'll assume you're correct in your explanation. I've never heard it explained that way.

But as a native speaker "that" sounds slightly more correct than "which" because it seems less formal, however, if I heard a non-native speaker say "which" it wouldn't sound weird nor would it be something I would have corrected them on.

friggeri10• 14 years ago

You're perfectly right, I've reworked the text several times and these mistake slipped through the proofreading. Thanks for pointing them out.

_delirium• 14 years ago

Typo in the Hagel quote: it's "almost every area", not "almost every are".

EvilTerran3• 14 years ago

It does look nice... and it does raise interesting questions. I'm particularly intrigued as to John Kerry, John Edwards, and Joe Lieberman's presences in that anomalous 108th Congress. Can you tell us any more about how you define an "agreement group"?

Visually, the result reminds me of [http://xkcd.com/657/] - is this sort of diagram a recognised concept in data presentation, or a new idea?

[edit - hmm... http://en.wikipedia.org/wiki/108th_Congress]

toyg3• 14 years ago

Those three all voted to invade Iraq. There were probably a number of related votes for logistical and budgetary arrangements to start that operation.

EvilTerran3• 14 years ago

Yeah, hence my edited-in "hmm". When I first wrote that post, I hadn't yet hit that Wikipedia article, and, as a non-USian, I don't know what years which Senate sat for off the top of my head.

Makes more sense now.

friggeri10• 14 years ago

I was not aware of this xkcd comic, funny how things seem to pop out. My inspiration was more in the line of Sankey diagrams. Among others, the famous map of Napoleon's invasion of Russia by Minard (http://en.wikipedia.org/wiki/File:Minard.png). I've called this representation a "groupflow" but any name would do.

The notion of agreement departs a little bit from the classical fraction of common votes/total votes and instead is based on a vector model of votes, using cosine similarity as a value of agreement. The agreement groups are computed by heuristically optimizing a quantity called the cohesion and which turns out is really correlated to the way people perceive social communities (the old preprint of the first paper: http://arxiv.org/pdf/1107.3231.pdf).

specialist2• 14 years ago

Minard's graph is brilliant.

On my to do list: a graph that shows the loss of voters casting ballots. I needed a way to show how postal balloting disenfranchises voters (what it gives, it equally takes away).

Voting age population -> registered to vote -> voter received ballot -> returned ballot -> elections rcvd ballot -> signature verification -> challenged ballots -> overvotes/undervotes, voter intent, adjudication -> tabulated.

EvilTerran3• 14 years ago

Actually, now you mention it, I think I remember seeing the author of XKCD mention that he was inspired by Minard's diagram.

I'm afraid your paper's too much for my brain on a Sunday morning, but the diagrams and typesetting look really well done :)

sesqu3• 14 years ago

For what it's worth, I thought the gray background bars were far too light - it took me a while to find them. I also couldn't intuit what the branching lines were all about.

Also, Chrome had some difficulty with typesetting, though it's possible that's a localized issue.

*edit: I figured out why the boxes seem so light - it's because I view my screen at a slight angle. From a straight angle they show up fine.

friggeri10• 14 years ago

Sorry about that, I should have clarified that each Senator could be in different groups, hence the branching. I'll add a little darkness to the background bars.

What kind of typesetting issues did you have, and with which version of Chrome?

sesqu3• 14 years ago

Here's a screenshot from my Chrome 18 on Linux: http://i.imgur.com/vPwl8.png

The visualization is quite nice, and it would be useful to link to or include some kind of short description about the algorithm and visualization - to give a different example, it looks like the groups are scored numerically but the senators are simply ranked, so I wonder if the algorithm found more than two groups per congress.

joshu2• 14 years ago

Good stuff. Can you talk more about the algorithm?

Can you use this to detect issue clusters rather than voting clusters?

I have tried to build subgroup-in-graph detectors with only partial success.

friggeri10• 14 years ago

Thanks, the algorithm is a heuristic which optimize a quantity I've called social cohesion (http://arxiv.org/pdf/1107.3231.pdf), the article presenting the algorithm itself is currently under review, but basically: first it computes the best possible group for selected edges in the graph, then it merge groups which overlap too much. The whole difficulty is to identify how to select those edges, how to expand around them and then what does "too much overlap" means. I'll link to the preprint as soon as I upload it to arxiv.

What do you mean exactly by "detect" issues? C^3 should work if there is a notion of transitivity in the network (that is, if A and B are similar and B and C are similar, then with very high probability A and C are similar).

specialist2• 14 years ago

Great stuff. Thanks for sharing.

It'd be great to apply your algorithm per issue.

During Camp Wellstone training (grassroots political organizing), they taught us a tool called a Power Map. The two axis are yes v no and influence. You graph each person voting. Then you draw arrows to show who influences who. It helps to identify where you should focus your lobbying.

One point that stood out is that a Power Map is needed to be done for each issue, because many many votes don't come down on partisan lines (eg local and state, ymmv).

joshu2• 14 years ago

You have a bipartite graph of bills and representatives. You use co-voting for the similarity metric to cluster representatives. Flip that around and cluster the bills using similarity of voters.

LyleK• 14 years ago

Best use of SVG I have seen yet!

drx• 14 years ago

Would be interesting to see PCA and CA* applied to the same data.

On a quick google trip I found

[1] Principal Component Analysis of Senate Voting Patterns (http://escholarship.org/uc/item/3xm9z62w#page-1)

[2] Shlomo S. Sawilowsky, Real data analysis; Chapter 28, Principal Component Analysis of Senate Voting Patterns (on Google Books)

* Principal Component and Correspondence Analyses respectively -- http://en.wikipedia.org/wiki/Principal_component_analysis and http://en.wikipedia.org/wiki/Correspondence_analysis

olihb• 14 years ago

It's not the same data, but I did a PCA with the voting record of Canadian MPs. It's a bit old, so it's in flash instead of html5. You can access it here: http://www.votum.ca

friggeri10• 14 years ago

I'm not sure this would give the same kind of results, it was shown [1] that Kleinberg's HITS algorithm is PCA, and HITS gives [2] hubs (nodes linked to many others) and authorities (nodes linked from many hubs). I have the intuition that this would qualify the floor leaders as authorities, but I'd have to run the numbers.

Anyway, in order to obtain groups from PCA (that is, groups which are not only "visual"), one would have to run some kind of clustering algorithm on the reduced data.

[1] http://www.uclouvain.be/cps/ucl/doc/iag/documents/WP125_Saer...

[2] http://en.wikipedia.org/wiki/HITS_algorithm

pacmon• 14 years ago

Looks really great and seems like a useful tool. My only real complaint, is that I use a high sensitivity on my mouse and I found it difficult to highlight a line. If you could make the lines slightly thicker, it would be easier to select them.

caf• 14 years ago

Simon Jackman calculates ideal point estimates based on non-unanimous roll calls: http://jackman.stanford.edu/ideal/currentSenate/x1.pdf

I think his R source is available too.

rorrr• 14 years ago

1) Many senators shift to the left side (more conservative) as time goes by

2) It's especially true about republicans

uptown• 14 years ago

"However [political parties] may now and then answer popular ends, they are likely in the course of time and things, to become potent engines, by which cunning, ambitious, and unprincipled men will be enabled to subvert the power of the people and to usurp for themselves the reins of government, destroying afterwards the very engines which have lifted them to unjust dominion."

- GEORGE WASHINGTON, Farewell Address, Sep. 17, 1796

Our democracy has evolved into what Washington had prophesied in his farewell address ... a system where diversity of thought and political compromise are rejected in favor of the strength of a unified political party and the power that brings with it.

When political positions consolidate into just two choices, so do the proposed solutions to the world's problems. These solutions are artificially constrained for the sake of political loyalty. Ultimately, party loyalty requires individual politicians to sacrifice their personal beliefs on individual issues ... and once those values have been compromised, the only thing worth clinging to is control.

OstiaAntica• 14 years ago

Political parties are the weakest they've been in 200 years. The driving factor in American politics are Internet-powered movements like MoveOn.org and the Tea Party...these groups are capturing the the traditional party structure, and they are highly issue-driven. Even in fundraising, the traditional party committees are outflanked by the new Super PACs.

There is more diversity of thought and action in the U.S. today than at any other time in our history.

Show HN: Beautiful Visualization of Agreement Groups in the US Senate

Related Stories