Sentiment Analysis on Enron's Emails with Apache Spark

http://mammothdata.com/sentiment-analysis-on-enrons-emails-with-apache-spark/

by mindcrime • 10 years ago

22 6 10 years ago

etjossem• 10 years ago

If this type of analysis can't answer "who Ken Lay likes best", maybe that's not the best question to use as an example.

There are plenty of more interesting questions ("when did Enron employees start getting nervous or alarmed?") that can be answered by this dataset.

kafkaesq• 10 years ago

It's not that kind of an "analysis."

More like a, you know, "Look, Ma -- I got Spark, Hive, and HDFS on my resume now!" analysis.

mindcrime2• 10 years ago

If this type of analysis can't answer "who Ken Lay likes best", maybe that's not the best question to use as an example.

"Who did Ken Lay like best" was originally going to be the title of the post, until we realized that you can't actually answer that question. It was someone else's idea anyway, but I'm not naming names!

That said, I only had so much time to work on this, and there was a limit to how much depth I could go into. I wanted to go deeper and introduce some network science / social network analysis but had other priorities.

But still, the point was just to illustrate a cool application of that particular technology stack and pique people's interest in it. If that interest helps us land business, that's a nice outcome. If not, maybe somebody learned something or came up with an idea for their own project, as a result of reading it. Those are nice outcomes as well. :-)

There are plenty of more interesting questions ("when did Enron employees start getting nervous or alarmed?")

That is a pretty good one... maybe somebody else will take the bull by the horns and write a post on that. If not, maybe I'll do it later for a "part 2".

vs2370• 10 years ago

did you do this as part of a coding interview challenge ?

mindcrime2• 10 years ago

LOL, no. It's just a fun application of some neat technology. We like to publish those kinds of things on our company blog. It's a chance for our staff to do something fun in between engagements, it helps promote the company, and somebody may learn something from it. We see that as a win-win.

WWLink• 10 years ago

Fine, fine I get it .next time I have an interview I'll solve the silly puzzle in Scala. Got it. XD

Sentiment Analysis on Enron's Emails with Apache Spark

Related Stories