Python for Data Analysis (new O'Reilly book from creator of Pandas)

http://shop.oreilly.com/product/0636920023784.do

by plessthanpt05 • 14 years ago

186 57 14 years ago

shop.oreilly.com

th0ma5• 14 years ago

Python is great for doing weird transformations of data that ideally you shouldn't have to do, or normalizing legacy things into modern standards. I like developing in several languages, but I often pop open Python to really "look" at something. I sadly haven't had much of a reason (yet!) to use Pandas too much, but anything we can do to get the word out about these kinds of techniques will hopefully increase procedural literacy in the world, and help ease the flow information even.

monatron• 14 years ago

Pandas is an extremely solid, well-tested and super-powerful library. Congrats to Wes on the release of this book.

wesm8• 14 years ago

Note, this is the "Early Release" version, code for "not done yet". I'll be putting the wraps on the full draft with luck by end of June.

aklein• 14 years ago

Congrats Wes on this Early Release milestone!

As a very early reader of some of Wes's first chapters, I'm eagerly awaiting the final product. It's some fine work, and Wes is not one to cut corners in his exposition or skim over important things to know.

I hope this book will advance recognition of a great set of standard tools that will make working with imperfect data as painless as possible, in a tech stack that is very amenable to 'productionization'.

Kudos, sir.

SatvikBeri• 14 years ago

Hey Wes, is there a table of contents available yet?

wesm8• 14 years ago

I'll see if I can get one posted; should be soon.

TheCowboy• 14 years ago

Consider browsing the criticism of Data Analysis with Open Source Tools(O'Reilly) for suggestions. Many people, including myself, expected a book that would have been more in line with what your book is probably trying to accomplish.

wesm8• 14 years ago

I was also very disapointed with DAwOST; it's a book about data analysis techniques with a bit of code mixed in. This is a book about _programming tools for working with data_, pretty much the exact opposite.

obtu2• 14 years ago

I was going to ask, but the early release does come with (free!) upgrades towards the final product: http://shop.oreilly.com/category/early-release.do

tatsuke95• 14 years ago

Stupid publishers.

I'm buying the hard copy, because I love books, but they don't allow me to get in on the early release? They need to offer an "all editions" package.

brianobush• 14 years ago

does this include numpy, or is it just pandas?

wesm8• 14 years ago

NumPy, IPython, matplotlib, pandas, a little bit of SciPy, scikit-learn, statsmodels, and plenty other fun stuff

neovive• 14 years ago

Sold! Any suggested prerequisites?

apechai• 14 years ago

Ayone have experience as to how Python compare to R, Matlab (Octave) and other tools for data analysis?

R has great libraries but I would prefer to use Python.

aggie• 14 years ago

You might want to look into RPy for using R libraries with python.

dbecker3• 14 years ago

I've very briefly experimented with rpy2. It got the job done, but I thought it was tricky enough that I'd want a good reason to combine R and python. Otherwise, I'd try and do the whole project in one or the other (And I haven't used rpy2 since I first tried it out.)

dbecker3• 14 years ago

I've done a lot of work using both R and Python, and a little bit using Matlab (and SAS and Stata).

Matlab is behind R and Python for both data cleaning and analysis. This is especially true if you have string variables and factor variables. Matlab's great if you are doing matrix operations of clean data (and you don't need to do anything fancy in how you report them). But, I don't find it worth using for real data analysis.

Python and R are both great, though they feel pretty similar to me. Python obviously has the big advantage if you want to do general purpose computing too. Python has been faster than R in most cases I've compared them. For my work, development speed is more important than execution speed... so this hasn't been a huge factor for me.

R has a couple of big advantages. First, the libraries. This is a big deal for me. I saw that there is a python equivalent to ggplot2 in the works. This will definitely strengthen the case for python, but the availability of libraries in R is awesome.

Second, the community and help resources in R are amazing. I rarely run into a problems in R that haven't already been addressed on stackexchange.com.

Perhaps I should be more proactive about asking python questions when I run into them, but I usually just work it out myself (which is more time consuming than looking the answer up online.)

Lastly, I'm not an expert on big data. But, spending relatively little time with both R's bigmemory and Python's PyTables, it seems easier to get up to speed on big data with R at the moment.

Though I haven't met them, my sense is that Wes, Travis Oliphant and the other relevant python developers are putting in a heroic effort to get Python up to speed. I have every expectation that Python will be my choice of the future.

clebio• 14 years ago

This is a really detailed comparison and seems balanced. Thanks for that. Do you not use both in conjunction (R for the stats libraries and Python for the more general computing)? Having done some background reading (but without yet getting down into the weeds with them), that was my take on the relationship.

Edit: re-reading your last paragraph, I guess you're saying that Python can reach parity with R's libraries, at which point it's elegance and speed will win out. R's decades of lineage do seem to hobble it in terms of style and syntax, after all.

ballooney• 14 years ago

I would add that an advantage of Matlab for number bashing is a much more native handling of linear algebra. Let's say you have two matrices A and B, in matlab I could write:

  A*B*A'

Whereas in python it would be (approximately):

  dot(A, dot(B, inverse(A)))

so matlab can evaluate everything in the right order (right to left) whereas for numpy I end up writing a little recursive function to dot a list of arguments together from right to left, which feels a bit cludgey and more of an impediment to getting your ideas down in code. Especially when your equations get very big as they often do with stats!

drunkpotato• 14 years ago

Note: This reply is mostly helpful if you work with legacy Matlab code, or have colleagues who primarily know Matlab, and you have to work with string data.

In general I agree with you; Matlab's age and origins show through in some warty ways, and one of them is string processing. Whenever I have to process anything that's not simple CSV or Excel, I use Python. (For XML, there's Perl Xpath command line tool, which has come in pretty handy for simple XML extraction.)

That said, however, the Statistics Toolbox has classes dataset, nominal, and ordinal that take a huge amount of the pain out of working with string data. Dataset lets you mix column types and refer to them by name, and lets you name rows if you like. I think it's similar to a dataframe in R. Nominal and ordinal are efficient representations for string columns. They are a workaround for Matlab's lack of a runtime string pool, but also are fast and small.

leephillips• 14 years ago

"A[n]yone have experience as to how Python compare to R, Matlab (Octave)"

Amusing coincidence: this article appeared on Slashdot this Wednesday:

Comparing R, Octave, and Python for Data Analysis:

http://developers.slashdot.org/story/12/05/23/1956219/compar...

joshu• 14 years ago

I'm thrilled that Pandas is named Pandas and not Pyndas.

hkmurakami2• 14 years ago

As I get my feet wet again in programming, I've gone through phases in dabbling with web, app, and game programming.

But about a month ago I realized, "You know, what I really loved about my studies back in academic years was numbers. That's what I want to do with code; I want to crunch numbers"

This realization made me switch over from learning Ruby to learning Python. This book is going to be perfect for my interests.

Wes, thank you for your efforts in putting this book together.

Cieplak2• 14 years ago

If you really want to crunch numbers, I highly recommend Haskell. It falls short of Python in terms of libraries and by some accounts, readability, but it is substantially faster and excels at number crunching. I find it easier to express number crunching in functional rather than object-oriented style programming. Python has lambdas, but usually theres a more pythonic way to express something in Python. Although I must say, I find Haskell's list comprehension syntax slightly more elegant.

http://tryhaskell.org/

edit: I take it back about Haskell's list comprehension syntax. At the end of the day they're both great. Haskell's syntax just feels like reading set theory notation.

hkmurakami2• 14 years ago

Thanks for the advice :)

I haven't programmed in any substantial quantity or quality since 2004, so I'll start with a more approachable language (Python), but I'll definitely keep an open mind.

carterschonwald3• 14 years ago

And there's also a great and growing set of tools for numerical and data related computation.

Scratch that, the current ongoing dev and research work that is making its way onto the haskell hackage is both astonishing and often superlative code. The libs released in the past month and the ones I know will be made available over the coming months... It's just great work! (I'm biased owing To using some of these tools for fun and profit :-). )

That aside, Pandas is a really nice piece of engineering that really works. It's a good role model for other libs, even when restricted to just its data frame part

kfk• 14 years ago

I never programmed in haskell, but I did in clojure and I must say that functional style and number crunching fit really nice. Said that, I am not sure you can beat things like numpy or pandas when it comes to complex/big datasets. I looked around in the haskell world and it seems that to do the same as, say, in pandas you need to write a lot of things yourself and you can't rely on the same superb documentation.

But I will be really happy to be proven wrong. I really think lots of things would be easier in Haskell.

pwang2• 14 years ago

Most number crunching is definitely functional; however, most programming (even in science) is about all the window dressing before you get to the actual heart of the number crunching. For those tasks, an imperative syntax is usually more approachable for those without formal CS backgrounds (especially if they know a little C, FORTRAN, or MATLAB). This is why Python is so popular with the scientific computing crowd.

yummyfajitas2• 14 years ago

I'm not sure I believe you that Haskell is faster than Python. In general, most of your python operations will be raw matrix operations handled by blas/lapack/etc.

While Python's functional programming constructs leave a lot to be desired, they are usually good enough for numerical work. Instead of using reduce(...) to sum an array and having it compile down properly, you can just use arr.sum(), which is implemented in C by numpy anyway.

I love Haskell (and want it to beat Python), but at this point, Python is the clear winner for numerical work. Libraries matter more than language in this case (as matlab demonstrates).

up_and_up• 14 years ago

If you want to crunch numbers, you should keep an eye on the Julia Language (http://julialang.org/). It's "a high-level, high-performance dynamic programming language for technical computing". Seems to math friendly with a nice syntax.

look_lookatme• 14 years ago

Don't forget to brush up on the basics (with Python!):

http://greenteapress.com/thinkstats/

neves2• 14 years ago

And don't forget to say that this guy publishes a free pdf version in the website above. I'm reading it now in my Kindle.

wgrover3• 14 years ago

wesm, do you cover using cloud-based computing resources for tackling computationally intensive data analysis problems? I use Python and PiCloud to turn what is an hours-long data analysis problem on my Mac into a minutes-long job on Amazon's hardware.

I don't know MATLAB, R, etc. well enough to say whether they can use the cloud as easily as Python, but it sure was easy with Python.

wesm8• 14 years ago

The focus is primarily on data manipulation / preparation / cleaning / processing tools, data algorithms, visualization, and practical analysis case studies. The platform where you use them is a separate matter-- I will have some material on distributed computing tools, however.

wgrover3• 14 years ago

Sounds good. I can't tell you how thrilled I am to have a book like this coming out. I'm very interested in using it as a text for an instrumental analysis class I'm putting together. Thanks so much!

msellout• 14 years ago

For R, MATLAB, or Python in a cloud Map/Reduce framework, try out Opani. http://opani.com/help/wiki

wgrover3• 14 years ago

Didn't know about that one - thanks for the tip!

npcomplete• 14 years ago

Can we get the table of contents?

wesm8• 14 years ago

I'll blog/tweet/G+ a ToC pretty soon, doing a bit of reorganization before the next Early Release update

pwang2• 14 years ago

If you're in the LinkedIn "Python Data" group, I posted a comment with a list of the ToC (hope you don't mind, Wes!)

http://www.linkedin.com/groupItem?view=&gid=4388870&...

sodelate• 14 years ago

can we use it for trade analysis?

wesm8• 14 years ago

Without a doubt (how I got my start in Python)

ruffyen• 14 years ago

Looking at the table of contents, this seems like a great reference book for several technologies. I think I am sold, especially with the free updates.

salimmadjd• 14 years ago

I recommend reading this blog,for those interested in crunching numbers on python http://slendrmeans.wordpress.com/

neves2• 14 years ago

Where's the table of contents?

Python for Data Analysis (new O'Reilly book from creator of Pandas)

Related Stories