The Rise of the Data Scientist

http://flowingdata.com/2009/06/04/rise-of-the-data-scientist/

by physcab • 17 years ago

47 16 17 years ago

jimbokun5• 17 years ago

Recommendations for becoming a data scientist? Sounds like a wonderful job.

What books would be on the required reading lists? What is the best set of tools to learn and know?

An alternative approach: how much of this could be automated and delivered as a web application/service? Provide easy ways for people to parse and upload their data, algorithms to run with good descriptions and recommendations for lay people, and well designed visualization tools for presenting the output of the data. This would be a very valuable service, but would take a broad range of knowledge and expertise to implement well (experts in the CS, Statistics, and Visualization fields as described in the article).

physcab• 17 years ago

It does take a wide-variety of knowledge. But I don't think you need to be an expert in all these fields. You just have to have a passion for presenting information to others. As far as what tools are needed? Here's my take.

Computer Science (Data Mining, Machine Learning):

- Hard science book like PRML from Chris Bishop or Neural Networks from Haykin

- Fun implementations from Programming Collective Intelligence by Toby Segaran

- See practical implementations by these people: http://www.doloreslabs.com/

Statistics:

- Good foundation of probability theory and calculus

- Understand these slides: http://www.autonlab.org/tutorials/index.html

Visualization (something I'm working on but not good at yet):

- Check out http://processingjs.org/

- Dabble in Adobe Flash/Flex Framework

paulsb4• 17 years ago

Don't forget the graphic design and HCI parts, and definitely do not underestimate them. I often find that people with a scientific/technical background find it hard to visualise and present data well because of their background - 'I love lots of information and data; therefore, everyone else must love lots of information and data'. Wrong. You can number crunch all day long, but if you can't translate the output into something that people can understand (i.e. turn information into knowledge) then there is no point.

You will also find that a lot of the top data scientists/information visualisers actually come from an artistic background, e.g. http://www.visualcomplexity.com/vc/ and http://www.vimeo.com/5091290

The original article inspired me to go back to university so that I can become a data scientist. Can't wait.

RK• 17 years ago

People definitely love Tufte's books about information graphic design.

Then of course for the actual data processing and visualization you'll need software such as Matlab, R, Processing, or Python (with Scipy + Matplotlib).

acgourley• 17 years ago

As for your latter suggestion - I think its unrealistic to build a general web-based solution. As you are describing it, you would essentially be hosting R, some visualization tool, and a scripting tool. That's absurd enough and doesn't even address the bandwidth issue - these datasets are usually huge.

You'd be better off writing enterprise software in this field. Especially considering anyone who would pay for data munging and visualization tools/services probably doesn't want their data out on the internet.

neilc• 17 years ago

I think its unrealistic to build a general web-based solution

Maybe not a perfectly "general" solution, but there are lots of domains where data is analyzed that would benefit from a powerful web-based analytic environment.

these datasets are usually huge

Not all interesting datasets are huge. Some large data sets can easily be shared among users -- Amazon already hosts a lot of common data sets in EC2 already, for example. Given the steadily-decreasing cost of bandwidth and the fact that you don't need interactive response (at least for the initial data load), I don't think it's that impractical for a pretty wide set of scenarios.

vbar• 17 years ago

Hosting R is already implemented - see http://predict.i2pi.com/

jimbokun5• 17 years ago

"As for your latter suggestion - I think its unrealistic to build a general web-based solution."

Might be a dual product strategy, the public facing webapp with limits on data size to test drive, and customer site installed enterprise version. Or maybe some solution that uses Amazon storage (in other words, outsource data storage and security issues to Amazon).

Of course, Google had the two tier search idea (they wanted to monetize by selling custom search servers to businesses) but found the real money was to be made on the consumer facing side. So you never know.

paulsb4• 17 years ago

I think there is a need and place for it.

One of the things I am hoping to do with my start-up is to build simple tools to help wet-lab scientists access informatics solutions to analyse and visualise their data, and simple tools for informatics scientists to visualise their data so that a wet-lab scientist can understand it (more easily). There is a huge communication gap between (bio)wet-lab and (bio)informaticians, and I think 'data science' tools can help to bridge that gap.

Estragon• 17 years ago

The Visual Display of Quantitative Information http://www.edwardtufte.com/tufte/books_vdqi

vijayr• 17 years ago

This link has tons of data visualization examples and resources

http://www.meryl.net/2008/01/175-data-and-information-visual...

The Rise of the Data Scientist

Related Stories