sciboy

Honestly, the only way to understand regression is to study something like Mccullagh & Nelders book. Anything else and you are going to have a very hard time really being useful without misinterpreting the results. There are some real subtleties to interpretation of regression coefficients, and more importantly structuring your data in such a way that you will answer the questions you want.

It's not an easy book, but if you've gone through to at least third year level in statistics it's approachable and you will understand it to a deep level.

We've been using phantomjs, but it has it's issues too. I'm not sure there is a good solution anywhere in the open source world unfortunately - and tools like prince are expensive.

Just so you are aware, flying saucer while nice when you first use it has tonnes of bugs and isn't really being developed these days. You'll find yourself more and more diving into the code because the output is substandard. We used it for years and have now moved away because we couldn't stand customising it for every little edge case more, plus it doesn't support html5/CSS3 which is essential nowadays. Take a look at the codebase - you won't want to be adding to that! Additionally it expects documents to be completely in memory, which means it will take down your Java server sometimes.

B = (X^TX)^(-1)X^TY anyone?

This was our assignment for an AI class many years ago - take USDA and a GA with constraints set by users to generate meal plans.

I remember being told to eat a kilo of salt :)

The point is that it shouldn't matter how the person chooses to put their thoughts into the world, they are their thoughts not yours. If they choose to do a private performance (a concert) and it gets pirated, that's not much different then if they are having a conversation and it gets pirated. The distinction isn't theirs it's the pirates.

Just because they are already granting permission to "read their thoughts" for money doesn't give anyone license to just do what they want does it?

I fail to see how the users choice on how to distribute files grants pirates license to go against the thought originators wishes.

Note I'm not saying I agree with this - but I find it interesting to think about the opposite viewpoint to my own as it helps me rationalise my position.

I completely agree - technically.

If we invent a technology that allows us to read thoughts, should you be allowed to read them without the thinkers permission? The thinker still has their thoughts. What about disseminate them via torrents? Would that be "fair"? Is that the society we want?

What about monitoring conversations via surveillance means? The speakers still have their conversation. Should we be allowed to a) surveil anyone and b) distribute that surveillance digitally as the surveilled still have their conversation?

The time is close approaching where such questions will need to be asked and I see little difference between piracy of songs, movies or other artistic expression and the extraction and distribution of thoughts against the owners wishes. The only real difference is that the artists chose to let it out into the world in a way the pirate didn't like. One way was via concert/itunes/cd/dvd, another way could be a conversation with a friend. Both ways the pirate says they have the right to use it how they want - original thinker be damned.

In my view it comes down to respect. While technically it's not stealing, you're still an asshole for using others thoughts without their blessing.

I can imagine the day where someone hacks into your laptop, records you in a compromising position, shares it with the world and shrugs and says "how is it stealing if you still have that something"? I just copied your dignity.

Not all that can be stolen is a thing.

Given a real world problem, algorithms or the mathematics behind it do not provide insight on their own. The only value knowledge of the algorithms provide is a general understanding of the ways in which you can approach a problem. If you can do the mathematics, you can look at how the way you structure your question influences the results you are seeing.

The process of exploratory data analysis is best done in (imho) the style of a Lakatos research programme.

- prepare and clean the data

- explore using many fast methods and charts, come up with some working hypotheses about the data that are important to your client

- select method(s) to test those hypotheses

- perform the analyses

- determine what your results mean in terms of your research goal.

- alter your list of working hypotheses

- repeat [possibly collecting more data]

Obviously the only hard part about this is step five. And unfortunately this is the step that isn't t really taught in my experience. A simple case: Let's say you had a linear regression and you ran it once with 2 variables, got some parameter estimates (a,b) , and ran it again with 3 parameters and got some more parameter estimates (a1,b1,c). If b != b1, what does this mean? If you are using a custom link function (e.g. cloglog or logit), how should you interpret this now? This is where having a deep understanding of the mathematics behind the techniques starts to pay off. And this is the simplest example of a basic regression.

That's my point. Algorithms are the domain knowledge of cs.

It wasn't this. The example was trying to show that a lack of knowledge can be catastrophic in terms of productivity for even the gurus. He spent more than six months on this particular problem and if he had known more math (very typical of people in comp sci unfortunately) he would have been enormously more productive. The problem was formulated before he started any work on this, it really was his lack of (graduate level) math and statistics that caused this particular. I'm not blaming him, it's just he didn't have the tools in his toolbelt to understand the best way to solve the problem, and nor would anyone who didn't have a strong math background.

Math is unreasonably effective. Just like knowledge of basic algorithmic analysis. I don't think that should be controversial.

We once hired a guy with PhD in computer science & he was a (great) lecturer & researcher for many years. We got him to do some "complicated stuff". I'm not going to go further into the details in case he recognises this.

After he left I went back to learn math for interest and got really into it. I now realise that literally six months of the work he did I could do in an afternoon as I now have a deeper understanding of the mathematics behind it than he did. I'm not smarter, far from it, just have more tools to call on. And this was a guy that taught the algorithms course at a good university.

We are all capable of learning almost anything, but a lack of knowledge about the subject matter constrains your solution search space to a large degree. I believe strongly that fluency in algorithms is a minimal prerequistite to be a decent developer. It's like learning calculus without knowing trig, you can do it, but you'll be reinventing (poorly) a tonne of work that came before and you'll be slower and less accurate. It's not like it's rocket science either, you could get a basic understanding in a concentrated day of work so there's no reason not to.

I used to love data mining. Then I formally learned statistics, causal inference, mcmc and other similar methods. I was amazed at how poor the computer science was for anything outside of computer science problems (i.e. search, collaborative filtering etc). Looking back I now realise that my amazement was misplaced - tools are good for what they are designed for. How many computer scientists who write the tools run experiments or do real exploratory data analysis with data they have collected?

Agree completely with their model selection criteria; model selection is useful when used to compare evidence for k physically plausible models, but should be treated with extreme caution where exploring an infinite model space.

Personally I hold very little hope for automated tools anymore. Considering how complex a seemingly simple study is to analyse correctly, or how hard it is to model physically realistic processes, I think the future does not bode well for tools like eureka as they are currently aimed. At best eureka may present some general hypothesis, but it seems unlikely to be able to search the model space for models that are physically plausible, except in the most simple of all cases.

Besides, try eureka on some of your real data and prepare to be deflated :)

I find it hard to understand how writing open source software has a strong value proposition for someone who isn't worried about "getting the next job" or "looking good in front of peers". For me, I believe the best use of those extra hours I have is to learn entirely new subject areas, not writing tests for code I don't care about, answering support or setting up documentation. I don't care if people know what I am doing, I'm not in it for the glory or whatnot.

Instead, over the last few years I have added two entirely new subjects to my repertoire (mathematical statistics and neuropsychology) to a level where I can read, critique and contribute to the latest research in the field. Over the next few years I plan on repeating this process with other subjects. Comparing this with writing more software about things I already know seems to me to be comparing progress with diminishing returns.

If you believe I'm wrong, I'd love to know why. Convince me there's value in opening up the 20+ projects sitting in my source control.

With your claimed grades and intellect you could obtain scholarships to almost anywhere of your choosing, fully paid fees + stipend, to work with the greatest minds in your field. Oxford, Cambridge... the world is your oyster. So it's a completely one-sided proposal; work with the best in your fields, and they will pay your way. How is this a bad deal, and why didn't you take it?

I still find it weird that people charge an hourly rate. My company has been doing fixed price projects, with changing scope, guaranteed bug fixes & payment on delivery for the last eight years. We regularly beat out the "big boys" and have never lost money on a project, even though some have gone on longer than we planned.

I don't see the value for a customer in an hourly rate. If I am charging you by the hour, I have an incentive to be slow. If I am charging fixed price, I want to be fast, and if we are "liable" for bugs, we have an incentive for bug free code.

I see it as putting our money where our mouth is. I'm confident that even if your scope changes we will still deliver for this fixed price. Why is this not the norm?

Then the gpl is a bad choice for them as a license?

I disagree. He makes two unsupported claims that reek of cargo-culting:

"1. The optimal wake feature reduces daytime grogginess only in the sense that it puts you in a better mood in the morning. 2. Yes it is very unlikely to have adverse effects. You are just waking a little bit earlier at a point where your body feels more inclined to wake up."

Two claims. No data from either published literature (from other studies) or their own data. How do they even know it works? Why should we treat it any differently than claims of magnets improving your sleep? A free pass shouldn't be granted just because they are YC.

What is the confidence interval for the size of the effect that wakemate produces on percieved grogginess vs waking without wakemate?

What is the size of the effect in the research?

Unreasonable faith in myself to pull it off!

1) I would disagree. The definition of polymath is someone who is an expert at orthogonal subjects. Your definition of expert seems to be slightly lower than mine. My definition would be someone who could give a lecture to other experts in the field with a days notice and be able to answer questions to the satisfaction of the audience.

2) If this were true it would be a short time until science itself crumbled. How then are people able to keep on top of their fields with finite time? The number of scientists isn't exponentially increasing, nor are the number of publications. It's growth, but it's not exponential as far as I can tell.

3) A polymath is someone who has knowledge in orthogonal knowledge spaces. If your field requires knowledge in other very related areas then I wouldn't consider that polymath-like activity.

I often help (good) researchers with experimental design and statistical analysis of quasi-experimental data, and it's shocking how little they understand. It pains me to think how much waste there is in science at the moment because the researchers do not have the statistical or numerical background to even know what questions are possible.

What would you consider uber-hardcore?

Would be nice to see the fonts rendered

Is there any data that supports this blog posts contention that the best developers are in the UK on tier 1 visa's? If this were true, surely a large portion of the successful uk startups should be founded by tier-1 recipients?

Knowledge is valuable. Everyone knows it. But very few are actually delivering knowledge. Take a look at the "web dashboard" on dataspora's page. Does that "deliver knowledge" or is it a bunch of charts that the people on the ground won't use?

The key to a good analysis is new understanding & actionable recommendations with associated risk, not "big data" or visualisations. Unfortunately they are the easy part - the hard part is turning these into knowledge for your client. I've never seen this even attempted in a dashboard like this (not picking on them, it's industry wide), and I rarely see it in outsourced analysis because usually the analysts don't have the domain knowledge required. I've seen the results of many mid-6 figure analysis projects and they weren't pretty.

When probed to be truthful, many of the companies in these spaces struggle to provide any real value for the client. I challenge anyone to give me an example where their company has provided true value. There are two requirements in my experience to this:

1) A concrete example with a real monetary gain greater than the cost of the project 2) The company increased spend for another project and got similar results. If you are really that good, customers find more money and more projects.

I hate to be a skeptic about this as I am a trained statistician and this is my area, but from what I have seen this is typical of the industry. </rant> :)

I've looked at matplotlib but still think it's ugly by default.

Compare to:

http://www.texample.net/tikz/examples/weather-stations-data/

http://www.texample.net/tikz/examples/rna-codons-table/

The beauty of tex font's + vector graphics is impressive. Try zooming in on the sparklines in the first example

I have used R/ggplot for a long time now, and it's exceedingly difficult to produce beautiful visualizations.

The only thing that makes tufte-quality visualizations in my experience is hand-building your graphs in tikz.

The graphs you linked to are hideous from an aesthetics point of view; font's are ugly, data:ink ratio far too low, color choices poor etc.

No offence but discussing the most recent/interesting paper you have read shouldn't be a tricky question.

I simply give them a call and ask them what the most interesting papers they have read over the last year are. Without any forewarning of course. People who are passionate about things know their stuff; and it's not from blog posts or forums that they obtain this information.

And I can tell you, if I learn something while on the phone with them then it's a guaranteed good hire.

No, because once something is bsd or apache licensed you could repackage it as a commercial vm.