zissou

Glad you're enjoying Open Torque Viewer.

It's my creation. :)

What are you talking about? Did you even look at their section on job search? It is most definitely a macro application.

Disregarding the fact that job search is a subtopic of unemployment, one of the main concepts in macro, you'll notice their model parameters are human capital, investment, and wage. The only quasi micro flavor of that model is search effort as these variables will almost always be in equilibrium based on some kind of an optimal stopping rule. Whether that rule has to do with finding a new job or discovering information about, say, the prices in the market dictates whether it would be a micro or macro use case. Again, if you actually looked at the model, their optimal stopping rule is when the person finds another job. And once again, the idea of jobs are the main ingredient in the concept of unemployment.... a macro topic. Not to mention, whether or not I'm searching for a job dictates whether I'm factored into the unemployment rate or not.

So..... yeah, job search is definitely not micro.

I thinks it's great that 2 well known economists with household names in the academy have taken to marketing programming to economists. I just wish they would have called this venture something other than Quantitative Economics when they launched it last year. It's not that it's the wrong name, it is just misleading.

A better name would have been Quantitative Macroeconomics, because that is clearly the target market. Yes, I know there are some generic topics in there like Linear Algebra that both micro and macro economists could use, but it's not like there is a lack of numpy and scipy examples on the Internet.

You bring up a good point in that the use cases for each language varies. R is a good example because one of the main reasons it is chosen is due to the fact that it already has the largest set of drop in econometrics operations for tons of extremely specific situations. In any other language, you'd often have to code a bunch of complex linear algebra by hand, which would totally negate any marginal benefits due to computation time.

In terms of econometrics, Python is going to surpass closed source software packages like SPSS, SAS, eViews, etc soon simply because the rate at which econometric procedures are being implemented in Python is growing steadily (e.g. via a statsmodels/pandas/numpy/scipy based stack). I don't think Python will ever pass R in this respect though as R has a large share of the statisticians helping add the implementations relative to Python.

I used to be gung-ho about convincing economists to use Python (started econpy.org in the 1st year of my PhD in economics, stopped updating it after 2 years). But now I just don't care. The vast majority of economists are horrible programmers, however most know how to script in at least 1 language, and a small number of them are actually really good at scripting. An extremely small number of economists know they're way around at least 1 entire general purpose programming language.

It seems that every time this gets brought up online a sea of infamous Fortran-programming-economists always tell you how fast Fortran is compared to everything else, signing it with "Fortran or GTFO". I've met hundreds/thousands of economists and econ grad students at schools ranked from 1 to 200, and I don't recall a single one that actually used Fortran (I'm sure there is a non-zero quantity -- I probably just haven't cared to talk with them b/c they are most likely macro theorists).

The fact of the matter is that in economics graduate school, your professors could care less what language you use. It isn't an algorithm competition, it's a story telling competition. But the type of story telling economists do is no less noble than writing elegant algos. Writing a story based on economic mechanisms and behaviors and supporting it with quality data, sound statistics, and logical/exact theories is no simple task.

A case of self-selection, of course.

I have been researching the used car market for the last year in an academic lab. I would love to share stories about data collection and management if Carlypso folks are interested. Not looking to be a consultant or being consulted or any kind of a "gig". As I'm sure they know, used car data is a crazy world of imperfect data points, especially as the car becomes older (>6 years old I'd say data quality begins to break down fast). What proportions of your data are scraped, purchased, or maybe even obtained from a free API like Edmunds.com? I doubt you're using Edmunds as I didn't catch any affiliation info on the site.

I laughed at the "oh yeah that makes sense" story of the big truck being priced higher in TX. My goto story is always the price of convertibles in the winter in MN vs. TX, or the price of an AWD sedan in the winter in MN vs. TX.

Speaking of AWD sedans, I know it's a rare car so this is nitpicky but your 2008 Saab 9-3 data classifies a Turbo X as an Aero. Also the Turbo X trim isn't listed for the wagon. At any rate, your estimate for the Sedan Turbo X is about 20% too low. Granted they only made 600 for the US market, I was still bummed that a car which is so dear to me had this issue. Haha, I'm not trying to be that guy whatsoever! In fact I was happy when I saw Saab was even listed as a Make as many don't even acknowledge them anymore! :(

Anyways let me know if you guys are up for a friendly talk.

I've done a Google Ngrams API that generates plots in XKCD style here:

https://github.com/econpy/google-ngrams

and a webapp version with slightly more advanced XKCD tweaks here:

http://xkcd.culturomics.org

Everything starts with matplotlibs xkcd lib but has been tweaked to produce plots based on the output of the Google Ngram Viewer.

Thanks for the info and I do look forward to the new API!

However my question still remains with regards to historic posts/comments. The historic aspect is really the import element here. Generally speaking, building an ngram viewer requires a collection of texts over time, with each text having some kind of metadata that is categorical, boolean, datetime, or numeric. Categorical data can always be made of numeric data by creating bins -- i.e. posts by people with karma or a ranking of 1-50, 51-150, 151-300, etc at the time the comment/post was created. Datetimes can also be made into useful categorical variables for an ngram viewer such as day of the week (to spot weekly seasonality trends) or day of the year (annual seasonality trends).

If I was allowed, I would be willing to write a scraper/crawler to discover as many historic threads (since: threads -> comments) as possible using HNSearch, but this could take a long time depending on rate limits and/or be subject to unknown biases within my discovery method. I'm sure you can understand why a "top-down" approach like a database dump would make for a much higher quality corpus than attempting the "bottom-up" approach of a crawler. I have no idea if a "database dump of everything" is even feasible as I don't know anything about the HN's backend infrastructure. However, if it is feasible, then I'm certain that I can work with whatever would be available. Adding structure to unstructured data is my bread and butter.

I really think this would be a very cool tool that a lot of people would enjoy, so I'm willing to do what is needed on my end to help make it work. After all, I'd be on the clock while working on this rather than just a hobby project, so the incentives are definitely aligned on my end.

If you want to discuss anything in private, I can be reached at the following reversed address: moc{dot}liamg{at}yalkcin{dot}wehttam

eBay's completed listings is definitely one of the best applications of obtaining sales data on the Internet that I'm aware of. Besides that, in some cases there are ways to imperfectly estimate quantities when best seller rankings are available (e.g. at Amazon) -- Chevalier and Goolsbee where the first to suggest this approach back in 2003.[1]

As you mentioned, monitoring half-life is another imperfect approach, but it is of course plagued by false positives (a listing goes away but no sale was made). There was a Google Tech Talk many years ago where some economists took this approach[2], except they were looking at pricing power instead of measuring quantity sold.

[1] http://www.hss.caltech.edu/~mshum/ec106/chevaliergoolsbee.pd...

[2] https://www.youtube.com/watch?v=SfjAezl3-cU#t=27m20s

Economist and long time web scraper here.

In your original business model you wanted to understand the price of everything. In what ways did the problem of a lack of information on the demand side come up? That is, it is easy to scrape the price in many markets (supply side), but what kinds of conversations came up within your team about the lack of information on how many units were actually sold at a posted price?

By the way, glad to see you guys were able to make a business out of crawling. I've landed a handful of freelance gigs since leaving grad school based on scraping data for clients, but never tried to expand it to anything beyond consulting projects.

As a long time pandas user, I'd say this is one of the better write-ups I've seen that illustrates the versatility and functions of the Series and DataFrame objects without being too long winded.

Just one thing to point out regarding the final example: read_csv will actually fetch a URL if it's given as input, so there is no need to use urllib2 and StringIO. Instead, you can just do:

from_url = pd.read_csv('http://www.example.com/data.tsv', sep='\t')

Confirmed about the generational decline: https://books.google.com/ngrams/graph?content=by+and+large&y...

Minor in economics. Skip on taking extra macro classes and instead take as many econometrics and micro that you can. Economics (or another quantitative social science) is often the missing ingredient for a solid data science background. Sure physicists have good quantitative skills, but you also need to understand what the important/relevant [business] questions are to ask.

Although not primarily in the discovery space, if you're a user of arXiv, you may be interested in the n-gram viewer our lab made: http://arxiv.culturomics.org

Despite this clip being a classic classroom example in a game theory course, it's technically not a prisoner's dilemma, for a couple reasons. First, in the PD the players cannot communicate. The ability to communicate makes it more of a signalling game since a player can say/do something to alter the other players' beliefs about their type.

Secondly, in the PD both players have an incentive to defect from their cooperative strategy (stay silent), which results in a Nash equilibrium where both players are worse off. This is, after all, why the prisoner's dilemma is so interesting -- by playing their dominant strategies, both players end up in a worse situation.

In this clip, stealing is a weakly dominant strategy. That is, if I know the other guy is going to steal, I can't be made any better or worse by changing my choice to split or steal. If I know the other guy is splitting, then I of course would want to steal (which is why splitting is an unstable strategy).

Porsche is owned by the Volkswagen Group who has a completely different strategy than Tesla. Volkswagen already makes money off makes and models marketed across the spectrum of consumers (from Skoda to SEAT, VW, Audi, Porsche, Bentley, Lamborghini, Buggati) -- all of which are owned by VW.

The Volkswagen Group doesn't get nearly as much press as it should, but I suppose the incumbent doesn't need it -- they already sell hundreds of thousands of vehicles world wide every year.

That being said, Tesla certainly has a long ways to go, but I think they have a good strategy going forward. Not to mention, I think the Model S is a beautiful car.

I'd encourage any economist that has a knack for programming to really put in some time towards the area. I've owned econpy.org for about 3 years now (although I haven't updated it in a long time). I also own economics.io and run econpy.blogspot.com.

It's much easier (relatively speaking) for an economist to pick up some programming than it is for a programmer to pick up some economics. Economists are already familiar with the types of questions that are important to economists, and more importantly, how to frame them. The trouble with economics is that you can't just pick it up overnight as it is a way of thinking more than it is a tool set. Programming on the other hand is something that you can "get working" overnight (economic programmers don't need to be algorithmic theorists -- they just need to be really good at getting/scraping data and organizing it so they can run analyses on it).

Over a year ago, I dropped out of my PhD program in economics because I was not at a school that was going to allow me to do the econ/cs type work I was working on. Leaving my PhD program was one the best things I ever did because it has allowed me to pursue whatever I want to do with the skills I've acquired.

The problem with academic economics is that the data most economists use is so bad and outdated -- such as data from FRED, BLS, and other publicly available sources where everyone and their uncle can download the same CSV dataset that was aggregated by some government employee. The race then is to see who can put together the most elegant econometric model to handle all the issues with the data. The rules of the game change when you create your own dataset and thus have control over while variables to include, the aggregation, the frequency, etc.

Long story short, if you are an economist wanting to do programming, learn to adapt those skills in academia (best way is to find a great advisor -- if there isn't one in your economics department, check the business school as bschool professors are often much more open to highly empirical analyses and care [marginally] less about getting the theory perfect). Or, if you want an easier lifestyle that is much more rewarding, ditch academia for the private sector. You'll find the economists in the private sector to be much more knowledgeable about cutting edge technologies and willing to listen and learn from what you have to say.

Damn, just when I leave Harvard, Schneier comes to town. Would have loved to do some work with him on security theater.

Fun fact about Aeroflot: The slogan for Aeroflot back in the days of the USSR was "You Have Made The Right Choice", which is ironic since Aeroflot was the only airline in the Soviet Union.

Detailed examples of the model of failure described in the post are bound to be few and far between because of the massive selection problem. People who fail are way less likely than people who succeed to to tell the Internet about their experiences.

Don't get me wrong, as a general statement I support free markets. However, Keynes' thesis (which is misinterpreted 96% of the time I hear someone talking about it) was simply that sometimes there is an economic argument for government regulation. Keynes' simply pointed out that sometimes it takes a while for some markets to readjust back to equilibrium (i.e. the invisible hand is slow) -- sometimes the market price is sticky. In these cases, the government can implement policies that incentivize the actors to do more of something (by issuing a subsidy) or do less of something (issuing a tax) in order to push the market back to equilibrium. That's the economic argument of Keynes.

The other component here is the idea of a public good. A public good is something the private sector doesn't have an incentive to do on their own (b/c they can't make money off of it), but it is something that society would benefit from. The role of government is generally to provide these public goods, such as national defense and parks.

So, the world will never have completely free markets. Besides providing national defense, the government also provides services designed specifically to protect consumers from unfair business practices, like collusion, price fixing and other monopolistic behavior (although, I just want to point out that is NOT illegal to have a monopoly in the US -- it is however illegal to monopolize -- big difference).

University of Arizona

The underlying reason I think most DSGE models are absolute hogwash is that they do economics backwards. Economics is supposed to test theories against data. Instead, in the world of calibration in macroeconomic models, the creator of the model is now testing data against theory by tuning parameters to values they think are good. It is completely backwards.

While I appreciate the attempt to make macroeconomics more computational, I believe DSGE goes about it in the wrong direction. In an ideal world, I'd like to see models like the Leontief Input/Output model come back to fruition. In Leontief's model (which is often given as an example in many undergrad linear algebra classes), the economy is divided into many sectors. Data is organized on each sector to estimate its influence on other sectors. In an age where data is so vast, I just don't understand how one can decide that building deeper macro theories is a good idea. We need better empirical models, not better theoretical models (we have enough of those).

There are some truths in this article -- for example, it is generally true that economics PhDs don't have a hard time finding jobs. Don't want or can't get an academic job? No problem, there are plenty of consulting/industry/government jobs to go around for economics PhDs.

I'm an economics PhD dropout (after 2nd year) from a top ~40 program. Unlike the authors school, my former PhD program did not have the prestige of his top 10-20 school. I left my program because I didn't have a research match with my professors. Now I'm a computer programmer at a top ~1-5 ranked school and I'm infinitely happier than I was in graduate school. I may eventually go back for a PhD, but it will most likely be in a business school rather than an economics department because I enjoy applied work more than theory.

The main ridiculousness in the current coursework taught in most economics PhD programs is the macro course (e.g. freshwater/Minnesota macro). DSGE (Dynamic Stochastic General Equilibrium) is just stupid. For the uninitiated, here is why DSGE exists and what it means: A while back there was this thing called the Lucas critique, which was the observation that the models used in microeconomics and macroeconomics are very dissimilar -- that is, micro models say 1 agent will make a certain decision, so why isn't it that all agents will make that decision? Macro theorists took this criticism way too seriously and decided that their models needed to have micro foundations, hence giving rise to the macro modeling technique known as DSGE. How does the typical/introductory DSGE model work? It goes something like this: at the beginning of time, everyone in the world meets in a parking lot. At the meeting, they decide on the price of all K goods in the economy at all T periods of time by assigning a present value to each good at each period of time. Once all the prices have been decided, the world starts. That is the gist of this model, but if you want to read more about how silly it is, look up the idea of a representative agent DSGE model.

Anyways, my point here is that depending on what you want to do with an economics PhD, it may or may not be a waste of time. For example, if you study micro and econometrics and emphasize in industrial organization + game theory + experimental, you will make a great candidate for a cushy research scientist job at one of the big tech companies. If you study macro, well, you'll make a good candidate for a job.... doing macroeconomics?

TL;DR If you're going to do economics, don't do macro.

Like the wise scholar in England wrote, I'm most productive after a nap but before I go to bed.

Is taking a nap then "going to bed" later basically the same thing as a 2 sleep night?

I think naps are a good modern example of a 2 sleep night. I wonder when people began napping? The Google Ngram Viewer makes me think we've been napping for years [1], except perhaps right after WW2, but then 1970 happened and we started getting tired. :)

[1] http://books.google.com/ngrams/graph?content=nap&year_st...

Happy to finally see an intro to data science article that puts statistics as the #1 skill. While software engineering is also important, all the engineering in the world won't help you extract any meaningful insight from data without a solid foundation in probability and statistical theory.

'I Pencil' was first introduced to me in a History of Economic Thought class during undergrad. Still love the story if not only because of the message it delivers. Who cares how it is written, 'I Pencil' is a story about how trade, comparative advantage and the specialization of labor enable economies to produce goods and services that would otherwise not be feasible.

So why does the HN crowd seemingly let the story slip through the cracks? One hypothesis is perhaps that the self-selected independent/entrepreneurial people that make up a large share of the user base on HN like holding onto their independent spirits, hence passing on the idea that there exists a meaningful good/service that not one person can produce on their own.

Remember that there are many ways economic logic leaks into the the actions of governments at-large and within the institutions, but I think at the highest level this is an argument against government regulation. The principles of this argument are the same as a Keynes vs. Hayek/Friedman argument. Keynes basically pointed out that sometimes markets don't always adjust quickly, so in these cases there is an economic argument in favor to "helping" the market restore to equilibrium (a "visible" hand).

From macro issues like regulating the financial services sector to micro issues like antitrust and competition policy, consumers like you and I can be made better off via regulation. For example, competition policy is one of the most productive types of government regulation that takes place. Disregarding the DoJ's treatment of online poker [to date], the FTC and the DoJ have been doing some pretty great things over the last few years[1] (e.g. $500 million in fines issued breaking up LCD price fixing scheme; continued effort towards addressing the current and future problems of privacy on the Internet)

In the US, the goal of competition policy is to promote the economic welfare of CONSUMERS, however a good policy would be one that increases the economic welfare of both consumers and producers. Game theoretically, figuring out how to make a policy that increases the welfare of both sides of the market is a mechanism design problem.

[1] Relative to almost all the other sectors of government.

I've learned a lot from the NLTK library, but unfortunately NLTK is terribly slow. Nevertheless, it is a fantastic place for beginners to start with text processing and learn from as the documentation is superb. However eventually one may want to start digging into the NLTK source to rewrite necessary functions using multiprocessing if they plan to process any "big" textual datasets.