akshayubhat

I can only think of using 80legs as a crawler, since its distributed enough to make sure that you don't run into any IP address based rate limitation. But it's just a guess.

So many upvotes, I find information cascade happening here.

a harvard (or UIUC, stanford, cornell, MIT, UCB) CS grad can easily make 70k$ a year right out of college and much more in future. 100k$ tuition might be an issue for liberal arts and communication majors, but surely not for a CS major (assuming he is good enough to have a successful startup in first place).

Interesting! There is an open source Apache Mahout project for doing machine learning via hadoop. Also have a look at Vowpal wabbit an open source framework for fast SGD for online learning.

Also another interesting point is their use of boosting, since i recently attended a tech talk by facebook engineers where they told us that bagged decision trees while predicting the friends for a user.

Interesting anyone knows which strategies they use to decide to buy/sell, once they have determined the expected price?

do they use physics based techniques, like black scholes?

or do they use statical learning based techniques?

Something like this wont happen to Google App Engine, thankfully.

err HBASE is the database and not HDFS.

I completely agree, I find it weird whenever someone mentions Hadoop as as NOSQL. Hadoop is a Distributed Computing system. While HDFS is the Database.

Hadoop is more of an Distributed O.S. to run process and store data across multiple machines.

Well he has Indianized English Grammatical style.

On a side note, had this been a case with an MacOSX application or Windows application, would you have petitioned Apple or MSFT?

oh are i-OS API and code cross platform compatible? Do any other companies except Apple create iOS devices?

in fact Apple used 3.3.1 to make sure that code was not cross platform compatible.

no because you can run programs on Acer netbook which will also run on say HP and Dell netbook's.

no more 3.3.1 that's good. Finally sanity prevails over "interface decides programming paradigm/language argument".

Evolution, why DDT wouldn't work. http://en.wikipedia.org/wiki/DDT#Mosquito_resistance

People who use AMT for earning money may not necessarily be the right audience you are looking for.

Entrepreneurship isn't the cure-all pill, good Ideas are hard to come by, even if you are looking for them continuously. Second Large Businesses have huge efficiency advantage due to their scale.

While HN would seem extremely pro - entrepreneurship, in reality, starting your business is not always the smartest move, not from just personal perspective but from perspective of a nation as well. Sure you want Larry Page and Sergey Brin to create Google but before that they went to Stanford for Ph.D. in CS. Entrepreneurship just for sake of it and without any good idea is no better than playing stock market with different strategies.

     If 15% of the unemployed engineers in Corvallis could start companies that hired 5 full timers, they would (probably) have a booming economy all over again. Shit, maybe we could start exporting to China (I think they value entrepeneurship and industrialization there...).

The only way to generate income is via means which are non zero sum games [at least non zero sum within that nation].

Wow, Doctors are different case altogether

Their supply is limited and controlled, anyone can come and say that they are "web programmer" after reading HTML/CSS/Ruby, Even Universities increase class sizes arbitrarily. All this is not possible in medical profession. Doctors are few, they are trained extensively, not anyone reading anatomy book can go for a residency!

I could not understand several points made in this article:

1: He mentions infochimps, but according to my knowledge its more of an ebay for datasets rather than support/provider for Big Data Stack, also is it successful? I am unsure how infochimps is related to the Big Data stack.

2: From what I remember reading about 80legs, is that it uses distributed grid computing to run the crawlers (something like SETI @ Home), I doubt Hadoop was ever designed for such applications. So this is surely isn't a Hadoop use case.

3: Quoting:

       While the standard big data stack has made huge strides in making big data more accessible to everyone, it will always fall short against our stack when it comes to the cost of collecting data.  We actually don’t store that much data. Because 80legs users can filter their data on the nodes, they’re able to return the minimum amount of data in their result sets.  The processing (or reduction, pardon the pun) is done on the nodes.  Actual result sets are very small relative to the size of the input set.

Also he mentions """ We have about 50,000 computers using their excess bandwidth.""" 50,000?? The biggest Hadoop cluster That I know (Yahoo) has ~10-20k nodes, and Hadoop was never meant to be used at 50K scale for crawling. So they had no option other than building their own system, even if they had to build it today.

4: Quoting

      One advantage is optimization — an “off-the-shelf” system is going to have some generalities built into it that can’t be optimized to fit your needs. The opportunity cost of going “standard” is a slew of competitive advantages.

While I seem to agree with Authors conclusion that a highly focused startup should make their proprietary solution, I cannot agree with his evidence behind that argument. A grid based crawler with 50K machines isn't something that Hadoop was ever designed to support.

Yup he is young, he did a post doc for a year at Cornell [under Prof. Kleinberg] after his PhD, and directly became Asst. Prof at Stanford.

Well you can start with reading this book http://www.cs.cornell.edu/home/kleinber/networks-book/ for overview, regarding application of these techniques are considered you can look for papers at recent WWW, NIPS,ICML conferences. The most popular and well studied areas are Link Prediction and Community Detection. The SNAP library comes with some good example code. You can also have a look at Divisi project at MIT Media Lab if you are interested in reasoning/ analogy over the networks.

For real datasets, there are a lot of encyclopedic datasets such as Wikipedia / DBPedia /Semantic Web/ Music Brainz, as well as social ones such as Twitter follower network dataset. If you are in a university, you can even get full Web Graph from Yahoo [for research use alone].

iPad should come out with Disney specific version (in pink for girls and in red/blue for boys). Consider the opportunities, an inbuilt iTunes store with all superhero / Cinderella type cartoons [they will learn to pay for the content from the childhood itself no need for RIAA / MPAA lawsuits], there is so much opportunity in this field, I can totally see kids harrowing their parents for one. Wonder how Steve jobs hasn't already began producing one.

Java / Ruby is an odd choice for trading particularly HFT, everyone else I have seen seems to be using C++ / C.

Though its not as odd as compared to a quant [from a large bank which held an info session at my unive] who I met last week and told me that he uses VBA + Perl (for research not trading though).

Solved it, knowing an old math concept helped, did a Google search and got the values [writing the code to loop through the values for m,n was easier but felt someone must have computed them and posted on web].

Just wrote the code for cipher and used all possible characters and replaced using the encrypted value. Though I nearly sent it to the wrong address since it wasn't clear whether address contained the answer it self or answer%256. took ~30 mins [mostly to look up ord,chr and the solution on google].

final solution 12 lines python code, does not calculates a,b,c just the cipher and i/o.

Interesting. Sadly it does not mentions any practical tools for doing social network analysis:

Here are few that I know:

For small networks (up to a million or two million nodes such as Wikipedia Link graph from 2009)

Following libraries provide code to handle and manipulate Network datasets:

1: SNAP by Prof. Jure Leskovec [ http://snap.stanford.edu ] written in C++

2: Networkx by Lanl [ http://networkx.lanl.gov/ ] written in Python, esp. good for fast prototyping

There are few Databases for storing networks, e.g. Neo4J http://neo4j.org/ .

Additionally there is a Graph Processing Language called as Gremlin

http://wiki.github.com/tinkerpop/gremlin/ .

For Large networks with millions and billions of nodes, one can use Hadoop / Map-Reduce or Apache Hama [still in nascent stage]. Google has a special system known as Pregel which it uses to perform scalable computations over large networks.

     Your definition of success is your personal opinion, which you admit is irrelevant.

The method used in the info graphic mentioned in your article does not differentiates between profits from different divisions within Apple or even HP and Dell.

BI is an Apple fanboi site, so no wonder, it dishes out nonsense, to quote from a recent article "how apple is beginning to destroy Googles core search business "http://www.businessinsider.com/heres-how-apple-is-beginning-...

     The App Store is replacing the web search engine for mobile device users, at least for some searches. This should scare Google, because search is its core business, and it's being supplanted.

      but then were never used by the actual users

Cost of not getting a job done due to lack of a feature is very high compared to cost of adding that feature. And cost of adding a new feature only when a need arises is even higher.

It is hard to predict which feature you will actually need over course of time, thus the safest bet is to have as many reasonable features as possible.

Again your fanboiism at play:

1: you disregard my statement about evolution of ipod from a simple to a complex interface. Your link shows 3 different types of iPod, which shows that simplicity didn't work and design had to evolve, to incorporate different needs.

2: In regards to iPhone, Blackberry dominates the market share and guess why? Because they provide a much needed functionality "QWERTY keypad", which proves the point of the linked article.

Apple Mac's aren't a success considering their market share. [Personal opinions are irrelevant]

Your argument is fallacious: iPod evolved to address the feature issue: so from a single Classic iPod you now have shuffle, nano and touch. even within each type say iPod nano there has been considerable change. Check out the latest nano and the earlier one, adding touchscreen is same as adding complexity.

It's probably your Apple bias which seems to be at play here.

A good way to think about it is via Occam's Razor: Which states that "Multiplicity should not be posited without necessity"

Thus the pioneering products in any new segment are simple, however as time passes new "necessities" emerge, which means the product "must" be modified to fit those needs. This also assumes a well regulated market with rational buyers (no fanboi's).

Note that this does not means that Simplicity should always be favored or The simplest solution is usually the correct answer, even if there is a single legitimate need which a simple solution cannot address, then the solution must be made complex to address that need effectively.

Evolution of Personal Computer or any non monopoly product is a good example.

Congrats dude!

Thanks, I didn't knew that, It seems to be like Android model, where they open source it few weeks after public disclosure of features.