xavian

Many many years I was this annoying jr team member that came and asked other people tons of questions. In my case, the other members of the team has something like 10 - 15 years more experience than I. Ultimately what worked was it was requested that I try to find out the answers on my own for a minimum of an hour before I pop up and ask a question of another dev. If I had questions that were non-blocking, I was asked to write them down and email the set at the end of the day instead of having multiple interactions. Nowadays, we use ticketing systems and have watchers and stuff on active tickets and any questions that come up generally get asked within a ticket and other team members are able to respond with timing that is convenient for them. If I am working on something, an interruption could cost me 15 minutes to get my "working stack" back into my head if I am working on something complex. I now have folks that report to me. I basically tell them if they are hung up to work on it for at least a couple hours trying to solve it but consider reaching out to a team member if it takes longer than that. If they are utterly blocked can cannot work on something else, my door is open at any time, but if its just curiosity, gather questions and email and I will get back to them when I can. Ultimately, I try to cultivate as much autonomy as possible on my team so folks can, for the most part, solve most issues on their own. :)

I maintain web crawlers for a large internet portal and this stuff is a pain enough to deal with when people unintentionally make their websites recursive, much less INTENTIONALLY. facepalm At least google crawler had issues as well. Thanks a lot, troll. ;P

The needs of our customer are too transient for reserved yet, so it's primarily been as needed (on demand?) We run into problems during provisioning sometimes and Amazon has to then raise our quota. Some customer architecture and capacity requirements have stabilized, so they are talking about getting reserved instances. We also have been experimenting with provisioning a bunch of servers for short durations for "burst processing" of content, but we're never totally sure if AWS will have the capacity we need. Things like reserved instances might get more use by us soon.

Most of our instances we keep up for months at a time, although they can turn flaky. We tend to use EBS, so we reprovision instances when needed. The biggest overall issue we have with EC2 is clock skew issues that can fool our fault-tolerance layers into thinking there is a major timeout issue when it's only some clock skew. That and some instances randomly rebooting in the middle of a time critical processing operation. We primarily use m2.xlarge or m2.2xlarge instance types.

It fluctuates, but most recently somewhere between 34 - 56 instances for dev, staging and production search, crawler and content build clusters. (we index terabytes of data and hundred of millions of documents)

In the immortal words of Cave Johnson, "Why not?!" :-) Dealing with structured data is easy compared to unstructured data, but there is a vast wealth of unstructured content just waiting to be tapped. I suspect there is so much interest BECAUSE TextMinr seems like a tool to explore a topic most folks find difficult to grasp. I'm a search engine architect and it's not a "solved" issue by any means. I think there is a lot of curiosity and a lot of untapped content that has yet to be leveraged. I'm certainly interested. :)