pablohoffman

Hey folks, I'm the one interviewed, happy to answer any follow up question here.

Not surprised, I'm pretty low profile :)

Portia from Scrapinghub is 100% open source: http://scrapinghub.com/portia/

We released a similar open source tool for visual scraping, earlier this year, called Portia: https://github.com/scrapinghub/portia

It's been getting quite a bit of traction and we're currently working on the integration with Scrapinghub platform (disclaimer: I work there) for those who prefer a hosted version.

Not yet.

Yes, you just need to select a different field type ("text", instead of "html").

We are using CDH4.2 and have had a very positive experience so far.

Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.

We are not plainly complaining about MongoDB, nor saying it's useless. We are just explaining why it's a poor choice for a specific use case: storing scraped data.

FWIW, we still use Mongo in other internal applications, it's just not the right choice for our crawl data storage backend.

We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.

The title of this post is misleading, it makes you think GitHub is actually broken, when the article is about complaining how their fork feature works.

I initially submitted this post, but then deleted it and resubmitted to the original post on Common Crawl blog: http://news.ycombinator.com/item?id=3208853

I now regret since this one got much more attention. I was under the impression that linking to the original post was more welcomed here HN, but it seems this is not always the case.

I found strange that no one mentioned Privnote (https://privnote.com), a much older/popular service that does this but more securely since it generates URLs with keys in the fragment to encrypt the messages on client side (so that the plain message never hits the server).

Did it work?

I've been using tiled window managers for 7 years now (currently on awesome, which I switched 3 years ago) and the interface has always remained pretty consistent. I wonder if Linus has tried tiled WMs?

After checking a few registrars, I found they only sell third-level domains (.com.ng, .net.ng), not second-level domains (like "amazi.ng") which is probably what most people here would be interested in.

Example: http://www.webdomains.com.ng/myaccount/whois.php?step=1

It's a shame they're missing the opportunity to sell such a nice top-level domain - or perhaps it's done on purpose?

Someone already wrote a Delicious scraper in Scrapy (Python): http://news.ycombinator.com/item?id=2015680

Feeling sad would imply that you hold some respect to the judges that decided who won, but I think Time magazine just lost the respect of many readers. I'm kinda sad for them, not for Assange.

That reminded me of a recent Paul Graham tweet:

  You've found market price when buyers complain but still pay.

Primary source thread: http://news.ycombinator.com/item?id=1849274

Is it me, or Zed is starting to get annoyed at the Python community now?. I wonder when he'll write "Python is a Ghetto".

As much as Facebook does.

I also find annoying that Posterous, for some reason, appends numeric suffixes arbitrarily to certain post slugs (for example: http://blog.pablohoffman.com/hi-posterous-58). It's like they have to make the post slug "globally unique", instead of "unique per blog", although they shouldn't have to.

It wouldn't be that bad if you could change the slug/url but, as the article correctly points out, you can't.

See http://news.ycombinator.com/item?id=1355072

See http://news.ycombinator.com/item?id=1255858

Fair enough. But my point was that, sooner or later, you'll need to deal with the details and I've found that is, quite often, the most time consuming part (sometimes you just misjudged which ones where the issues, and which ones the details).

You can always delegate the "details", but there's a difference between delegating and deferring (the work for yourself). Knowing what to delegate is very important, of course, which is the main point of edw519's comment, I think.

Good advice. However, more often than not, the devil lies in the details.

I bet the guy didn't mention YouTube just to avoid giving competitors "free publicity". Just another example of how fixated his mind is on legacy business/marketing practices...

Yes, thanks for reporting!. It's fixed in the version we deployed today.