HN user

lil_tee

2,483 karma

https://toddwschneider.com

Posts0
Comments18
View on HN
No posts found.
[GET] "/api/user/lil_tee/stories?hitsPerPage=30&page=0": 500 Failed to fetch user stories
To Keep Track of Reddit Conversations Around NYT Articles, We Built a Slack Bot 7 years ago

nytimes.com is #3 domain of late, really #2 behind only github.com if you exclude ycombinator.com (all of the Shows, Asks, etc.)

Top 10 domains by # of items on front page since 1/1/18:

   rank |     domain      | count 
  ------+-----------------+-------
      1 | github.com      |  2041
      2 | ycombinator.com |  1911
      3 | nytimes.com     |  1818
      4 | bloomberg.com   |  1028
      5 | medium.com      |   826
      6 | techcrunch.com  |   735
      7 | theguardian.com |   666
      8 | github.io       |   615
      9 | bbc.com         |   558
     10 | arstechnica.com |   493
0ThreadHN
Analyzing 1.1B NYC Taxi and Uber Trips 11 years ago

Yes, the database is all on the machine's local 512 GB hard drive. I did store the downloaded flat text files to an external drive and loaded them into the db from there.

0ThreadHN
Analyzing 1.1B NYC Taxi and Uber Trips 11 years ago

Sure, there are undoubtedly lots of examples of businesses that opened in desolate areas and created new taxi activity where there had been none previously, I just happened to focus on Williamsburg for the post.

Another idea that I didn't get around to doing was to look at concert venues and measure taxi traffic around particular concerts to see if it would correlate to bands' overall popularity

0ThreadHN
Analyzing 1.1B NYC Taxi and Uber Trips 11 years ago

Yep, all on a 2012 MacBook Air. Data size was over 400GB with indexes

Simple queries on indexed columns of the trips table take a minute or two, more complicated queries that require a full sequence scan can take up to a few hours

0ThreadHN
Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails 13 years ago

Simulation author here with some additional analysis using a faster distribution of request times. If you use a distribution with median 50 ms, 90th percentile 225 ms, and 99.9th percentile 898 ms, then you need 30 intelligent dynos to handle 9000 requests/minute without queueing. In the same scenario with 30 naive dynos, 44% of requests get queued.

Animations and results are in the explanation at http://rapgenius.com/1502046

0ThreadHN
Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails 13 years ago

I'm not familiar with minimum-of-two-random routing, but it does seem like assigning request to dynos in sequence would perform much better than assigning randomly (ie in a scenario with n dyno capacity, request 1 => dyno 1, request 2 => dyno 2, ... request n => dyno n, request n+1 => dyno 1, ..., repeat)

That'd be probably significantly better than the case of (request i => dyno picked out of hat) for all i

0ThreadHN
Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails 13 years ago

But you never want to have a queue on any of your dynos! A queued request means that a user is waiting with no response. If your goal is to have 0 (or less than epsilon) requests queued, it takes far fewer dynos if the requests are routed intelligently

If you have 10 dynos and 1000 simultaneous requests, the difference between naive and intelligent might well be reduced, but that's also a scenario in which your end user response times would be horrendously slow and so you'd need more dynos either way

0ThreadHN