jd20

I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?

I'd really love to see more data on whether ergonomic keyboards actually work. From what I've read, it sounds like the results are mixed: I kind of want to try a split keyboard like the ergodox or Kinesis, but I feel I tend to cross-over a fair amount when typing, and I wonder if a split keyboard would be less efficient.

I also overthink a lot about the position of frequently used keys like Cmd/Ctrl/Alt (on a Mac for instance), and what the optimal placement would be, and I feel like there's very little data about this topic.

One does (it's a southpaw, numeric keypad on the left). A few are TKL (full size keyboard without the numpad).

Actually, as a programmer, I pretty much never use the numeric keypad. But when I start seeing smaller layouts with no arrow keys, Fn keys, or even number keys, I tend to agree: there's a definite trade off between function and aesthetics. The beauty of custom keyboards is people get to decide those trade-off's themselves.

A clarification: these are interviews with people who assemble custom keyboards, I was expecting chats with the people who actually design and produce custom keyboards (like yuktsi, Rama, Wilba, ZealPC, etc...)

Still, very cool to see what people are building. I've just recently fallen down into the rabbit hole of custom keyboards, after my Apple Keyboard stopped working. As someone who spends almost half my life at a keyboard, I'm surprised it took this long for me to look into improving the tool I interact with most every day.

It's possible that wasn't the default setting on Macs back then. I don't know that cgo would be a good choice either, if you're resolving a ton of domains at once. Early versions of Go would create new threads if a goroutine made a cgo call, and an existing thread was not available. I remember this required us to throttle concurrent dial calls, otherwise we'd end up with thousands of threads, and eventually bring the crawler to a halt.

To make DNS resolution really scale, we ended up moving all the DNS caching and resolution directly into Go. Not sure that's how you'd do it today, I'm sure Go has changed a lot. Building your own DNS resolver is actually not so hard with Go, the following were really useful:

https://idea.popcount.org/2013-11-28-how-to-resolve-a-millio...

https://github.com/miekg/dns

Trash cans :)

Nope, you don't really need a database. What you need for fast, scalable web crawling is more like key-value storage: a really fast layer (something like RocksDB on SSD) for metadata about URL's, and another layer that can be very slow for storing crawled pages (like Hadoop or Cassandra). In reality, writing directly to Hadoop/Cassandra was too slow (because it was in a remote data center) so it was easier to just write to RAID arrays over Thunderbolt, and sync the data periodically as a separate step.

Some Apple services were written in C/C++. One downside is it's very hard to source engineers across the company who can then work on that code, or for those engineers to go work on other teams.

Everything was on 10gigE. The average page size was around 17KB gzipped. Everything's a careful balance between CPU, memory, storage, and message throughput between machines.

Apple's corporate network also had incredible bandwidth to the Internet at large. Not sure why, but I assumed it was because their earliest data centers actually ran in office buildings in the vicinity of 1 Infinite Loop.

You should check out Manning's "Introduction to Information Retrieval", it has far more detail about web crawler architecture than I can write in a post, and served as a blueprint for much of Applebot's early design decisions.

I think WebObjects was supporting Java even before it came to Apple from Next. In the early days, many of Apple's services built with WebObjects even ran on Sun server hardware, and XServe's. But nowadays it's all commodity Linux hardware, like you would find in any data center.

No idea, it's been years since I last worked on it. It was also not the only Go service written at Apple (90% of cloud services at Apple were written in Java), though it may have been the first one used in production.

Some fun facts:

- Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it's Go origins to the world, which Russ Cox fixed the next day).

- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.

- In it's first week of existence, it nearly took Apple's internal DNS servers offline. It was then modified to do it's own DNS resolution and caching, fond memories...

Source: I worked on the original version.

If Apple Maps has taught us anything, probably not. But Apple would first need to pour an equally large amount of resources into web search, they way it did for Maps.

Applebot was built for crawling web pages, to be used for search results in Spotlight and Siri. That user agent might also be used for attachment previews, but the original intent of Applebot was for search indexing.

Apple built their own search engine over 5 years ago, under the Siri / Spotlight umbrella. When people talk about Apple building their own search engine, they generally seem to expect a website dedicated primarily to web page results, but under the covers what powers Apple's Spotlight results is basically a search engine.

The big question would be what Apple would gain from a dedicated website for search results. Would people really switch to it from Google? Why would it be a better delivery mechanism for search results than Spotlight? Not sure the answers to these questions has changed much, from 5 years ago to today.

Thanks for clarifying, I figured I was generalizing it a bit.

The phone boots into an operating system known as “Switchboard,” which has a no-nonsense black background and is intended for testing different functionalities on the phone.

I think the article confuses the meaning of "dev-fused" hardware, with what OS is actually installed on the phone. When I used to work at Apple, I always understood "dev-fused" to mean a device on which you could install unsigned builds of iOS.

Internally, Apple puts out new builds of iOS daily. The engineers building features on top of iOS need to install these builds, to do their work. A normal iPhone from a store won't take these unsigned builds, hence the need for these dev-fused devices. There are regular builds like what a customer would get, debug builds with lots of logging and debugging checks enabled, and even bare-bones builds like switchboard, for employees who are not UI-disclosed or work in factories. As someone building higher-level iOS features, all my dev-fused devices just ran a normal looking iOS, unlike what the article describes.

Two people showed Motherboard how to get root access on the phone we used; it was a trivial process that required using the login: “root” and a default password: “alpine.”

Oh boy, that sure brings back memories!

I've found the same to be true for Uber Eats. If my order is wrong, they give a full refund no questions asked (and a meal is typically $20-25). And orders get messed up a lot (maybe 1 out of 5). Even something as small as missing ketchup, I'll get half off.

Made me wonder how likely it was that "Funded by YC" was one of the criteria used by the analyst to come up with the list.

Would also love to hear more about how this is done... I'm in the process of setting up Stripe subscriptions on a website, but have been scratching my head over the best way to accomplish everything without setting up my own back-end server.

One unfortunate side effect of being at a FAANG company is that most of your best work is probably proprietary and you will never have the chance to show it to anyone outside the company, nor will you have much free time for personal projects.

As an interviewer, I can definitely say that GitHub has become a bigger part of helping to evaluate people.

Not sure why it's not possible, though I could've had a defective unit. Specifically, I had a Fiio D3 ($20) which had bad background noise, even when not playing. I always assumed it was a power supply or shielding issue. The audio output from it actually sounded decent, just I could always hear the background noise. Don't think it was a grounding issue either, because it was the only DAC I ever had that consistently did that, no matter what system I plugged it into.

I've played around with the DAC built-in to the Raspberry Pi 3 as well, and that one was more of a distorted output problem, similar to what you describe with the $2 DAC.

Other DAC's I've tried (from around $75 and up) I've never heard noise / shielding issues, so again rapidly diminishing returns once you get past the super budget options, in my experience.

There's a sharp curve of diminishing returns, for DAC's especially. The super cheap units you find for like $10-20 will usually have serious shielding / noise issues, which is immediately apparent upon listening. Beyond $100-200, the design and components are usually pretty competent, and what you are mainly hearing is just how the implementation was "tuned" by the manufacturer (not necessarily better or worse).

Playing from an iPhone or iPad is perfectly fine. Make sure EQ and Sound Check are turned off. Most music streaming formats are nearly impossible to differentiate from lossless (i.e. Apple Music, Tidal, Spotify), so I wouldn't sweat that.

It's not a new idea, really. I remember seeing Nagra come out with a battery-powered preamp for home stereos, many years ago (and I think they still make them).

I also remember hearing back in the day of some audiophiles powering their systems with regenerating UPS from APC (like the kind you'd see in a data center), though PS Audio has been making such units marketed for audiophiles for a while now too.

Looks cool! The problem of enumerating the best routes between two given airports is indeed an interesting problem. My favorite discussion of the problem is from Carl de Marcken: http://www.ai.mit.edu/courses/6.034f/psets/ps1/airtravel.pdf

I've been involved in solving a similar problem, on the award flight search side. There are about 12,000 direct commercial routes offering award redemption, and the airline websites have different limitations: some only let you search one-way or only round-trip, or only one cabin at a time, or only give hints about available quantities. So you have to formulate the minimal set of queries that will give you complete information. And then given an origin and destination, figure out the valid routings, like you've discovered.

The other really interesting challenge with many of the airline websites are the extreme measures they go to, to block you from crawling their data. My favorite is British Airways, which runs 800 different tests to fingerprint your browser, from generating audio files to drawing to an off-screen canvas, all inside a custom VM with encrypted byte code. (Similar to how I've seen Google obfuscate the ReCaptcha code base.) Surprising to hear that Ryanair makes it so easy to access prices in comparison!

I'm surprised Ryanair is supported, given that they've been engaged in a long and nasty lawsuit with Expedia, over scraping their fares. I was under the impression Ryanair only wanted to sell fares through their own website.

Kind of OT, but I agree phosphate pollution's a big problem and there's a lot of measures people can take to wash their cars responsibly. I've switched to just a bucket and low-flow pressure washer, which uses about 30x less water than a garden hose (and even less than my shower head, and unlike my shower, only runs when I pull the trigger). As a result, I have no run-off from the driveway, and what little water does get used is mostly recaptured and used to water plants. Honestly, at this point I think I use more water to shower, and I definitely take far more showers than I wash my cars.

Driving through Cupertino neighborhoods, I've seen several homes with not one, not two, but three(!) Tesla's in their driveway.

When I wash my Model S in the driveway, people walking by will often come up to tell me how much they love their own Tesla's (or wanting to know how I like mine). Never had that happen when I wash my Audi :) The Audi is even the more expensive car.

Same here. It seems to check all the authors needs, other than being constantly updated.... But then, if it's not broke, not sure what need there is to keep pushing updates?