etrain

There was a company called Petridish that did exactly this. They pivoted after six months and became Blue Apron: https://vator.tv/news/2017-09-05-when-blue-apron-was-young-t...

Users expose their model to our Trial API (https://docs.determined.ai/latest/topic-guides/model-definit...), the base class then implements a training loop (which can be enhanced with user-supplied callbacks, metrics, etc.) that has a whole bunch of bells and whistles. Easy distributed (multi-GPU and multi-node) training, automatic checkpointing, fault tolerance, etc.

Concretely, the system is regularly taking checkpoints (which include model weights and optimizer state) and so if the spots disappear (as they do), the system has enough information to resume from where things were last checkpointed when resources become available again.

Check out Determined https://github.com/determined-ai/determined to help manage this kind of work at scale: Determined leverages Horovod under the hood, automatically manages cloud resources and can get you up on spot instances, T4's, etc. and will work on your local cluster as well. Gives you additional features like experiment management, scheduling, profiling, model registry, advanced hyperparameter tuning, etc.

Full disclosure: I'm a founder of the project.

Tom7’s sigbovik submissions are true works of art. https://www.cs.cmu.edu/~tom7/mario/mario.pdf

The exposition is tight.

Relational database queries are supposed to be _declarative_. As a user, you're not supposed to think about the mechanics of execution because the database system is supposed to be able to decide how to execute your query using whatever magic it wants, as long as it satisfies the contract that it gives you the right answer.

It's absolutely useful as a debugging tool to build up some semantic understanding of what the query means, and I encourage every database user to learn to use EXPLAIN, but relying on a mental execution model is borderline dangerous.

Assume 100 pages on each onion address (it’s probably power-law but let’s just assume that’s the mean). Latency with Tor is super high. Assume average of 5s to load a single page. This is generous because tail latency will probably dominate mean latency in this setting.

These things can happen in parallel but let’s also assume no more than 32 simultaneous TCP connections per host through a Tor proxy.

So we’re looking at ~75k1005/32 seconds = 14 days to run through all of them. You may not need to distribute this but there are situations (e.g. I want a fresh index daily) where it is warranted.

Yes, this is correct: http://composition.al/lindsey_kuper_cv.pdf

It’s also hard to separate the design of the neural architecture from the definition of the feature extractor.

I had one of these phones too - it was a bad batch of serial numbers according to Apple. If this happens to you: https://www.apple.com/support/iphone6s-unexpectedshutdown/

Crunchbase has a good deal of data here - it is incomplete particularly for smaller companies, but is pretty good for larger investment rounds.

A word of caution - these talks are highly technical and theoretical and I don't recommend them for anyone looking to understand practical applications of ML. If, however, you're interested in intro to cutting edge work at the frontier of theoretical underpinnings of ML, this is a fabulous group talks from some of the sharpest academics in the field.

We've used this dataset to build a product review classification pipeline as an example application that can be developed using our project, KeystoneML (which runs on spark) - code is here: https://github.com/amplab/keystone/blob/master/src/main/scal...

You might find the OSDI slides useful: https://people.eecs.berkeley.edu/~apanda/assets/slides/osdi-...

I would also be interested in seeing this.

Microsoft is 23 years older than Google and it usually takes time for a corporation to create an R&D lab. Given Google's rich academic roots this happened faster there than usual. Google is doing a good job given that it's a relatively young company. I hope their research output continues to increase in quality and volume.

If you go by publication count alone, I suspect that IBM Research is still near the top. They have a large research organization that made some fundamental contributions in computer science in everything from speech recognition to databases.

This is pretty awesome. One key bit of information that the compiler has is that the coefficients are a constant array of length 12, which makes the loop unrolling possible and also means that the register magic is in play - it's seriously awesome that the compiler does this.

That said, I'd expect something similar to happen with a well-written C program. Would equivalent abstractions in C++1{1,4,7} be "costly"?

GPUs in the cloud aren't targeted at gamers. They're targeted at people doing things like running render farms and training deep learning models.

Keep in mind that with EC2 you're billed hourly so the fastest a 70 cpu-hour job could finish on m3.medium for $0.70 is 1 hour, and that's ignoring setup time, etc.

Meanwhile, on Lambda, you can actually run 1600 60s jobs (or 27 CPU-hours) in 3 minutes. This is inclusive of setup time, job submission, stragglers, etc. [1]

Of course, if you've got sustained load, it's cheaper to go with spot instances, but the "occasionally I need a buttload of compute," model is well-served by Lambda.

[1] http://ericjonas.com/pywren.html

One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals.

Assuming 70 hours at $0.000000834/100ms [1]

The whole job costs $2.10.

[1] https://aws.amazon.com/lambda/pricing/

This varies highly depending on the organization. I've worked at places that do it both ways. Particularly in small places when resources are tight, teams are distributed, or there is a non-code safety measure in place (e.g. Business decisions are made ex-post of the code running dependent on the quality of its output), code reviews pre-deployment may not be necessary and can even be harmful (due to wasted resources).

If I were trying to eat that diet and not break the bank/use all my time I'd invest in 4 things: 1) A ricecooker. 2) A food processor. 3) A sous-vide/vaccum sealer. 4) A Costco membership.

The ricecooker makes rice and keeps it warm all day. The food processor makes quick work of chopping veggies.

You could pack the cod/chicken in vacuum bags and freeze it, and then pop it in the water bath when it's time to eat. The sous-vide means you're not worried about over-cooking/attending to the process so you can do other things while your food cooks.

Costco will sell you high quality cod and chicken at ~$10/lb (probably less in store) and steak around $20/lb. You could similarly buy rice/potatoes/fish oil there in bulk.

Admittedly you're still spending $50+ on food/day, but that's not out of the question for someone earning $150k/year.

That said, it is probably much easier when you're earning $150m/year and can afford a private chef.

If you're willing to try something more bleeding edge: http://btrdb.io/

type systems are great.

Going rate for humans breaking captchas is like $1/1000 captchas solved. A fully automated service could make some money, sure, but not exactly a killing.

Short answer is actually: maybe!

The bulk of the work done in this code (in terms of FLOPS and, likely, wall-clock time) is going to be in BLAS-3 operations in the feed-forward and back-prop steps. That is, almost all of the work is done using Matrix-Matrix multiplies and in-place arithmetic/transcendental functions.

CUBLAS[1] will allow you to run these types of operations on your GPU at highly accelerated rates, without much more effort than replacing your BLAS library with a new binary. Additionally, if you want finer granularity control over what gets done on the GPU, there are other libraries[2] which provides a direct interface to CUBLAS.

[1] https://developer.nvidia.com/cublas [2] https://github.com/JuliaGPU/CUBLAS.jl

I published a paper this year that used it directly and even proposed a modification of it to model job time for a distributed execution environment with a centralized master governing scheduling, task serialization, etc.

When you talk scale-out of any large system, or manycore whatever, Amdahl's law is absolutely fundamental, and when you hear someone say "perfect horizontal scalability," Amdahl's law governs when that will no longer be true.

People who do theory on distributed algorithms use it even if they don't realize it.

Fiduciary Duty.

Spanner maxes out at ~10 transaction (batches)/second in that paper. By comparison, a standard PostgresSQL installation (on littleish iron) is capable of 50k TPS depending on the benchmark you're running.

In the Spanner design the maximum throughput is governed by the accuracy of GPS and atomic clocks and, ultimately, the speed of light. You can only make 7 round-trips per second between a data center in New York and on in Singapore unless you're willing to bore a hole through the center of the earth.

Order is expensive and, more fundamentally, coordination is expensive. At the center of traditional OLTP workloads is a promise given by the system to the developers that their transactions will happen in some order.

Peter Bailis and Joe Hellerstein are two prominent academics working on addressing this problem, and at the core of their research is giving up pieces of this promise.

I'm a researcher in the same basic field, and I too am pretty good at this "bullshittery."

However, Philip trivializes the "10,000 hours" spent learning this stuff as merely learning how to cope with this interface. This is so far off it hurts. Those countless hours spent messing with free software is how I learned to use other people's work, compile it, read it, fix bugs in it, and learn how other people think and write software. This is an incredibly important educational experience and there's sometimes no substitute for time and patience.