stuntprogrammer

Secret sauce, at least for the team involved, and well-received. Nice combination of extreme perf and decent productivity for them.

I've moved into non-finance stuff for quite a while now though, so not sure what's become of it. Given business challenges in that particular sub-field, who knows..

Not kdb+, but a proprietary (internal-only) language that it heavily influenced was designed around execution on GPU clusters.

The FPGAs were used mainly for feedhandlers and there was a different DSL for that (compiling to verilog).

It was indeed rather something to see :)

Yep, he said while eyeballing a dashboard, there are plenty of larger private machines..

And you too sir!

If the combination of such languages, high-performance hardware, and large scale compute problems is interesting.. the startup I work for in Mountain View is hiring...

Indeed; as I've heard it ordered: 6 parts gin and a moment of silence for the vermouth.

The problem is less the spurious DRAM accesses etc, as awful as they would be. The compiler problem is really a mix of 1) understanding enough about fixed-bound unit-stride loops to nonoverlapping memory (or transforming access to such) and 2) data layouts that prevent that. E.g. while there are well understood data layouts at each point of the compilation pipeline, it's hard in general for compilers to profitably shift from array of structs to struct of array layouts.

You are correct that, generally speaking, most STL heavy code would be hard to vectorize and unlikely to gain much advantage. (Plus there are the valarray misadventures). You will sometimes see clang and gcc vectorize std::vector if the code is simple enough, and they can assume strict aliasing. Intel's compiler has historically been less aggressive about assuming strict aliasing.

Various proposals are working through the standard committee to add explicit support for SIMD programming. E.g. if something like http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n418... were to be standardized we could write matrix multiply explicitly as:

  using SomeVec = Vector<T>
  for (size_t i=0; i<n; ++i) {
    for (size_t j=0; k<n; j+=SomeVec::size()) {
      SomeVec c_ij = A[i][0] * SomeVec(&B[0],j, Aligned);
      for (size_t k = 1; k < n; ++k) {
        c_ij += A[i][k] * SomeVec(&N[k][j], Aligned);
      }
      c_ij.store(&C[i][j], Aligned);
    }
  }

Current publicly announced AVX512 does not support fp16. Skylake Server (SKX) and Knights Landing (KNL) are at a disadvantage here. They've not publicly said anything about extensions in Knights Hill (the long announced successor to KNL).

That said, Intel have announced the emergency "Knights Mill" processor jammed into the roundmap between KNL and Knights Hill. It's specifically targeted at deep learning workloads and one might expect FP16 support. They had a bullet point suggesting 'variable' precision too. I would guess that means Williamson style variable fixed point. (I also guess that the Nervena "flexpoint" is a trademarked variant of it).

I assume the FPGA inference card supports fp16. And Lake Crest (the first Nervena chip sampling next year) will support flex point of course. I would expect subsequent Xeon / Lake Crest successor integrations to do the same.

Fun times..

Aside on the compiler work -- I think it's not that hard to emit this instruction at least for GEMM style kernels where it's relatively obvious.

An important detail is that this is not 130 petaflops of double precision floating point. Their target is >130 single or even half precision.

One tidbit is Google themselves saying they'll roll skylake out in cloud early next year.

https://cloudplatform.googleblog.com/2016/11/power-up-your-G...

I always recommend these:

https://scalableinformatics.com/assets/documents/Unison_Peta...

Surprisingly cost effective given the massive performance and support. Cloud is great for some things but not everything.

Based on random sampling of those that contact me about such transitions, the latter is somewhat true. A couple have mentioned Julia but I wouldn't say it's the majority.

Bill

Fixed; added to 'about' as well as email field.

I shouldn't say too much. But more than one large firm has explored writing an in-house replacement and floated actual $ offers.

Occasionally someone pops up wanting to do a company to compete with them but they're typically offering paper of dubious value, covered in slime.

In any event, I'm not particularly interested in going to war to steal their market share. I do worry that they're going to go into a long decline to irrelevancy though -- I'd very much prefer that not to happen. I'd rather they/FD write a new generation building on the best of arthur's work plus some things more suited for new platforms and workloads. There are big opportunities there.

For his sins, Fermin got dragged into the q/kdb+ world. I assume he's still using it at MS :-)

Drop me a note (email in profile). I'm not a recruiter, these are the offers I've been contacted with, through my personal network based on past gigs.

It's not just knowing an APL though -- far from it. For the chi/nyc it's a mix of hardcore low-level stuff, and some market knowledge. For the west coast, a mix of low level stuff (less hard core) and more distributed systems stuff.

It's not obvious to me that an open core business can sustain the necessary margins to be interesting as an engineering company rather than a glorified services business. There are been very few examples (friend of mine argues it's just RedHat).

You are correct though that their model over the years has been to extract a large amount, up front, from a small number of users. They (mostly FD) have failed to make the leap to users outside finance due to what I'd call cultural reasons. They also lack strong technical leadership imnsho (they're, at heart, not an engineering company).

I'd certainly take a swing at doing an open source version for them but not clear to me that they'd know how to play it.

A lot of the decisions that some might quibble with are not actually Arthur's end of things. The former CEO drove a lot of it.

In the interests of transparency..

That sounds like the low-end base salary for ~entry level. It probably gets topped up with ~100k bonus. There are a couple places that hire many but don't pay well.

At what I think is the high end, I see about an offer per quarter of $500-900k base with $1-2M bonus (usually guaranteed for the first year) for pure technology role. Add more front office work and the bonus potential shoots up (but it's a tough gig at the moment at least).

For context, non-kdb principal level roles I've seen on west coast top out at around $1M, mostly in RSUs, from a say a $250k base at the usual names.

I've seen offers in the $1-10M range on the east coast to build competitive technology. Amusingly, on the west coast, I've only seen the stereotypical "be my technical cofounder and get screwed on comp" offers to do so.

Technically, yes, a teraflop is a teraflop, and is directly comparable. It just means you can do an awful lot of floating point operations per second. But many systems are sensitive to memory size, memory bandwidth, and as a result communication costs (i.e. latency/bandwidth of the interconnect between machines).

The benchmark is essentially bottlenecked on FP64 matrix multiplies. If that's what you need to do, then sure, it's indicative.

Some machine learning workloads are also bottlenecked on matrix multiply, but don't need FP64 precision. They can use FP16. Fits a bigger model in a given memory size, makes better use of memory bandwidth, and given the right hardware support, you can get extremely high rates as on Pascal.

Personally, I find the memory system on Pascal more interesting than raw flops rate. Also, the use of nvlink to link multiple GPUs..

Pascal won't be cheap so comparing to a top-end Xeon E5 v4, it's about 7x the theoretical FP64 performance (assuming the Xeon is 2.2GHz * 22 cores * 8 avx pipes per * 2 for FMA per socket, given the price range). Similar story at FP32. However, the GPU wins on FP16.

For historical performance, just pick one of the machines that did x teraflops. E.g. the first teraflop computer used ~6000 200MHz pentium pro chips around 1996.

Essentially, no, at least not in the usual sense.

The "best" options are to either 1) short ETFs that have a suitable concentration in some subsector of the industry you think will be affected by falling unicorns, esp. leveraged versions of the funds or 2) short investors heavily concentrated in private unicorn investments such as NASDAQ:GSVC.

I strongly agree with you re 'conservation of distributed systems problems'. I'm being vague because I know what they're doing, but I'm limited in what I can comment on.

Their public comments are that they are running their own servers and not reselling other storage.

Personal opinion: no one can do (c) because of (a). That is, any possible (c) must tackle the fundamental hardness in the distributed systems problem, and this is what we agree (a) is doing. Using immutable objects as in S3 just shifts the problem elsewhere, while it reduces it, it doesn't solve it.

I agree with many things you say. Take AWS S3 as an example: it reduces the problem by making objects immutable. Internally, they can cache aggressively without scaling fine-grained consistency on the objects as such. As an S3 user I can aggressively cache on the client too.

Now we've reduced the problem to consistency on the metadata structure which aggregates objects for the user. There are "well known" ways to do this for traditional trees. Other well known options include doing an all search based approach (i.e. always talk to a server for metadata, perhaps with local result caching), and so on.

Correct

Yes, exactly.

(I've built things for a few chicago/newyork hft shops).

That's why the NDA is so frustrating; we can't talk re the fun features and demos.. I know the AMD and Cavium parts rather well also. The Amd one doesn't support what I have in mind. The Cavium does on paper but the cores, even with just public info, can be seen to be underwhelming. Typical network processor style, which struggles with other workloads. Unfortunately for them, they picked up a couple of the less clued in Calxeda execs that don't understand SW or workloads very well.

Unfortunately, the only public information I recall seeing was the use of a quad Cortex-A15 based SOC with integrated dual 10G in a NAS box.

However, the real capabilities of the product line were far more interesting, with very cool demos up and running with awesome metrics. The server possibilities are huge... especially if you provide opaque optimized services in a cloud to user workloads running on the x86 side.

These are invaluable references -- I hope we have many people here familiar with them, or new readers appreciating them.

It's a shame that nothing quite so comprehensive exists for IO, as network and storage accesses, patterns and quirks are often more of a bottleneck than CPU, for many applications.