jhartmann

This has been true for years, Intel CPU's can't efficiently perform math operations when hyperthreading is involved. Generally there is only a single shared FPU/AVX/SSE unit doing the math over two hyperthreads. Since the Eigen implementation often can keep that unit 100% busy, it makes no sense to try and run two threads at full tilt through the units.

I tested all this very heavily before Eigen had AVX-512 support. In that environment there might be some differences and I would suggest you benchmark both configurations.

Performance of DOM based rendering is very problematic and not unified across browser implementations. Canvas rendering will likely increase the performance of Google Docs, and make the UX more unified across platforms. Google Docs is really an application built on the web platform. HTML DOM rendering was never intended to give developers the control they need to build fully featured high performant applications, we just shoe horned things until they sort of work. I think this is a positive thing, UX will be better and the integration API's will become much cleaner and not depend on structure of how they design the UI. Concerns will be separated and the end result is something much cleaner, more performant, and more supportable.

Could be interested in the right opportunity, very experienced dev and manager with Ankylosing Spondylitis who has been greatly helped by medical mj. Would be remote for now, not in LA area. Would potentially be open to relocate after the dust settles on the current situation. Email is in profile, can you send me some more info?

Depends on the cameras, but I would imagine Facebook and Google's tech + data to be able to positively identify 99% or more of the people. Especially with multiple shots per person. Accuracy might be as high as 99.9% if you have drones go over the crowds, and have high res cameras at chokepoints. Correlate this with cell phone data and probably they have 100% accuracy. There is a reason Google Fi exists.

I say all this as a happy Google User and a developer of Face Recognition Technology. Those in the know realized privacy has been dead for awhile now. Generally computers can do most anything we can do in say 200 ms or less with the proper training data, and they generally do it better than humans. The giants have all the data they need, they just need to spend time annotating training sets and running analysis over larger and larger sets until they reach super human performance.

I know of several hackers that have gone through YC have autoimmune conditions, and I would say there is a larger population of people that have autoimmune conditions then our parents generation. I myself have ankylosing spondylitis, and live with chronic pain, back and neck problems. Probably just really interesting to people like us, who want to hack our way out of problems that medicine doesn't have good answers for yet.

Regarding 1, there are systems @ Facebook and inside Google that can do this reasonably well and I know the technology intimately. I would consider this a nearly solved problem, I give it 1 to 2 years at most to be more generally available.

There is absolutely no reason to store anything for 100,000 years. If we were sane and reprocessed our fuel (like France), something like 400-500 years would put us below standard ambient radiation levels. Still a long time, but no we don't need to store anything for 100k years. Remember, almost all the long lived Transuranics are generally usable as further fission fuel. The other Transuranics and fission by products will typically decay rapidly to something stable or barely radioactive within a reasonable timeframe.

Last time I looked there wasn't a way to view on the web, that at least works for me. Appreciate the reply, good luck on Whale and Zero-F

Justin, I love the concept of Whale, but many of us don't have iOs phones. Could you run a more traditional AMA at some point?When will there be a web player or an Android app for Whale? I feel that many people in the early user base for Whale are being alienated here.

I use a tiny bit of python, a little more LUA, and a TON of C++ in the machine learning work I do. Things like opencv, fbthrift, folly, boost, fblualib, and thpp make writing this sort of code in C++ very time efficient and if you know what you are doing it will end up performing much better than the alternatives. I only use python for some light scripting, data collating and reformatting type tasks, and LUA due to using Torch as my Neural network framework of choice.

I'm not saying its easy, I'm saying a developer using the tools and processes available at a large internet corporation building something small need not be insecure. Basically until we look at the code and the security processes in place, or do pentesting, we have no clue one way or the other. I'm mostly just saying we shouldn't discount something like this automatically as insecure. It might be terrible, it might be very well thought out and have security be one of the most important considerations that was top of mind when he coded it.

He did reuse things that exist inhouse, like the face recognition piece. To rebuild a service with Google/Facebook/Microsoft/Amazon level accuracy in that task, it is definitely more effort than 100 hours by itself.

To use an existing private service, not nearly as much effort.

That being said: Of course he could have learned how to train a model himself, but even in that case he is using training infrastructure and data that is available to him that is not public. I highly doubt he had the time to build those sorts of things himself. But his goal was more talking to his home and using tools to build an interactive home assistant. Sounds like he was very successful at doing that. He wasn't out to build the individual AI pieces, doesn't make it not cool or interesting.

Actually, there are some ways to make this very secure that don't take much effort that in are common use around facebook. It is quite easy to build secure thrift services, and to design a thrift api it is just creating a text file with the necessary idl. You can then access the service from most any language by generating the right client. I'm sure same thing goes for gRPC and Stubby on the google side (more familiar with the fbthrift, folly, wangle, proxygen tech stack myself, but I'm sure its just as easy.)

I think you should also mention work queues and producer / consumer thread pools. These are very powerful concepts you don't mention here, and they are often the first tool I use. You very often don't need anything else.

Dunno about unicorn, but Brad Feld was an very early Angel investor in Harmonix music systems and they were working for many years before the had breakout success.

I wouldn't say no distributed object storage system is particularly tuned for that. Facebook's haystack and systems patterned after it can work very well for tiny files.

Doing the dev is the only way, I would recommend though reading the Hinton dropout paper and trying to implement a non full featured Neural Network framework to implement those ideas. Then pick papers to implement in your toy framework. Once you have done this a few times, you will get a really good idea how all this stuff works and ideas to extend things. It also REALLY helps if you have a very strong linear algebra background, so if your weak there do lots of reading and practice getting those skills up to snuff. I may not be the best example, because I worked in computer graphics heavily over my career. My suggestion though is to try and apply DNN's and ML techniques in somewhere you have domain experience. It really will help a lot with the intuition part since you have valuable domain experience to give you ideas to try.

This is a path I took around 4 years ago, and I have built some seriously valuable stuff in that time.

Also I would personally say to not discount the value of backend stuff, doing scalable ML processing is not off the shelf and lots of value can be created by making scalable reusable systems that can run machine learning models. Most of those ML PHd's can't (or don't want to) build scalable distributed systems. If you can train experiments faster, or actually do the work to make an end to end system work outside the lab, you can make real measurable contributions.

They use a system called haystack for binary large object storage. They very much don't use mysql for those.

Understood, but GPU's are very good at linear algebra. GPU's are HORRIBLE for crypto, just very parallel.

3 generations ahead of moore law??? I really wonder how they are accomplishing this beyond implementing the kernels in hardware. I suspect they are using specialized memory and an extremely wide architecture.

Sounds they also used this for AlphaGo. I wonder how badly we were off on AlphaGo's power estimates. Seems everyone assumed they were using GPU's, sounds like they were not. At least partially. I would really LOVE for them to market these for general use.

I'm just curious, why is it terrifying? Would a super intelligent human being produced by science be any more terrifying then one that occurred naturally? Couldn't we be creating a being that could push us forward collectively? Think about the contribution to modern life Einstein, John Von Neumann, and Bohr had. If we had a way to make very intelligent designer humans, there could be a chance they could push us collectively forward in a similar way. I imagine that an Einstein born now, with access to the whole worlds information could be much more impactful (or maybe he would just succumb to liking cat videos.) All humans designer or not can be terrifying, beautiful, impactful, and other worldly. You don't have to be unnatural to be an evil bastard, and maybe an engineered human could be like Gandhi. Don't prejudge, just because its unnatural doesn't mean it is not good.

Just like GMO doesn't mean bad (I eat organic food by the way), a lot of this goes to intent. If science wants to push us forward and make us better. I'm all for it. If they want to engineer us to be resistant to industrial chemicals or pollution so that companies can pollute more, or make super soldiers that is when you have to be terrified . Not just because we have something artificial.

My main questions was why always store the same blocks together, and you definitely answered that.

Interesting approach. It definitely achieves your goal of spreading out the load. I think this can be achieved without making sure you store the blocks always together though, and it might help move towards a less coordinated system.

Here is how I solved the problem:

You can arrange each instance (you call it cell in your post) so it has different pools of storage. Maybe something that might make sense is each machine at the top of each rack is one pool, all the next machines down is another pool, and so on. All replication is always done across a pool. All the machines in a pool are stored to in a round robin fashion at the client. The end user can choose how many pools they want to replicate over.

Abstract each of the machines as a storage unit. Each storage unit actually contains a series of 'slabs' which are largish files (32 gb in size generally), that grow in some large increment and close out when they hit a high watermark. Each write to a machine goes to a different writable slab in a round robin fashion. Each server knows the path and the contents of each slab, and on disk failure all the individual pieces are replicated to another set of open slabs just like the blocks came in fresh (with an exclusion of the slabs that already have a copy). Since all requests are balanced across the pools, and each client has a different round robin list it manages locally the individual pieces are evenly distributed over the other machines in the other pools.

I tried really hard though to optimize for a zero master situation, so my clients handle contacting all the different machines and doing caching and retrying. So it sounds like I had different goals than you guys did, so I'm not surprised your have different trade offs.

Thats very interesting. When you post about that could you go into why you choose that over say large files (say 32 gb or so) that contain each of the collection of blocks? XFS or Ext4 can preallocate large chunks of the files beforehand, so the data looks nice on disk. A fully custom block level device sounds like a nice way to minimize overhead, guess it goes to show that extreme scale makes certain decisions that seem crazy from a support point of view start to make sense.

I believe a team of one or two talented individuals can build something that scales well and that will save a HUGE amount of money once you get into any significant portion of a petabyte with your data. One or two months of S3's list price can easily pay for the cost of hardware. You can easily go with whitebox hardware from say supermicro, and colocation is SOOOO much less expensive. If your not storing a significant portion of a petabyte, its going to probably be cheaper development wise to just use S3. You also don't need to build something as complex as their system, they have to scale to exabytes. Unless you plan to hit that scale immediately or have that much data, you can build for something less complex and smaller to start.

First thanks so much for doing this, I really appreciate it. I only did a quick read of your post apologies if it addresses these comments/questions. I recently built a system for image storage and made some similar but different tradeoffs so I'm very curious about some of the motivation behind your decisions. Definitely would be cool to go into them in a follow on post.

I'm curious why you choose to what seems like an identical bucket stored on many nodes. I have recently built a specialized system that resembles facebook's haystack for storing several petabytes of images. I think it is way more simple and efficient to precreate all the possible 'slabs' on each machine, and just place the data in a round robin fashion over all open slabs. You can periodically grow these with system calls to allocate blocks of disk to make it efficient, maybe several 100's of megs at a time when running at scale. That way each server makes its own placement decisions, and no master is necessary to figure out data placement. Especially since you went with a mysql architecture, I think this makes a lot of sense. I guess it does make things more complex on the erasure coding side, since each block would then have to be erasure coded individually. It might be worth exploring though, since you could completely eliminate the requirement of a master.

Of course my system is considerably simpler since my smallest atomic unit is a whole file (keyed by sha1 hash + length to deduplicate), like facebook's haystack it was designed primarily to be an image store. I also do full replication rather than erasure coding, and only plan to implement that later.

One thing you didn't go into very much that would be very interesting would be the on disk format of the bucket. I used a modified version of folly RecordIO that returns the disk offsets and stored my payload using thrift with the sha1 hash and some flags. This allows for both quick indexing to each block and a nice recovery story. I'd also be interested in what sort of index format you are using, I store mine in both mysql and in a google sparsehash table by record token. Storing the index locally has great benefits if the mysql server goes down, I'd be curious if your system does that as well.

A reasonable compromise is to use a threadpool (which is a big part of what comes with wangle) with a fixed number of threads and you queue the tasks to be run by the threads. It works quite well in practice, and you can use coroutines (I'm most familiar with the implementation in boost coroutine and the Ewan webserver personally) to go the whole hog and get something as lightweight as Erlang green threads or goroutines. I would only bother doing that of course if in your particular use case you find the threadpool doesn't perform well. With proper sizing and broken up units of work, it really works quite well in practice.

The wangle native way of doing this is just to use an IOThreadPoolExecutor: https://github.com/facebook/wangle/blob/master/wangle/concur... Just make a block of threads to run your db tasks, and you can use a promise/future to return the results back to the async code.

I have used this library pretty heavily in some of my recent work and I love how easy it makes building performant async network servers in C++. Minor nitpick: It would be nice though, if they built shared libraries by default though, last release I took I had to tell cmake to make a shared library. Since this is used by fbthrift as a dependency, I think it really makes sense to build it that way by default. I also use fblualib and it builds with a very old version of some of these libraries probably because the FAIR team didn't want to mess with tweaking the builds so they can work with standalone wangle (parts of the library used to be part of folly).

This is from a Freedom of information request made by Brian Wang of nextbigfuture. His article on the subject: http://nextbigfuture.com/2016/04/2013-independent-review-of-...

Seems very interesting, I really think these approaches will work rather well on a variety of games. Very exciting times. Btw, there is another already Open Source Neural Network Chess engine that might be worth exploring for similar ideas. That codebase already has a minmax search based on the network evaluation. Paper is at: http://arxiv.org/abs/1509.01549 Code is at: https://bitbucket.org/waterreaction/giraffe