CPU-based algorithm trains deep neural nets up to 15 times faster than top GPU

https://techxplore.com/news/2021-04-rice-intel-optimize-ai-commodity.html

by gumby • 5 years ago

444 143 5 years ago

There are basically no details in the article or the linked one (inside the article). I'll keep an open mind but this is the sort of article that should make you roll your eyes. (◔_◔)

freeone30002• 5 years ago

It's legit. For the network types this works on, it's much less computationally complex. A lot of the networks we use now are based on the assumptions of fast matrix multiplication, so it's not universally applicable -- but there's a ton of networks this does work on, and there's a good indication that we can rephrase a model to solve the same problems in the hashtable space.

king_magic2• 5 years ago

I call BS until a paper & repo is released.

exegete2• 5 years ago

https://arxiv.org/abs/1903.03129

https://github.com/keroro824/HashingDeepLearning

yor1001• 5 years ago

44 Cores, and careful cache management. I'm wondering whether the GPU implementation has been similarly optimized.

taf2• 5 years ago

Interesting the golang version https://github.com/nlpodyssey/goslide

king_magic2• 5 years ago

I stand corrected!

exegete2• 5 years ago

Paper: https://arxiv.org/abs/1903.03129

Code: https://github.com/keroro824/HashingDeepLearning

Edit: Here is the new paper referred to https://arxiv.org/abs/2103.10891

Here is the new code: https://github.com/RUSH-LAB/SLIDE

The paper I linked above was the first paper of which this is supposed to be an improvement.

guywhocodes5• 5 years ago

This is the initial SLIDE paper, this is supposed to be an improvement on top of SLIDE. Having looked at and run the original implementation I'm a lot less skeptical than the average post here. It was quite fast already and should be considered a toy implemenation.

djrogers• 5 years ago

What do you mean by ‘toy implementation’?

gumby3• 5 years ago

Did you mean, "should not be considered a toy" or "this original was a toy/PoC and this new paper is much better"?

foerbert2• 5 years ago

I think the reason you're getting so many questions about the 'toy' statement is that often toy programs are simple in large part because they omit handling complex or difficult cases. I don't think I've ever heard the term applied to a complete problem that is merely implemented naively.

As a result, saying it's a toy implementation is making people think you mean the speedup is a result of simply handling a simple portion of the problem, rather than thinking you mean even a naive implementation is quite fast.

d110af5ccf2• 5 years ago

Is there anything preventing this algorithm (or a substantially similar one) from being used on the GPU?

0-_-0• 5 years ago

After a quick read of the paper, no. You could adopt this to the GPU (which would require the hashes work on groups of neurons instead of individuals) and might get a similar speedup. Locality sensitive hashing in fact seems like a primitive attention mechanism, with proper attention implementation you could get maybe even better results.

baybal22• 5 years ago

Just as expected.

Turning back to nineties, it's easy to see that almost all ventures into specialty hardware ended up with mainstream consumer CPUs catching up, and swallowing the niche in a few years time as computer science, and logic design advanced.

Very few computing tasks came out to be truly brute force demanding as people learned how things really work.

Multi-channel audio, and sound effects on CPUs — once thought to be impossible, are now everywher, even smartphone chips.

Getting there was very tough though. Very few people can write a software DSP, and accoustically correct mixers with real time performance even today.

gpderetta2• 5 years ago

AKA the Wheel of Reincarnation.

I do think that GPUs are here to stay for a long time though.

mcosta• 5 years ago

Yes, but how much until integrated beats separate card in $/performance?

Once I bought a separate i387 chip for floating point. Later it was inside the main CPU.

zelon882• 5 years ago

Yup. Controversial opinion incoming, but I actually can't wait for the same thing to happen to ARM. The RISC crowd so quickly forgot the purpose of the RISC architecture. Every time I hear someone tell me how the M1 is so fast compared to x86 I just get this picture in my head of a guy in a 1992 Civic DX with a 2 foot wing revving his engine at the Lambo parked next to him. [1]

[1] https://wccftech.com/why-apple-m1-single-core-comparisons-ar...

mhh__• 5 years ago

I think you're wrong but I upvoted because the article does raise a valid point.

JohnJamesRambo• 5 years ago

Lol what? Can you explain this view more fully? If anything the x86 currently has the two foot spoiler on it.

ummonk4• 5 years ago

That article is incredibly stupid. From a user perspective, “single core” performance is irrelevant. What matters is single thread performance, and multi thread performance of the entire processor.

nl5• 5 years ago

This article is... misguided.

They argue that we should be benchmarking saturated single-core performance, but then show in their chosen benchmarks that the M1 is very competitive, benchmarking just below an i9 9880and above a Ryzen 7 4750.

That's a super-fast chip! No excuses like "oh it's Apple's first CPU" are needed - it's right up there.

And in real world usage it's kind of a useless thing to benchmark. See their note on how they had to wrestle with our Cinebench R23 program to get it to accept the load (threads locked to 2, affinity needed to be set after initiating the run but before the benchmark actually started and needed to be reapplied after the first pass) and a cleaner execution would almost certainly be welcome. It would also allow us to test cores working at their full potential.

It kind of shows how these micro-benchmarks aren't very reflective of real-world use.

gumby3• 5 years ago

Turning back to nineties

And 80s (consider Britton-Lee).

And the opposite too: "mainframes are too complicated; we'll write an OS that does all IO in the kernel (e.g. Unix) and run it on minis that don't have channel controllers". Come the 2000s and every disk drive and even keyboard has a processor it it.

And edge-vs-core computing in the network...

The best part is hardly anyone reads the literature much less history, so if you've lived through a couple of cycles you can see the pendulum starting to swing back and, to mix metaphors, "skate to where the puck will be"

Ericson2314• 5 years ago

Amen to that. I do welcome the return of "spread out" computing. I agree ease of use is important, but putting everything the CPU microcomputer style makes for a very imperative notion of ease of. Ergonomics without that would be much more elegant.

SatvikBeri6• 5 years ago

It's interesting because in some ways, GPUs are much less specialized than CPUs – GPGPU programming mostly involves treating the GPU like a huge, dumb collection of very simple processors. I expect effects in the opposite direction as well, i.e. that programming languages get better at using GPUs to do a wider set of workloads. For example, I was recently able to prototype an ETL pipeline with no machine learning or matrix multiplication to a GPU and get a ~10x price/performance improvement, which would have been a lot harder even 5 years ago.

lumost4• 5 years ago

In practice it's pretty tough to make this work. If your ETL can't be turned into just math on a <12GB dataset then you will be contending with streaming data from io -> CPU -> GPU -> CPU efficiently. CPU's are really good at multi-threaded io given an evented runtime these days, but GPUs can only do 1 thing at a time. This means that your application will need to batch data to the GPU.

So the real race is between how good you can be at batching vs. parallel dispatch when reading disk from I/O. As parallel dispatch is a more general problem it tends to get more attention.

tapirl• 5 years ago

CPUs will have more and more cores to reduce the gap.

Maybe future CPU and GPU and TPU will merge as a new kind of compute unit.

ummonk4• 5 years ago

Yeah I disagree with the comparison of GPGPUs to specialized hardware. It’s more a distinction between hardware optimized for throughout of general highly parallel computations and hardware optimized for latency of highly serial computations.

einpoklum• 5 years ago

I would suggest a different metaphor, since these days, CPUs also have a bunch of cores:

* GPUs invest more chip real-estate in integer and floating point crunching (ALU, FP).

* CPUs invest more chip real-estate in control logic, prediction and speculation.

Not that these are the only hardware differences of course, like how caching works, and larger vs higher-bandwidth memory etc.

jmartrican• 5 years ago

Genuine question here.... why are GPU's still used for graphics in video games? Is this one use case where the CPU will never catch up or has yet to?

SatvikBeri6• 5 years ago

Oversimplifying a bit, CPUs are smart and can do a lot of different things, and a lot of their chip area is devoted to that intelligence. GPUs are "dumb" but their chip area is heavily devoted to absolutely enormous throughput on really big batches. Looking at AWS prices right now, I can pay ~4/hour for 50 4gHz CPU cores or 5,000 1.75 gHz GPU cores. That means the best case scenario for GPUs is ~2,000 times faster. In practice, you'll rarely achieve that, but GPUs can be tremendously faster for extremely parallelizable tasks (which graphics are.)

netflixandkill• 5 years ago

The geometry calculations across matrices for 3-dimensional euclidean objects are both relatively easy to do directly in hardware and repeated in only slightly varying ways for the various rendering stages.

Given the increasing sizes of framebuffers and textures it makes sense to localize that, although many CPUs now include a GPU equivalent on the same die and many of the same sort of instructions.

A lot of it comes down to how big and independent the working sets are as opposed to strict advantage in computation as well.

IvanAchlaqullah• 5 years ago

Reading comments in this thread, it feels like people still can't believe that in some case neural networks can be faster on CPU instead of GPU.

In fact, there is already real world use for neural networks that only optimized for CPU: an chess engine. More specifically, chess engine that use NNUE (Efficiently Updated Neural Networks) [1] like Stockfish 12 [2]. It run much faster while consuming less watts compared to GPU, can be run on average CPU, and managed to beat GPU based neural networks! [3]

This model (NNUE) already exist far earlier than the model discussed in this thread, yet there is almost no discussion about it on HN nor Reddit's r/MachineLearning

[1] https://www.chessprogramming.org/NNUE

[2] https://www.chessprogramming.org/Stockfish_NNUE

[3] https://www.chess.com/blog/the_real_greco/evolution-of-a-che...

ummonk4• 5 years ago

Was the NNUE trained on the CPU though? It’s intuitive that with the small size of a chessboard and the ability to use incremental update of evaluations, it will be faster to do neural network evaluation on CPU. Training, on the other hand, is typically highly parallelizable, so it would be a big development if we are able to do it faster on CPU.

nl5• 5 years ago

NNUE is weird. No one outside the chess/shogi community talks about it because they seem to have a very case of strong not-invented-here syndrome. It's hand-optimised CPU code that doesn't (yet) run on the GPU (which is why it is "faster").

To be fair, they do want to embed it into a consumer friendly application, and the integration for embedding TF or something that can run pytorch models on a GPU without python is non-trivial.

If someone ported it to a GPU, it almost certainly would be faster. See http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76986

There is a PyTorch port available, but no benchmarks unfortunatly. It does seem to be fairly widely used for training though, which is indicative of the speed gains available.

whimsicalism• 5 years ago

World of difference btwn "can be run" and "is trained on"

bob1029• 5 years ago

You will find most developers don't actually understand how fast a single x86 core can be when used appropriately. Most are too busy hiding from real hardware concerns to think about such things.

crazygringo• 5 years ago

almost all ventures into specialty hardware ended up with mainstream consumer CPUs catching up

All except for... 3D rendering, image compositing, video encoding, and video decoding?

I'm really struggling to see what you're trying to argue, because we do have multiple meaningful brute-force tasks that most computers today ship with dedicated silicon for.

Yes we can do more real-time audio processing I guess, but that's mainly because CPU speeds increased, not because "people learned how things really worked", or am I missing something?

aidenn0• 5 years ago

Don't forget encryption

rasz• 5 years ago

Multi-channel audio, and sound effects on CPUs

We only got here because Microsoft was afraid of one vendor, Creative, monopolizing positional audio market. This led to killing off audio hardware acceleration in Vista and leveling the playing field.

_0ffh• 5 years ago

Only it's not purely specialty hardware in this case. It's a massively parallel computation subsystem that happens to be also able to generate a video signal.

This really makes sense in a way. I once read in a paper that the best thing to have when faced with the task of performing an arbitrary calculation with maximal speed is 1) one core that's a fast as possible (for stuff that must be calculated sequentially) and 2) as many as possible slower cores (for stuff that can be calculated in parallel).

kolinko• 5 years ago

CPUs didn’t swallow GPUs though.

stjohnswarts3• 5 years ago

That's not how this works. Dedicated hardware for video games won't be on a general GPU anytimes soon. Neither will those problems that FPGA is solving when the matrices and properties of the calculation are massively parallel and static (or pseudo static require little change). Same with lidar processing or numerous other end uses. Dedicated hardware (in the form of FPGA's and ASICS) is everywhere.

guywhocodes5• 5 years ago

It's not the first similar claim, there was one that large CPU's can beat GPUs by creating a very sparse matrix using spatial hashing. Preserving the gradient but finding it with vastly less computation by simply not multiplying elements that don't have much or any impact.

EDIT: orginial work afaik https://arxiv.org/abs/1903.03129

AndrewKemendo• 5 years ago

AKA work smarter not harder

cat199• 5 years ago

This is literally the same research group, and the paper you link to is referenced in the article as being the predecessor 2020 publication to this announcement of a forthcoming 2021 paper expanding on the 2020 one.

So yes and no - if you are correct and that paper is indeed the first claim (not up on the literature), this one is also the same first claim, or at least, an extension to it

g42gregory• 5 years ago

Latest 2021 Code: https://github.com/RUSH-LAB/SLIDE

xaduha• 5 years ago

Reminds me of https://en.wikipedia.org/wiki/Cell_(microprocessor) and how that was considered a bad direction for game AI and general game depth, at least some were proclaiming it as such, that it would lead to prettier, but dumber games. Didn't make much sense to me at the time and in general I think number cruncher cores were a boon for AI.

monocasa2• 5 years ago

Game AI, being simple symbolic AI, has really branchy unpredictable instruction streams. The SPEs were really more optimized for GPU like tasks with clean loops where you can gang together a few SIMD lanes and process them all at the same time.

All of that being said, I've heard from a few PS3 devs that the simple PPEs didn't really do a great job with that kind of AI code either, due to their extremely simple core designs including their branch predictors. So it didn't end up being a concern and a lot of AI ended up running on the SPEs just because they could be laying around with nothing better to process.

At the end of the day, game AI wasn't really held back by a particular console's design.

baybal22• 5 years ago

I think there is a more direct comparison; Amdahl Trilogy https://en.m.wikipedia.org/wiki/Trilogy_Systems

A great case of half of the industry riding into a concrete wall following the trendsetter.

gumby3• 5 years ago

Here's the new paper referred to in the article: https://arxiv.org/abs/2103.10891

RcouF1uZ4gsC• 5 years ago

Study co-author Nicholas Meisburger, a Rice undergraduate, said "CPUs are still the most prevalent hardware in computing. The benefits of making them more appealing for AI workloads cannot be understated."

I don’t think this statement means what he thinks it means. I think “cannot be overstated” is what is desired.

jdeaton• 5 years ago

"The cost of training is the actual bottleneck in AI,"

This really isn't true in general. Maybe in select subfields of AI like language model pertaining this is true.

canjobear• 5 years ago

What other bottlenecks do you have in mind?

RhysU2• 5 years ago

Researcher time: Data munging, apples-to-apples model evaluation, identifying and fixing broken assumptions only detectable due to subtle model behavior.

trott2• 5 years ago

This is currently #1 on the front page, but I think HN is celebrating the defeat of GPUs prematurely. The architectures used in this paper have just one hidden activation layer. It remains to be seen whether the ideas in SLIDE are applicable to general deep learning tasks and architectures.

From the previous paper:

    We choose the standard fully connected neural
    network with one hidden layer of size 128.

ladberg• 5 years ago

Yep, I feel like anyone realizes when they first start playing around with NNs in a jupyter notebook that you get faster training on the CPU until you level up to bigger networks.

freeone30002• 5 years ago

That's normally due to the cost of copying the data to GPU space. Here, they've reformulated the training algorithm to be a less-computationally-complex problem. It's a huge difference.

high_byte• 5 years ago

so you think it was only faster because GPU init overhead?

0x008• 5 years ago

Wait... How can 1 Hidden layer be called DEEP neural network? I thought the whole "deep" thing was about having lots of hidden layers.

It's like saying we have a super fast fibonacci algorithm, but we can only show it for n=1.

trott2• 5 years ago

Wait... How can 1 Hidden layer be called DEEP neural network?

Yes. They describe a general algorithm, but show results for shallow nets. So it's unclear if this works well in general.

sumnuyungi3• 5 years ago

The title is a bit misleading as this algorithm is for feedforward networks and doesn't yet support convolutional layers or any of the SOTA techniques for image classification... which is why GPUs reign supreme for training deep neural nets.

waheoo• 5 years ago

doesn't yet

Does that mean it can / will?

sumnuyungi3• 5 years ago

The original paper includes convolutional layer support in their future work & next steps. But it's not a foregone conclusion that the same speedup will occur.

choppaface• 5 years ago

NeRF is a good example of a network that doesn't have convolutions yet requires a ton of iterations to train. This paper is particularly relevant to wide networks which are important because CPU memory is currently much cheaper than GPU memory (even for FANG researchers!).

sumnuyungi3• 5 years ago

Interesting, I didn't know that NeRF was simply a feedforward network.

I hope that this research group can make more headway into training on CPUs, but I also would like to (naively) see less hyperbolic titles. This paper is not just particularly relevant to wide networks - it's only relevant to wide networks.

ramoz• 5 years ago

True. I know it exists for inference though. Wondering where/when solutions like MKL might work for training.

shgidi• 5 years ago

Right, that's kinda nasty. Titles of papers refer deep learning, but I don't think fully connected networks might be considered a as deep learning.

sanxiyn• 5 years ago

What? No. Fully connected networks are deep learning, and actually the most important deep learning workload. See: Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective:

https://research.fb.com/publications/applied-machine-learnin...

Table 1 shows News Feed service uses fully connected networks model, and table 3 shows this workload dominates all other workloads.

Cybiote2• 5 years ago

Transformers, which are currently waging a successful campaign to conquer all Deep Learning, are largely stacked feed-forward networks, matrix multiplies and maps. Some ideas to make attention more scalable, such as LSH or large sparse attention matrices seem like they'd be well suited to this approach.

Their approach should also be readily adaptable to RNNs, including LTSMs.

Certainly worth investigating as an alternative for efficiently running and training giant networks on less expensive hardware.

stjohnswarts3• 5 years ago

This is why I come to HN, to find out why it doesn't work in the general case. I can always count on you guys to point out why something is an evolutionary change rather than revolutionary

dragontamer3• 5 years ago

Hmmm, it's a bit clickbait to make this a cpu vs gpu claim, unless there was a reason they believe that SLIDE wouldn't work on a GPU.

carlmr• 5 years ago

If I understand the paper correctly it's because GPUs are good at doing a lot of parallel matrix multiplications, at the cost of copying to and from GPU memory. The CPU has the advantage that it doesn't need to copy, and it's more flexible, but not as parallel, so if you use caching and limit the parallelism of the computations by exploiting sparsity the CPU wins as it does here.

So I don't see the clickbaitiness here, it's the central claim of the paper.

dragontamer3• 5 years ago

If you are training on the GPU, the bulk of the weight data never leaves the GPU in the first place.

It's a fundamentally different programming style, which makes comparisons extremely difficult.

Furthermore: you hide latency on the GPU by running more and more parallel instances. Yeah, there is a 5 to 10 microsecond delay in all CPU to GPU comms and back, but just double, triple, or 100x fold the parallelism and batch up more tasks (aka Gustafson's law).

Don't run one SLIDE. Run 100x of them in parallel to hide the latency. AMD Vega 64 has 16384 hardware threads (one instruction every 4 clock ticks across 4096 hardware SIMD cores) with up to 10 occupancy. That's 163840 max hardware SIMD threads of conceptual execution. (Occupancy is similar to hyperthreads/SMT).

In practice, you run out of VRAM before running out of threads typically. 1000x instances uses 1000x the RAM.

---------

Anyway, I've seen an entire generation of programmers try to make CPU only algorithms and only fail year after year. The cryptocoin community would generally prefer CPU mining rather than GPU or ASIC mining.

But time and time again, the algorithms adapt and someone is brilliant enough to make the new alleged CPU algorithm work really well on a GPU.

The_rationalist2• 5 years ago

Is it faster than deepspeed zero 3?

The_rationalist2• 5 years ago

if you aren't fixated on matrix multiplications does this means it is only faster for networks up to a size? If so what is the estimated threshold?

abakus• 5 years ago

I hope this or related technology won't be used for cryptomining.

rbanffy2• 5 years ago

If it does, we’ll have a lot of surplus GPU hardware available to run our number crunching.

Or room heating

nl5• 5 years ago

Note that this is an alogirthmic improvement, and the same algorithm wasn't implemented on the GPU version they are comparing it to.

I don't see any particular reason why we wouldn't see a similar speedup with the GPU version.

Random_browser• 5 years ago

Seems to work well for sparse data, I like these little clever tricks that yield big impacts.

peter_d_sherman• 5 years ago

"Deep neural networks (DNN) are a powerful form of artificial intelligence that can outperform humans at some tasks. DNN training is typically a series of matrix multiplication operations, an ideal workload for graphics processing units (GPUs), which cost about three times more than general purpose central processing units (CPUs).

"The whole industry is fixated on one kind of improvement—faster matrix multiplications," Shrivastava said. "Everyone is looking at specialized hardware and architectures to push matrix multiplication. People are now even talking about having specialized hardware-software stacks for specific kinds of deep learning. Instead of taking an expensive algorithm and throwing the whole world of system optimization at it, I'm saying, 'Let's revisit the algorithm.'"

Shrivastava's lab did that in 2019,

recasting DNN training as a search problem that could be solved with hash tables.

Their "sub-linear deep learning engine" (SLIDE) is specifically designed to run on commodity CPUs, and Shrivastava and collaborators from Intel showed it could outperform GPU-based training when they unveiled it at MLSys 2020."

PDS: Quote: "All programming is an exercise in caching." - Terje Mathisen

hebetude• 5 years ago

This post has Intel Inside.

satellite2• 5 years ago

It seems that Intel new CEO is bringing his connections with ycombinator troll farms.

nestorD• 5 years ago

One big caveat is that they use a parallel algorithm on a 44 cores machine to compete with a single GPU.

I can train a neural network on the GPU in my laptop but it has only 8 cores.

justAnIdea• 5 years ago

It all depends on economics, what you have now doesn't really matter a couple years down the line when the simple progress of time will show if this research has been used and built upon or not. 44+ cores, that's already available for workstations, and with the 4th generation of Threadripper (that's supposed to be shipping this year), that amount of cores is probably going to be even cheaper than it is now. On the other hand, GPUs are absurdly expensive right now as a result of multiple factors (people being home and gaming due to covid, crypto miners, lower supply of parts due to covid), but we'll see in a year.

Also, I'm pretty sure the goal of this research was not that you can train a DNN on your laptop, but that there is more democratization in the distribution of training power. Maybe some day a mid sized organization will be able to train a competitive model on a buch of off the shelf (or rented) servers...

unnouinceput• 5 years ago

If an algorithm runs faster on CPU than GPU, then it's only a matter of time before it gets optimized for GPU and at the end of the day will still run faster on GPU.

HelloNurse• 5 years ago

At the very least, generic types of parallel training are applicable to these models (e.g. an ensemble of models that see batches in a different order or are initialized differently).

villgax• 5 years ago

Dang, title is misleading. Should be shallow neural networks

polyrand• 5 years ago

This reminds me of this paper that also caught my attention. They used an array language to make convolutions run very fast on CPUs.

Array Languages Make Neural Networks Fast: https://arxiv.org/abs/1912.05234

zepmck• 5 years ago

I think this work is quite ridiculous. As far as I could see, they trained the net with a batch size equal to 1.

novaRom• 5 years ago

ELI5:

1) any input vector can be represented by a "similar" another vector, e.g. [.4, -.3, .1] => [+1, -1, 0]

INIT:

2) generate a set of random input vectors using only {+1, 0, -1} numbers

3) for every neuron: compute a set of activations for every random input vector

4) combine those sets producing a large hash table where keys are random vectors and values are indices of active neurons

FEED-FORWARD:

5) for every input vector, find its nearest neighbor, e.g.: [.4, -.3, .1] => [+1, -1, 0]

6) use this vector as hash key to get indices of neurons

7) compute activations for only these neurons using only their weights and the input vector

8) repeat for all layers

etc.

This way, you avoid lots of computations, but the key is to have a good hash function and a lot of memory.

29athrowaway• 5 years ago

Just a reminder that biological networks do not do backpropagation.

There's a lot yet to learn about neural networks and the state of the art is still biological evolution.

ilaksh• 5 years ago

I wonder if this is somehow similar to Reformers.

Roark66• 5 years ago

My understanding is that this works better because it is faster to hash their data (and compare with prehashed weights) rather than multiply the data and calculate activation functions.

I imagine this could work for more layers easily,and using other than fully connected layers could be done by partitioning data (for each set of inputs).

The next obvious question is, can their hashing function run even faster on a GPU?

They say: >In particular, SLIDE uses recently proposed DTWA hashing which works nicely on sparse data. If we represent the data vector as a list of indices and values. DTWA operation computes a random hash map of every non-zero index in the list. Then the map is partitioned into a fixed number of bins. Finally, the index with the maximum coordinate value in each bin is the hash value.

What is the "coordinate value"? And how does the above map to find instances where data and weights values would result in a value above the activation threshold?

They themselves mention it "works well on sparse data".

If they detect irrelevant data by simple value comparison(wrapped in that hashing function) couldn't GPU algorithm be sped up by skipping those multiplications too? How about instead of running matrix multiplication on entire data and all weights we "preprocess" data and weights finding specific indices where they are both above a certain value and multiply just those. It would be interesting to try that. How expensive would running such comparison, then data copy (or create a map of indices to process), then process only those chosen be in comparison with multiplying everything every time?

I'm guessing that it really depends on what percentage of data is actually relevant.

Another idea to make GPU AI faster is improving scheduling of those matrix multiplication. Multiplication by zero, another small number or a power of 2 should be able to execute faster than multiplication of high values (assuming it is implemented in hardware by addition and bit shifting if multiplying by powers of 2). I don't know if current GPU algorithms make any use of that, or if they simply divide the data by number of cores and run all chunks in parallel for however long the longest chunk takes to complete?

I think GPUs are pretty far from being dead in AI as a result of this.

Ambix• 5 years ago

Interesting. As I know, Stockfish [1] chess engine exploits NNUE nets optimized for CPU and regularly beats up Leela [2], which explots traditional GPU-based networks based on AlphaZero architecture.

[1] https://github.com/official-stockfish/Stockfish

[2] https://github.com/LeelaChessZero/lc0

Prof-Anshu• 5 years ago

https://intel-hpc-ai-pavilion.gallery.video/detail/videos/hp...

CPU-based algorithm trains deep neural nets up to 15 times faster than top GPU

Related Stories