Upmem Puts CPUs Inside Memory to Allow Applications to Run 20 Times Faster

https://www.hpcwire.com/off-the-wire/upmem-puts-cpus-inside-memory-to-allow-applications-to-run-20-times-faster/

by rbanffy • 7 years ago

170 78 7 years ago

hpcwire.com

oysterfish• 7 years ago

How can i get my hands on this?

seren• 7 years ago

There are some information on the SDK here :

https://www.upmem.com/developer/ or more directly https://sdk.upmem.com/2019.2.0/

It seems you have to compile an executable for the in memory processor, and they have some sort of daemon/infrastructure to communicate from the main cpu to the PIM.

jacknews• 7 years ago

The concept of "smart memory" is good (though not new), but I think the devil could be in the process-technology details; DRAM and logic are not that compatible.

But let's hope they've figured it out.

Symmetry2• 7 years ago

You do have eDRAM where big blocks of DRAM are put on board a CPU as a last level cache, like in the POWER7. It does special processes features.

monocasa• 7 years ago

Eh, we've shipped that a lot before. The eDRAM chip on the xbox 360's GPU had a decent amount of logic as well.

nickpsecurity• 7 years ago

Venray tried this:

https://www.wired.com/2012/01/russell-fish-venray/

They didn't get huge that I can tell. Might be a tough market.

petra3• 7 years ago

DRAM is a commodity business. It works in huge batches.

Convincing DRAM manufacturers to change things for something at that small scale is a huge challenge.

I don't think venray managed to do that.

But OPMEM probably did.

rasz2• 7 years ago

Micron dabbled in this. 2003 project Yukon (shutdown in 2004) and again 10 years later with 2013 “The Automata Processor” (erased from their website).

petra3• 7 years ago

More details: https://www.anandtech.com/show/14750/hot-chips-31-analysis-i...

bem94• 7 years ago

This is really interesting, but note the power increase seen inside the DRAM modules.

Anandtech have an excellent analysis: https://www.anandtech.com/show/14750/hot-chips-31-analysis-i...

The operation itself [inside the dram module] actually requires 2x the power (20 pJ compared to 10 pJ [inside the main CPU]), but the overall gain in power efficiency is 170 pJ vs 3010 pJ, or just under 20x
One thing that this slide states that might be confusing is the server power consumption – the regular server is listed as only 300W, but the PIM solution is up to 700W. This is because the power-per-DRAM module would increase under UPMEM’s solution.

I'm assuming that those numbers represent "peak" power, and that when idle the compute parts of the DRAM can be power/clock gated. The implication being that if you are doing lots of in-memory analytics, you'll get a power saving, but if you switch to "something else" and don't get good utilisation of the in-memory compute, you'll probably ruin your power/energy efficiency.

I guess that means the future improvements will involve bringing the power consumption of the modified DRAM modules back in line with their "normal" cousins.

baybal23• 7 years ago

While it is very true that the memory is much bigger bottleneck than the CPU in computing today, it is a quite a bit disingenuous for the "in-memory" computing crowd to keep pointing to dramatic difference in energy needed for a memory transaction in "their vs ours" solution.

Yes, onchip SRAM can take single picojoules vs hundred picojules per bit on external DRAM, but using external DRAM is a perfectly valid design decision that does not dramatically increase the power consumption because CPUs have large caches and very smart prefetch logic. Some algorithms may seem to demand tens of gigabits of memory bandwidth on paper, but in reality smart prefetchers can reduce that to hundreds megabits.

Where things like this matters is scalability. Parallel computing performance scales not much with memory bandwidths per CPU, but with the number of individual CPUs, threads and their prefetch units. Here, the decision to have hundreds of wimpy CPUs, but all with individual memory access will be valid.

Edit: typo fix

tl• 7 years ago

Some algorithms may seem to demand tenths of gigabits of memory bandwidth on paper, but in reality smart prefetchers can reduce that to hundred megabits.

Are tenths of a gigabit and hundreds of megabits not the same thing?

Symmetry2• 7 years ago

Prefetchers don't do anything do reduce the power consumption involved in streaming data from memory, they only reduce the latency seen by the CPU. In fact, since the prefetcher sometimes makes mispredictions, they tend to increase power.

sp3323• 7 years ago

Watts are Joules / second. If you're doing operations that take 1/20 as much power each but you're doing them 20x faster, it takes the same amount of power in Watts. For a better comparison you'd need to measure Watts * seconds for a given task.

ljcn• 7 years ago

Indeed, like how the best CPU reviews don't just report the power consumption but also the energy consumed to do a given benchmark (power integrated in time for a fixed amount of work).

skavi• 7 years ago

*1/20 as much energy

mpweiher• 7 years ago

First incarnations of this I heard of were SLAM, Scan Line Access Memory [1]

"A SLAM consists of a conventional dense semiconductor dynamic memory augmented with highly parallel, but simple, on-chip processors designed specifically for fast computer graphics rasterization. "

and PixelPlanes[2].

[1] https://dl.acm.org/citation.cfm?id=13358

[2] https://pdfs.semanticscholar.org/0c1b/8ffdf7074a73883139da41...

basicplus2• 7 years ago

So what are the security implications?

One would really need to trust the manufacturer.. but how?

rbanffy5• 7 years ago

It can actually be useful for security - you can tell the DPUs to erase memory the moment the OS frees it, reducing the risk of data being read or decrypted. That's probably a very trivial use of the tech, but it's a low hanging fruit.

cestith• 7 years ago

I suppose one could put more of an OS's work with memory into this level, too. Zero it when it's freed, but also keep the free block list updated at this level. Maybe use these processors for aggregating available contiguous blocks when necessary, too. Maybe use these processors to update the virtual to physical mappings and keep free space optimized. Then, when the application actually needs to do something close to RAM it could be given access.

cellular• 7 years ago

This is a great idea! I'm surprised we haven't thought of this before. Add a simple ALU in there with some request to perform some NN calculations like a co processor!

sitkack• 7 years ago

The concept is quite old.

https://en.wikipedia.org/wiki/Berkeley_IRAM_project

http://iram.cs.berkeley.edu/

justapassenger• 7 years ago

Integration and disaggregation of various parts of computer architecture is a constant cycle that depends on what’s the current bottleneck. For decades computers are connection of compute, memory, storage and network, and you can trade them based on what’s needed.

If transferring data is bottleneck, then companies start to integrate. But that has downside of not being able to grow resources at different speed, so once transferring data is no longer a bottleneck, everyone runs to disaggregate. And then it repeats.

A lot of current ideas were actually used in the past, but cycle phased them out, and they make a comeback right now.

Zenst• 7 years ago

Interesting that they went with a 32bit ISA, given the growth and demand for ML/AI, which seems to lean upon lower precision (8bit), then I wonder if they could of gone for that instead of something more a jack of all trades design.

Which makes me wonder, when will somebody do an ISA dedicated for ML/AI and will we ever see old 8bit CPU's reborn as on memory CPU's.

But we have seen this approach for processing upon memory before and whilst that has not tractioned into a product, this might. Though with Chip design going towards chiplets, ram can and may well become another chiplet in the evolution of that design process.

radarsat1• 7 years ago

In what domains do people do 8 bit machine learning? Most example code I've seen e.g. for TensorFlow uses float32 matrices.

jdietrich• 7 years ago

It's quite common to train at float32 but quantize down to int8 for inference. The loss of accuracy is negligible, but you make huge performance gains.

https://medium.com/apache-mxnet/model-quantization-for-produ...

tachyonbeam• 7 years ago

The thing is, you still need 32-bit or 64-bit values for pointers, loop counters, etc. You're also going to want your CPU to interact and convert between richer data types. What you probably want is to have a 64-bit CPU and control unit, with some SIMD ALUs that operate on multiple 8-bit values at once.

Sidenote: I think that ML people are more interested in 16-bit floats than 8-bit integer values. Though none of that is mainstream yet, most ML code still uses 32-bit floating point numbers at the moment.

rbanffy5• 7 years ago

Since the processing elements seem to be unable to reach memory outside their own 64 MB block, 32 bit integers seem sufficient for pointers. Also, it has a 64-bit wide access to its memory block, but can't map it - all transfers are via DMA.

It must feel like a Cell processor with thousands of SPUs to play with.

imtringued• 7 years ago

Venrays did this in 2012 with their TOMI Borealis PIM but it didn't catch on.

https://hothardware.com/news/cpu-startup-combines-cpudramand...

wmeddy• 7 years ago

I'm glad to see that you mentioned this. I had the pleasure of working with Russell Fish on a different project, and I immediately thought of TOMI when looking at the parent article.

rbanffy5• 7 years ago

I wonder what having these in-memory processing units can do with cryptocurrencies. Serial processing performance is rather low, but there are thousands of DPUs in a memory DIMM.

crb0022• 7 years ago

You have to obey speed of light latency, and that means putting compute in RAM right next to the data. Key is to have a reusable fabric so all the TSMC customers can pop their own chips in RAM, especially ASICS and micro-FPGAs.

crb0022• 7 years ago

Here is a link to the assembler instruction set for the PIMs. https://sdk.upmem.com/2019.2.0/200_AsmSyntax.html#assembly-i...

tempsolution• 7 years ago

This sounds sketchy. Either the rest of the industry is incredibly incompetent or they are deliberately omitting a huge drawback of this solution. And I don't see lack of expandability as a drawback, just buy a bit more RAM to start with.

Then there are NVIDIA GPUs and Intel Phi (i.e. Knightslanding)... So they are trying to sell us that neither AMD, NVIDIA or Intel have though of this "ingenious" decision to increase the efficiency obtained by co-locating CPU and memory? The fact that they deliberately avoid addressing this elephant in the room makes me super skeptical.

sp3323• 7 years ago

It requires significant changes to the compiler, so this RAM would not make any difference to existing binaries. This is a really big change in the way computers are structured, so it's no wonder it hasn't taken off before. And this isn't the first attempt; look up computational RAM, IRAM, RADram, smart DRAM, processor-in-memory.

bigtrakzapzap• 7 years ago

No pun intended, I remember IRAM. Then there was Tilera and Tabula (MPPA/MIMD). https://en.wikipedia.org/wiki/Massively_parallel_processor_a....

LoSboccacc• 7 years ago

the drawback is the same as with all the others NUMA solution, the whole thing (mmu, os, code, controllers etc) needs to be aware of where the data comes and go because moving something from one bank to another stalls the compute pipeline for a lot, which makes compilers and thread scheduling and pagination etc a complex affair.

dooglius• 7 years ago

So is any kind of disruption impossible then, if everything has already been thought of by other big players? The "big drawback" could be that it takes a big investment to try this, and it's a big risk as to whether that investment pays off.

ChainOfFools• 7 years ago

Not an expert here but the last time I got curious about the state of processor-in-memory my understanding was that the infeasibility of PIM is in large part due to mutually exclusive optimization paths in the fab processes used for CPUs verses for DRAM

Diederich• 7 years ago

This somewhat reminds me of the Blitter chip, which first came to my attention in the Amiga 1000:

https://en.wikipedia.org/wiki/Blitter

"A blitter is a circuit, sometimes as a coprocessor or a logic block on a microprocessor, dedicated to the rapid movement and modification of data within a computer's memory."

tempguy99994• 7 years ago

IIRC the blitter was just a normal coprocessor. I'm pretty certain it wasn't embedded in any way in the memory.

tejtm• 7 years ago

As I recall there was "chip" ram and "fast" ram. So maybe not embedded for all definitions but certainly processing chips with privileged access to a subset of memory

abstract7• 7 years ago

How does shared memory between processes on different chips work with something like UPMEM?

cmrdporcupine• 7 years ago

Seems to me that a full CPU is overkill for this; a hybrid approach where you just have some arithmetic on vectors, with no program counter or branching really, could be effective without adding a bunch of complexity.

Also seems like concurrency and data consistency issues could arise pretty drastically when each memory unit is potentially making changes independent of the processor...

And what about CPU cache coherency?

flaviu1• 7 years ago

It uses a custom CPU and ISA that's specialized for the application: https://www.anandtech.com/show/14750/hot-chips-31-analysis-i...

Internally the DPU uses an optimized 32-bit ISA with triadic instructions, with non-destructive operand compute. As mentioned, the optimized ISA contains a range of typical instructions that can easily be farmed out to in-memory compute, such as SHIFT+ADD/SHIFT+SUB, basic logic (NAND, NOR, ORN, ANDN, NXOR), shift and rotate instructions

Re. CPU cache coherency, they have a software library that automagically hides that: https://images.anandtech.com/doci/14750/HC31.UPMEM.FabriceDe...

Not exactly sure how it works, I'm not super familiar with the internals of CPUs.

xiphias2• 7 years ago

Where can I buy put options to this company?

20x speedup compared to a CPU looks worse than just using a stock GPU, and we haven't talked about the AI inference chips that are optimized for NN inferencing.

petra3• 7 years ago

GPU's have limited amount of memory.

proverbialbunny• 7 years ago

But they can prefetch data. This makes writing code for them a bit more complex, but outside of that the ram limitations are mostly moot.

This fails when data needs to be read and written randomly. However, even if the entire dataset was on vram random reads and writes would still be slow, because of the way a stream processor works. All reads and writes need to be laid out in memory in an order for a gpu to do it's best work.

teilo• 7 years ago

1980s: RAM in the CPU to speed up the CPU. 2020s: CPU in the RAM to speed up the RAM.

manav• 7 years ago

I’m not much of a hardware person but I was wondering why not do the opposite? Pair a massive amount of SRAM/DRAM with a normal cpu as l2+ cache. Or is that what they are essentially doing?

vvanders• 7 years ago

SRAM and DRAM are two totally different things.

An L2 cache is just a huge chunk of SRAM, they take more die space and use a lot more power at the benefit of running with lower latency.

m0zg• 7 years ago

Micron has been pursuing something similar for a while. The "CPUs" that you can put on the same die, however, are very limited in what they can do, owing to the specific lithography used on memory chips, and the general lack of die space. They also don't get uniform access to a huge memory range that you'd expect from a "real" CPU, and require you to partition work to fit within the constraints of the memory access pattern they can, in fact, support. The instruction set is very limited, floating point can only be emulated (i.e. slow AF, not that you actually need it for neural networks most of the time). The upside is the unlimited memory bandwidth and very low pJ/byte, with a few catches.

Don't know if this is similar, but if it is, it's going to be a hard sell, especially in the era when 90% of programmers can't even understand what I wrote above.

fnord77• 7 years ago

Sorry - I don't know much about chip design: Will this increase the heat produced by DIMMs and will they require active cooling?

Does increased heat decrease the life of RAM?

karmakaze• 7 years ago

Seems like MemPU would be a more fitting name/ordering of letters.

Pywarrior• 7 years ago

computronium

Upmem Puts CPUs Inside Memory to Allow Applications to Run 20 Times Faster

Related Stories