Does my data fit in RAM?

by louwrentius • 6 years ago

216 162 6 years ago

ramraj075• 6 years ago

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

chmod7752• 6 years ago

Most DB engines will use what RAM is available and even if they don't, your OS's page cache will make sure stuff is fast anyways.

What tool should I use it to load this dataset on RAM and run these queries?

The question should really be: What tool should I use to make this fast?

Postgres can be pretty fast when used correctly and you can make your data fit.

falcolas• 6 years ago

Exactly this. DBs are really good at utilizing all the memory you give them. The query planners might give you some fits when they try and use disk tables for complicated joins, but you can work around them.

efficax• 6 years ago

Both mysql and pgsql bypass the page cache if they can and maintain their own page caches. You have to do this, otherwise you’re double caching! That is, you’d have your own page cache, which you need to manage calls to read() and to know when to flush pages, while the OS would also have the same pages in its own cache.

(mongodb I believe uses direct mmap access instead of a pagecache, and lmdb does this as well)

saber6• 6 years ago

I am not a big-data guy but wouldn't it be along the lines of A) get a big honking server B) fire up "X" SQL server C) Allocate 95-98% of the RAM to DB cache?

chmod7752• 6 years ago

A single terabyte is a few magnitudes from what you need big-data-anything for. You could probably work with that just fine on your average 64GB ram desktop with an SSD.

mitchty• 6 years ago

Option A is the best imo, I worked on many sql db's that the rule was to fit it into ram. Option c will bite you in the ass eventually. The kernel and your other processes need some space to malloc, and you dont want to page in/out.

Having 4/16TiB servers or "memory db servers" as I thought of them solved a lot of problems outright. Still need huge i/o but less of it depending on your workload.

nikhizzle• 6 years ago

You should use the cache or persist call on the spark data frame/dataset. Persist gives you more control.

ramraj075• 6 years ago

This is what most people in my org do, this is orders of magnitude slower than running queries on the same dataset on redshift with optimized presorting and distribution. Redshift doesn't scale for tens or hundreds of parallel users though, so looking for options

RookyNumbas• 6 years ago

There are a few steps to consider before you are loading data into ram.

Can you partition the data in any useful way? For example if queries use separate ranges of dates, then you can partition data so that queries only need to touch the relevant date range. Can you pre-process any computations? Sometimes tricky things done within the context of multiple joins can be done once and written to a table for later use. Can you materialize any views? Do you have the proper indexes set up for your joins and filters? Are you looking at execution plans for your queries? Sometimes small changes can speed up queries by many orders of magnitude.

Smart queries + properly structured data + a well tuned postgres DB is an incredibly powerful tool.

amznthrowaway5• 6 years ago

Can I set up efficient indexes on parquet data to use with Spark, or is it necessary to use a DB?

lsb• 6 years ago

You can get a box with 4TB of ram on EC2 for $4/hr spot, so copy your data into /dev/shm and go hog wild.

For lots of databases, most of their time is spent locking and copying data around, so depending on your workload you might getsignificant speedups in Pandas/Numpy if it's just you doing manipulations, and there are multicore just-in-time compilers for lots of Pandas/Numpy operations (like Numba/Dask/etc).

If you have lots of weird merging criteria and want the flexibility of SQL I'd say use a modern Postgresql with multicore selects on that 4TB box.

dijit2• 6 years ago

How long does it take to copy your data in? And what’s the bandwidth cost involved?

People like to talk about the elasticity or compute, but startup is not free (or even cheap in most cases).

ramraj075• 6 years ago

Currently I have it loaded on redshift with as much optimization as possible, and the queries are far more analytical than end-user like (often having to self join on the same dataset). This works okay, but doesn't scale with more than a handful users at a time. I'll probably run some tests with the postgres suggestion but curious if this is still a better alternative or not

typon• 6 years ago

in memory sqlite is pretty fast

ken• 6 years ago

Yes, but SQLite is probably the least expressive SQL dialect there is. If you're choosing SQL because of its expressiveness, you probably aren't thinking of a dialect with only 5 types (including NULL).

ramraj075• 6 years ago

I tried this once, creating an index on a 20 billion row table isn't fast :/

endorphone• 6 years ago

Every major database will load the hottest data into RAM, where the scope of "hottest" broadens to whatever amount will fit in RAM. A small percentage of them require you to confirm how much RAM it can use for this cache.

Putting the data on a ramdisk just becomes entirely redundant because it's still going to create a second memory cache that it uses.

Many operations do a local cache warming by running the common queries over the database before they it is brought online for processing. As a secondary note, people often under-estimate the size of their data because they don't account for all of the keys, indexes and relationships that also would be memory cached in an ideal situation.

threeseed5• 6 years ago

You can just write Spark SQL, set the executor memory to whatever the machine is and not worry about whether it's in RAM or not.

Spark will naturally use RAM first and then disk as needed.

juped• 6 years ago

Postgres

quersive• 6 years ago

[Disclaimer: I worked on BigQuery a couple lives ago]

I'd give Google BigQuery a shot. Should work fast [seconds] and scale seamlessly to [much] larger datasets and [many] more users. For a 1 TB dataset, I have a hard time imagining crafting a slow query. Maybe something outlandish like 1000[00?] joins. They also have an in-memory "BI Engine" offering, alas limited to 50GB max.

On premise, there is Tableau Data Engine. I don't think they offer a SQL interface, you have to buy into the entire ecosystem.

Long shot: I've been working on "most expressive query system over multiple tables" as an offshoot of some recent NLP work. Your use case piqued my interest. I'd love to help / understand it better. My contact is in my profile.

FpUser• 6 years ago

"My contact is in my profile"

This makes for a good laugh: Queries for the masses. Contact: f'info@${user}.com'

dana321• 6 years ago

If you run on MySQL/InnoDB you can set innodb_buffer_pool_size=1000Gb and it should cache your data after the first query.

dredmorbius3• 6 years ago

I'll be That Guy and ask: what are you doing with the data, and can you change your processing or analysis to reduce the amount of data you need to touch?

In my experience, it's nearly always the case that pulling in all data is not necessary, and that thinking through your goals, data, and processing can often reduce both the amount of data touched and the processing run on it massively. Look up the article on why GNU grep is so fast for a bunch of general tricks that can be employed, many of which may apply to data processing generally.

Otherwise:

1. Random sampling. The Law of Large Numbers applies and affords copious advantages. There are few problems a sample of 100 - 1,000 cannot offer immense insights on, and even if you need to rely on larger samples for more detailed results, these can guide further analysis at greatly reduced computational cost.

2. Stratified sampling. When you need to include exemplars of various groups, some not highly prevalent within the data.

3. Subset your data. Divide by regions, groups, accounts, corporate divisions, demographic classifications, time blocks (day, week, month, quarter, year, ...), etc. Process chunks at a time.

4. Precompute summary / period data. Computing max, min, mean, standard deviation, and a set of percentiles for data attributes (individuals, groups, age quintiles or deciles, geocoded regions, time series), and then operating on the summarised data, can be tremendously useful. Consider data as an RRD rather than a comprehensive set (may apply to time series or other entities).

Creating a set of temporary or analytic datasets / tables can be tremendously useful. As much fun as it is to write a single soup-to-nuts SQL query.

5. Linear scans typically beat random scans. If you can seek sequentially through data rather than mix-and-match, so much the better. With SSD this advantage falls markedly, but isn't completely erased. For fusion type drives (hybrid SSD/HDD) there can still be marked advantages.

6. Indexes and sorts. The rule of thumb I'd grown up with in OLAP was that indexes work when you're accessing up to 10% of a dataset, otherwise a sort might be preferred. Remember that sorts are exceedingly expensive.

If at all possible, subset or narrow (see below) data BEFORE sorting.

6. Hash lookups. If one table fits into RAM, then construct a hash table using that (all the better if your tools support this natively -- hand-rolling hashing algorithms is possible, but tedious), and use that to process larger table(s).

7. "Narrow" the data. Select only the fields you need. Most especially, write only the fields you need. In SQL this is as simple as a "SELECT <fieldlist> FROM <table>" rather than "SELECT * FROM <table>". There are times you can also reduce total data throughput by recoding long records (say, geocoded names, there are a few thousands of place names in the US, using Census TIGER data, vs. placenames which may run to 22 characters ("Truth or Consequences", in NM), or even longer for international placenames. You'll need a tool to remap those later. For statistical analysis, converting to analysis variables may be necessary regardless.

The number of times I've seen people dragging all fields through extensive data is ... many.

Some of this can be performed in SQL, some wants a more data-related language (SAS DATA Step and awk are both largely equivalent here).

Otherwise: understanding your platforms storage, memory, and virtual memory subsystems can be useful. Even as simple a practice as running "cat mydatafile > /dev/null" can often speed up subsequent processing.

manigandham2• 6 years ago

You don't need to store everything into RAM to get fast results. Data warehouse relational databases are designed exactly for this kind of fast SQL analysis over extremely large datasets. They use a variety of techniques like vectorized processing on compressed columnar storage to get you quick results.

Google's BigQuery, AWS Redshift, Snowflake (are all hosted), or MemSQL, Clickhouse (to run yourself). Other options include Greenplum, Vertica, Actian, YellowBrick, or even GPU-powered systems like MapD, Kinetica, and Sqream.

I recommend BigQuery for no-ops hosted version or MemSQL if you want a local install.

arnon• 6 years ago

Some of those aren't designed for data that doesn't fit in RAM. They even have it in their name (MemSQL for example).

bordercases• 6 years ago

Short answer: It fits in RAM if it's <= 12288 GB

slumdev• 6 years ago

The original Twitter thread is funny. This site doesn't add anything. It looks like SEO more than anything, and now that HN has linked to it, it's been successful.

samspenc• 6 years ago

Could you share the link to the original Twitter thread? I guess the link was changed on HN before some of us got here.

louwrentius2• 6 years ago

The original site yourdatafitsinram.com is now a get rich quick scheme site.

Also, the site that remained wasn't ever updated. So I took some time and updated some links.

I liked updating/creating this because I love the simplicity of the concept / thought behind it.

excalibur• 6 years ago

Short answer: It fits in RAM if it's <= 12288 GB

For a few given system configurations, and a given definition of "fits".

siffland• 6 years ago

Funny story, about 6 years ago we got a HP DL980 server with 1TB of memory to move from an Itanium HP-UX server. The test database was Oracle and about 600GB in size. We loaded the data and they had some query test they would run and the first time took about 45 minutes (which was several hours faster than the HPUX), They made changes and all the rest of the runs took about 5 minutes for their test to complete. Finally someone asked me and i manually dropped the buffers and cache and back to about 45 minutes.

Their changes did not do anything, everything was getting cached. It was cool, but one needs to know what is happening with their data. I am just glad they asked before going to management saying their tests only took 5 minutes.

Dylan168077• 6 years ago

What an unfortunate cache-filling algorithm, though. With eight drive slots, 40 minutes of IO is about 31 megabytes per disk per second.

rconti• 6 years ago

Those were some poorly built systems. I worked on probably 10 of them and they not-infrequently had .. major issues. HP's support model was to send a tech out with 2 sticks of RAM, and try them in different places to try to trace memory failures... across 4 (or 8?) cassettes, and 64 sticks of ram, and 20+ minute POST times.

We eventually had one server entirely replaced at HP's cost after yelling at them long enough, and that one never worked well enough to ever use in production, either. I'd say we had maybe a 70-80% success rate with those servers. They were beasts, though, with 4TB of RAM as I recall, and 288 cores.

panzagl• 6 years ago

I wonder how Abe Vigoda is doing...

ghc3• 6 years ago

Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!

lmilcin3• 6 years ago

You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course).

Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components that WOULD NOT BE NEEDED if you just decided to put your stuff in memory.

Frankly, the proportion of projects that really need to work on data that could not fit in memory of a single machine is very low. I work for one of the largest banks in the world processing most of its trades from all over the world and guess what, all of it fits in RAM.

threeseed5• 6 years ago

I really don't understand comments like this.

Yes your company's data may fit in RAM. But does every intermediate data set also fit in RAM ? Because I've also worked at a bank and we had thousands of complex ETLs often needing tens to hundreds of intermediate sets along the way. There is no AWS server that can keep all of that inflight at one time.

And what about your Data Analysts/Scientists. Can all of their random data sets reside in RAM on the same server too ?

ghc3• 6 years ago

There's a world outside of web apps and SV tech companies. There's a lot of big datasets out there, most of which never hit the cloud at all.

Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year.

I worked with another company that had 2000 oracle servers in some sort of franken-cluster config. Reports took 1 week to run and they had pricing data for their industry (they were a transaction middleman) for almost 40 years. I can't even guess the data size because nobody could figure it out.

This is not a FAANG problem. This is an everage SME to large enterprise problem. Yeah, startups don't have much data. Most companies out there aren't startups.

By the way, memory isn't the only solution. In the past 15 years, I've rarely worked on projects where everything was in memory. Disks work just fine with good database technology.

adrianN• 6 years ago

There are a number of cloud database solutions that are very easy to manage and not all that expensive. For example I work for Snowflake and our product doesn't need a small army of people to babysit it.

I mean, I also prefer doing things on a single machine, but if that machine gets expensive enough, or writing a program that can actually use all that power gets too difficult, why not switch to a cloud database?

Dylan168077• 6 years ago

This is about saving you money, so that's not the right fine print.

Look at it this way: This is for the person that's already going to get enough ram sticks to fit the entire data set or multiple of it, across many machines, and deal with the enormous overhead from doing queries across many machines. The revelation is that you can fit that much ram inside a single machine for a much cheaper and faster experience.

qaq• 6 years ago

it will cost you 100K CapEx and will save you many 100s of Ks in opEx and untold amount of money in development cost.

bcrosby95• 6 years ago

I would love to know how you came up with these numbers.

Aeolun3• 6 years ago

Meh, that’s like half the budget of the databases on our dev environment.

We have way too many fucking copies of our DB.

bkanber2• 6 years ago

I prefer using mocked data in dev. Smaller dataset, no possibility of PII leaks.

bluedino• 6 years ago

$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.

belltaco• 6 years ago

Finally a machine that can run Chrome.

dijit2• 6 years ago

Can still “only” have 6000 tabs.

(Someone did it recently with 1.5TiB of ram on a Mac Pro. Then Linus from Linus tech tips did it on Windows with 2TiB)

https://youtu.be/ldDTgp5irKA

wmf3• 6 years ago

If the result is faster than a $250K Hadoop cluster then you're still ahead.

trhway• 6 years ago

the way things today, the Hadoop cluster will be just a bit faster on that thing.

25 years ago i was suggesting clients to upgrade from 4MB to 6-8MB as it was improving their experience with our business software, these days i've already suggested a couple of customers to upgrade from 6TB and 8TB respectively ... as it would improve their experience with our business software. What's funny is that customer experience with business software back then was better than today.

malux854• 6 years ago

Yeah, except Hadoop provides redundancy, easier backups, turnkey solutions for governance and compliance and easier scaling

I’m no fan of distributed systems that sit idle at 2% utilisation when a single node would do : BUT, reducing it down to “cost” and “does it fit in RAM” is way too reductive

threeseed5• 6 years ago

But that Hadoop cluster will also work just fine if the data set does not fit in RAM. And I've never met a data set that didn't expand over time. And with Spark you get a robust, scalable and industry standard way of distributing work amongst the nodes.

Also did you know that Hadoop is open source. So that $250K is purely for hardware.

kube-system• 6 years ago

if you can get by with 4TB, it's less than $30/hr on AWS

objektif• 6 years ago

What is? Anything I touch on AWS is 5k a month. /s

PeterisP• 6 years ago

If that allows you to simplify your architecture so that stuff just runs on a single machine instead of needing to develop, debug and maintain a distributed solution, then you save much more money in engineer salaries than this.

justizin• 6 years ago

ah and it is so fucking lovely to be hired by a company which has been running everything on a single machine for 2 years.

lunixbochs• 6 years ago

Looks like I'm all set with my -12PB dataset fitting in ram.

jagged-chisel• 6 years ago

Is that “negative twelve” or did you mean ~?

jkaptur• 6 years ago

12TiB, not PiB!

Hope you read this before you spent $50,000,000 dollars/month (although you might be able to negotiate a discount).

randyzwitch3• 6 years ago

While pithy, the implication that you are going to process 12 TB of data in RAM using mostly single-threaded tools doesn't reflect reality.

lotyrin2• 6 years ago

Where exactly does that implication come from? Are you from a world where you need a map reduce framework + cluster to have parallelism of any kind?

randyzwitch3• 6 years ago

No, but I frequently see people implying that you can do your data science in Python and R as long as you can fit the data in RAM. As you mention, it's not RAM that's the limiting factor for larger data volumes, it's finding tools that exploit parallelism.

ksec2• 6 years ago

The problem is DRAM price hasn't drop one bit.

The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period.

And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.

gameswithgo• 6 years ago

buying the ram probably cheaper than buying database liceneses and/or the machines to run them and/or the time you spend making things fast enough when using them.

markus_zhang• 6 years ago

just wait for China to enter the mkt in thia decade...

yorwba• 6 years ago

You mean 2 years ago? https://www.anandtech.com/show/12455/chinese-xian-uniic-semi...

It doesn't seem to have brought prices down much.

keanzu3• 6 years ago

Everyone knows that DRAM prices have been in a collapse since early this year, but last week DRAM prices hit a historic low point on the spot market. Based on data the Memory Guy collected from spot-price source InSpectrum, the lowest spot price per gigabyte for branded DRAM reached $2.59 last week.

https://thememoryguy.com/dram-prices-hit-historic-low/

You've selected out the low points on the graph: 2012, 2016, and 2019, most of the time DRAM has not been available at these prices. Now is definitely the time to load up on RAM.

jimnotgym• 6 years ago

Everyone knows that DRAM prices have been in a collapse

I'm afraid I had no idea that had happened at all.

since early this year,

Since..when?..now? Feb 12th is pretty early in the year AFAIAC

MR4D• 6 years ago

7.5 percent drop since 2012 (using your numbers plus the OP’s number).

That’s not what we in computing refer to as a “collapse”.

paulmd• 6 years ago

Now is definitely the time to load up on RAM

And it is predicted to climb back up this year, due to manufacturers dropping wafer starts at a point in time when a large launch of next-gen consoles is drastically increasing consumption.

Nope, definitely no collusion there. /s

ksec2• 6 years ago

the lowest spot price per gigabyte for branded DRAM reached $2.59 last week.

It would be better to reference this as quoted from the article which was written in November 2019. So not really last week

most of the time DRAM has not been available at these prices.

I did said price floor.

mr_toad• 6 years ago

If it was just a matter of adding more memory people would. A few thousand, or tens of thousands of dollars aren’t much to organisations that have that much data.

The trouble is that there are limits to how much memory you can fit on a motherboard.

bscphil• 6 years ago

And how much you can afford. My partner's scientific research group recently upgraded their shared server used for analysis ... to one with 32 GB of RAM and 8 TB of storage. I think they could have done better, personally, but it's telling how thin budgets are stretched in a lot of real world cases.

toast0• 6 years ago

Something like this [1] gets you to 4TB of ram. Dual socket Zen2 Epyc gets you 8 TB ram. A lot of stuff fits in 8 TB of ram.

[1] https://www.supermicro.com/en/Aplus/system/1U/1113/AS-1113S-...

iaabtpbtpnn• 6 years ago

Sure, my data fits in RAM if you take one of those example machines, which we have, and which are currently hosting dozens of production VMs, and instead dedicate the whole thing to my database. I'd love that, but it's never going to happen.

takeda2• 6 years ago

It's still way more efficient to use it for that database, than running a dozen of hadoop nodes as VMs.

sdenton4• 6 years ago

We do what we must because we can.

zaltekk• 6 years ago

The Amazon listing has an Azure instance type and links to Azure docs.

A quick search shows that you can get at least 24TiB from AWS: https://aws.amazon.com/ec2/instance-types/high-memory

louwrentius2• 6 years ago

Thanks for pointing out my sloppy mistake.

I have chosen for the cloud options to only select virtual instances that can be spun up on demand. The high-memory instances you link to are purpose-build.

On the other hand, it is true, they do exist.

jszymborski• 6 years ago

"Can I afford to fit my data in RAM?" is a whole other site, I presume...

vb6sp6• 6 years ago

"Can you afford to not put your data in RAM" is another one.

birdyrooster2• 6 years ago

"Did you even need all of this data?" is the one after that one

takeda2• 6 years ago

The point is that the alternative is far more expensive (taking from the tweet linked on the page that inspired it)

skwb• 6 years ago

I remember back in the early 2010's that a large selling point of SAS (besides the ridiculous point that R/python were freeware and therefore cannot be trusted on important projects ) was that it can chew through large data sets that perhaps couldn't be moved into RAM (but maybe it takes a week or whatever....).

This was a fairly salient point, and remember circa 2012/2013 struggling to fit large bioinfomatics data into an older iMac with base R.

vasili111• 6 years ago

I have not tested it myself but now there is a disk.frame ( https://github.com/xiaodaigh/disk.frame ). As far as I know, other options exist too.

dredmorbius3• 6 years ago

SAS Institute have long claimed this. It's been provably bullshit for decades.

In practice, an awk script frequently ran circles around processing. On a direct basis, awk corresponds quite closely to the SAS DATA Step (and was intended to be paired with tools such as S, the precursor to R, for similar types of processing).

The fact that awk had associative arrays (which SAS long lacked, it's since ... come up with something along those lines) and could perform extremely rapid sort-merge or pattern matches (equivalent to SAS data formats, which internally utilise a b-tree structure) helped.

With awk, sort, unique, and a few hand-rolled statistics awk libraries / scripts, you can replace much of the functionality of SAS. And that's without even touching R or gnuplot, each of which offer further vast capabilities.

And at an aggreable annual license fee.

staticassertion• 6 years ago

Pretty cool that single boxes have > 10 terabytes of RAM.

I would definitely imagine that most workloads rarely need more than a few to a few hundred TB in memory, since you may have petabytes of data but you probably touch very little of it.

H8crilA• 6 years ago

One of the best things you can often do for latency/throughput improvement is to just mlock() the data. Yes, it fits. Get a more expensive machine and 5x the throughput in a single day with a config change.

dmos622• 6 years ago

I just upgraded my laptop to 3 GB. Feeling a bit behind on the times.

scarejunba• 6 years ago

You can fit 48 TB on a HPE MC990 X though I'm pretty sure that's got one of those NUMA architectures that SGI had with the UV 3000 or whatever.

I remember jokingly telling my team to spend the millions of dollars we did expanding our clusters with one of these and just processing in RAM. I honestly don't think I did the analysis to make sure it would be actually better.

It was 'jokingly' because we couldn't afford three of these machines anyway. The clusters had the property that we could lose some large fraction of nodes and still operate, we could expand slightly sub-linearly, etc. which are all lovely properties.

It would have been neat, though. Ahhhh imagine the luxury of just loading things into memory and crunching the whole thing in minutes instead of hours. Gives me shivers.

andyjpb3• 6 years ago

This is why IBM have historically done so well.

You didn't have to have all the capital. They'd rent you the machine on a long lease and you could say you'd bought a million dollar computer.

Sun tried to get into similar markets but it was always a tougher deal on minis and micros.

rbanffy2• 6 years ago

IIRC, the largest IBM z15 can have up to 40 TiB of RAM.

The website needs updating.

timbit42• 6 years ago

How much is that in TB?

wmf3• 6 years ago

For RAM TB = TiB.

wmf3• 6 years ago

And Superdome Flex supports 48 TB.

rbanffy2• 6 years ago

These seem to be the least boring x86 machines. At 16 sockets the NUMA topology must be interesting.

einpoklum• 6 years ago

Does my data fit in RAM?

1. If you have to ask, then either it doesn't now, or it doesn't sometimes. So assume it doesn't.

2. If you can use a cluster, then maybe.

3. In some senses, it doesn't matter. How so? Reading and writing from RAM is very slow, latency-wise, for today's processors. If I can bend the truth a little, it's a bit like a fast SSD. So, if you can up the bandwidth to disk enough, it becomes kind of comparable. Well, if you can use 16 PCIe 4.0 lanes, it's roughly 24 GB/sec effective bandwidth, which is roughly half of your memory bandwidth. Now it's true that in real-life systems it's usually just 4 lanes, but it's very doable to change that with a nice card.

4. DIMM-form-factor non-volatile memory may increase memory sizes much more.

yongjik2• 6 years ago

Even more importantly, does your data have to fit in RAM?

There are tons of problems that need to process large data, but touch each item just once (or a few times). You can go a really long way by storing them in disk (or some cloud storage like S3) and writing a script to scan through them.

I know, pretty obvious, but somehow escapes many devs.

sevensor• 6 years ago

There's also the "not all memory is RAM" trick: plan ahead with enough swap to fit all the data you intend to process, and just pretend that you have enough RAM. Let the virtual memory subsystem worry about whether or not it fits in RAM. Whether this works well or horribly depends on your data layout and access patterns.

mrmrcoleman• 6 years ago

Interesting. Can you provide some examples of where this is the correct approach?

beached_whale• 6 years ago

Don't even need to do that. Just mmap it and the virtual memory system will handle it.

EdwardDiego2• 6 years ago

I've spent a lot of time writing Spark code, and its ability to store data in a column oriented format in RAM is the only reason why - disk is goddamned slow.

As soon as you're touching it more than once, sticking it in RAM upon reading makes everything much faster.

globular-toast• 6 years ago

Escapes many devs? Really? I used to work with biologists who thought they needed to run their scripts on a supercomputer because the first line read their entire file into an array. But if I saw someone who calls themselves a "dev" doing this I'd consider them incompetent.

ludamad• 6 years ago

Plenty of devs that don't do any sort of file streaming, say those who started with Game Maker or another specialized domain

saint_abroad• 6 years ago

I once got into an argument with a senior technical interviewer because he wanted a quick solution of an in-memory sort of an unbounded set of log files.

Needless to say I wasn't recommended for the job, and it taught me a valuable lesson: if you don't first give them what they want, you can't give them what they actually need.

Jupe• 6 years ago

Some pedantry...

Raw RAM space is not the issue... it's indexing and structure of the data that makes it process-able.

If you just need to spin through the data once, there's no need to even put all of it in RAM - just stream it off disk and process it sequentially.

If you need to join the data, filter, index, query it, you'll need a lot more RAM than your actual data. Database engines have their own overhead (system tables, query processors, query parameter caches, etc.)

And, this all assumes read-only. If you want to update that data, you'll need even more for extents, temp tables, index update operations, etc.

LeoPanthera• 6 years ago

I often use RAM disks for production services. I'm sure that I shouldn't. It feels very lazy, and I know that I'm probably missing out on the various "right" ways to do things.

But it works so well and it's so easy. It's a really difficult habit to kick.

Obligatory joke: "Oh boy, virtual memory! Now I can have a really big RAM disk!"

deepsun• 6 years ago

There is AWS and Azure, but no GCP. Why?

EdwardDiego2• 6 years ago

Telling me that my (example) 24TB data set fits into RAM, because it fits into 24TB on an AWS instance designed for SAP that's so expensive that it's price on inquiry, isn't overly helpful.

May I suggest that a) they consider applications need RAM too and b) that if the price is POI, it's probably better to mention "But you know, 188 m5.8xlarges might be cheaper"

sriku• 6 years ago

The question of whether my storage mechanism fits budget is more relevant, no? It costs less than 4cents per hour to store 1TB on S3 but an X1 with 1TB of RAM costs $13 or so per hour on Aws. So the issue is whether what you pay for is worth the result you're computing.

known• 6 years ago

Reminds me of https://en.wikipedia.org/wiki/List_of_Ultrabook_models

hkt• 6 years ago

Honestly thought this would be a page that says "yes"

1MachineElf• 6 years ago

I've no idea if the intended audience here would ever run their workloads on Solaris, but like the IBM POWER systems, Oracle & Fujitsu SPARC servers also max out at 64TB of RAM. I didn't see those included here.

Does my data fit in RAM?

Related Stories