Myths about /dev/urandom

http://www.2uo.de/myths-about-urandom/

by petrosagg • 11 years ago

186 106 11 years ago

aidenn02• 11 years ago

Is there a patch to make /dev/random behave like /dev/urandom? I've had at least some server software stall blocking on /dev/random, and I already build my own kernels, so incorporating such a patch is easy.

throwaway20482• 11 years ago

you could probly edit udev rules to make random be the same device major/minor as urandom

iso-8859-1• 11 years ago

maybe it is doable with overlayfs?

mappu• 11 years ago

  rm /dev/random && mknod -m 444 /dev/random c 1 9

throwaway20482• 11 years ago

this wont survive reboot, you need udev rules.

viraptor5• 11 years ago

Or run the app in its own namespace with custom /dev.

robmccoll2• 11 years ago

mount --bind /dev/urandom /dev/random

DiabloD32• 11 years ago

That won't stick across reboots, and at that point, I think you can just rm /dev/random; ln -s /dev/urandom /dev/random.

mikekchar• 11 years ago

Although in most cases it doesn't matter, I wouldn't do this. Remember that the PRNG will only output unpredictable values as long as the initial seed was unpredictable (and unknown). As stated in the article, there is actually one time where you care if you have enough entropy -- the first time you grab a random number. After that the PRNG is seeded with an unknown value and you are fine (assuming you waited until you had enough entropy). Before that, you aren't.

BSD's implementation seems sensible -- block the first time and then don't block again. Or potentially block if you haven't reseeded in X amount of time (X being configurable), to defeat the situation where someone figures out your seed. Although, I have a tough time understanding how someone will figure out your seed without completely compromising your system anyway... but systems are supposed to be robust against attacks that one can't immediately imagine ;-)

aidenn02• 11 years ago

Is there a linux kernel patch to use BSD's implementation then?

nshepperd2• 11 years ago

I use a solution based on udev rules, here: https://superuser.com/questions/309840/how-can-i-point-dev-r...

matheweis• 11 years ago

An easy hack is to use rngd with /dev/urandom as an "entropy source" (risky if you are going to use /dev/random for anything that REQUIRES true entropy, but otherwise should be safe as the article suggests).

po• 11 years ago

Why hasn't someone qualified re-written that unix man page by now? I've been reading cryptographers trying to explain all of this for a while now. I feel like that would help put a lot of this to rest and save everyone time.

AndyKelley2• 11 years ago

It would have saved me a lot of time, that's for sure. And here I thought I was being responsible and informed by reading the man page.

cbsmith9• 11 years ago

Honestly, anyone who cares & can understand will look at the implementation of the two and grok it. Those who don't will screw up the rest of it anyway, and that will create more than enough trouble even with some helpful guidance. Best to have them fail in pretty obvious ways.

AndyKelley2• 11 years ago

Thanks for this. Time to go fix my code[1].

[1] https://github.com/andrewrk/genesis/blob/0d545d692110d33068d...

masklinn2• 11 years ago

Since you're only targeting Linux, if you are fine with requiring recent kernels (3.17), you should probably use getrandom(2): http://man7.org/linux/man-pages/man2/getrandom.2.html

theophrastus• 11 years ago

Interesting and edifying, thank you. I wonder how the recent 4.2 kernel release affects this: Linux 4.2 Released Improving Cryptography Options[1]

[1] http://www.linuxplanet.com/news/linux-4.2-released-improving...

snorrah• 11 years ago

Very interesting article. Clears up the preconceptions I had about using /dev/random over /dev/urandom ("it's more secure!") and explains why in very straightforward language.

panic• 11 years ago

Why doesn't Linux follow FreeBSD and provide a single interface (under two names, maybe, for compatibility)? Is there any reason to use /dev/random?

geofft2• 11 years ago

Ego. They've put a lot of work into the entropy-tracking thing and they don't want to admit that it wasn't needed.

(Ego is the reason for a bunch of other security misfeatures in Linux. Securelevel comes to mind, where Linus explicitly said after fifteen years, okay, this was in fact the right model all along, you can merge it, just call it something other than securelevel so I don't have to eat my words about securelevel being a mistake.)

wangweij• 11 years ago

This is my understanding: /dev/urandom is just a blind producer returning any number in some pool; /dev/random is a producer as well as an inspector, it returns the same number from the pool but it also has an extra eye on the source of that pool and refuse to work if it believes the source is of low quality -- here, not enough entropy.

mikeash2• 11 years ago

The problem is that /dev/random is too paranoid. It refuses to work if there isn't enough entropy in the pool, yes. But when it produces numbers, it subtracts that amount from the supposed amount of entropy in the pool. Thus it will refuse to work again pretty soon in a lot of circumstances where it's still completely safe to proceed.

For a concrete example, you start out with zero bits in the pool. /dev/urandom will produce a predictable stream of bits. This is extremely bad. /dev/random will block. this is good.

Now you add 1024 bits to the pool. Both /dev/random and /dev/urandom will produce good numbers.

Now let's say you read 1024 bits from /dev/random. This will reduce the entropy counter back to zero. If you then try to read another 1024 bits from /dev/random, it will block.

But this is nonsense! Those 1024 bits you added before aren't depleted just because you pulled 1024 from /dev/random! It is perfectly safe to proceed generating more numbers at this point (or at least it's as safe as it was before), but /dev/random refuses to.

Many systems don't add entropy to the pool very quickly so it's entirely possible to "deplete" it in this way. Then your code using /dev/random wedges and you have a problem. However, few systems are ever in a state where they have zero entropy. Thus the advice to use /dev/urandom.

Ideally, you'd want a device which blocks if and only if the entropy pool hasn't been properly seeded yet, but which never blocks again after that point. Apparently this is what the BSDs do, but Linux doesn't have one.

lsh123• 11 years ago

"But this is nonsense! Those 1024 bits you added before aren't depleted just because you pulled 1024 from /dev/random!"

An observer of the produced random numbers can potentially deduce the next numbers from the first 1024 random numbers. This is the reason why /dev/random requires more randomness added -- to prevent the guessing of the next number.

tveita• 11 years ago

Ideally, you'd want a device which blocks if and only if the entropy pool hasn't been properly seeded yet, but which never blocks again after that point. Apparently this is what the BSDs do, but Linux doesn't have one.

The getrandom() call added in Linux 3.17 does this. By default it reads from urandom, but it blocks until it estimates that urandom has been seeded with "a sufficient number of bits", which seems to be around 128 bits of randomness.

http://man7.org/linux/man-pages/man2/getrandom.2.html

When available getrandom() should probably be preferred over both /dev/random and /dev/urandom.

zackelan• 11 years ago

Now you add 1024 bits to the pool.

A further complication - you don't actually add 1024 bits to the pool. Instead, you add N bits to the pool (N > 1024), estimate its entropy as 1024, and increase the overall entropy estimate by that much.

Estimates are always pessimistic, because it's better to be cautious and under-estimate the entropy than the alternative. So you might have 2048 bits or more of real entropy, but that can still be "depleted" by a request for 1024 bits.

The paper that describes Fortuna [0] goes into greater detail on this. Yarrow, the CSPRNG that preceded Fortuna, attempted entropy estimation. Fortuna rejects entropy estimation, and tries to build a CSPRNG that is secure without needing estimates:

"...making any kind of estimate of the amount of entropy is extremely difficult, if not impossible. It depends heavily on how much the attacker knows or can know, but that information is not available to the developers during the design phase. This is Yarrow’s main problem. It tries to measure the entropy of a source using an entropy estimator, and such an estimator is impossible to get right for all situations."

"Fortuna solves the problem of having to define entropy estimators by getting rid of them."

0: https://www.schneier.com/fortuna.pdf

msm23• 11 years ago

I prefer to trust the NSA on these matters. They end up saying much of what the author has written, but they make it clear why you want to use one vs the other.

The excerpt below is from https://www.nsa.gov/ia/_files/factsheets/I43V_Slick_Sheets/S... (which in turn also references https://www.nsa.gov/ia/_files/factsheets/I43V_Slick_Sheets/S... )

Unix-like Platforms (e.g. Linux, Android, and Mac OS X):

Application developers should use the fread function to read random bytes from /dev/random for cryptographic RNG services. Because /dev/random is a blocking device, /dev/random may cause unacceptable delays, in which case application developers may prefer to implement a DRBG using /dev/random as a conditioned seed.

Application developers should use the “Random Number Generators: Introduction for Operating System Developers” guidance in developing this solution. If /dev/random still produces unacceptable delays, developers should use /dev/urandom which is a non-blocking device, but only with a number of additional assurances:

- The entropy pool used by /dev/urandom must be saved between reboots. - The Linux operating system must have estimated that the entropy pool contained the appropriate security strength entropy at some point before calling /dev/urandom. The current pool estimate can be read from /proc/sys/kernel/random/entropy_avail.

At most 2^80 bytes may be read from /dev/urandom before the developer must ensure that new entropy was added to the pool.

wfleming• 11 years ago

Maybe I'm just tired & not able to detect sarcasm right now, but isn't 2^80 bytes more bytes than are currently stored in the world? That's on the order of 10^24, which is something like 1 million exabytes, which is a million squared terabytes, right?

raverbashing4• 11 years ago

No, it's called being safe

And I wouldn't put it past them to have an attack (at least theorectical) that exploits this

mfukar• 11 years ago

The amount of digitally stored information in the world as of May 2009 [1] is in the order of 2^71. There's no way a single process in a Linux distro will live long enough to read 2^80 bytes for the foreseeable future.

[1] http://www.theguardian.com/business/2009/may/18/digital-cont...

nshepperd2• 11 years ago

Is the joke that this is awful advice?

(Except for saving the entropy pool between reboots, which can be useful, afaik.)

viraptor5• 11 years ago

in which case application developers may prefer to implement a DRBG using /dev/random as a conditioned seed.

Wait, what? dev/urandom is a DRBG seeded the same way dev/random is. This doesn't seem to make sense.

sago• 11 years ago

Nods

You're very unlikely to be continually reseeding your own DBRG with new entropy, so it will be less secure than /dev/urandom, which is.

Suspiciously bad advice, there, from what I can see.

ben_bai• 11 years ago

application developers may prefer to implement a DRBG using /dev/random as a conditioned seed

No. Doing random in userland is just wrong. If your program has access to /dev/urandom, use it. If not, use arc4random().

“Random Number Generators: Introduction for Operating System Developers”

Or look how OpenBSD does it. (getentropy(), arc4random(), the subsystem)

The entropy pool used by /dev/urandom must be saved between reboots.

OpenBSD does this, and more. The bootloader basically seeds the kernel with old entropy from before the reboot.

geofft2• 11 years ago

- The entropy pool used by /dev/urandom must be saved between reboots.

Every single Linux distro does this.

tytso• 11 years ago

Actually, the preferred and recommended way to get randomness on modern kernels is to use the new getrandom(2) system call, with the flags argument set to zero.

http://man7.org/linux/man-pages/man2/getrandom.2.html

With the flags set to zero, it works like the getentropy(2) system call in OpenBSD. In fact, code that uses getentropy(buf, buflen) can be trivially ported to Linux as getrandom(buf, buflen, 0).

plonh• 11 years ago

Why didn't we get /dev/getrandom?

viraptor5• 11 years ago

One point of a direct syscall is that you don't have to open any files. That protects you from file descriptor exhaustion attacks.

kalleboo• 11 years ago

Here's the original syscall request http://lists.openwall.net/linux-kernel/2014/07/17/235 and discussion https://news.ycombinator.com/item?id=8049180

hsivonen2• 11 years ago

If the right way to use the Linux system call is to use it with flags set to such that it becomes equivalent to the OpenBSD system call,

1) Why does the Linux system call have flags at all?

2) Why does the Linux system call have a different name?

I.e. why not clone the OpenBSD system call exactly with the same name and semantics and not give the userland any flags to shoot self in the foot with?

masklinn2• 11 years ago

http://thread.gmane.org/gmane.linux.kernel/1750023

(you certainly don't have to agree with that, currently all the flags can do is draw from /dev/random instead of /dev/urandom, and do so without blocking, but that's the reasoning behind the interface change and thus the name change)

technion• 11 years ago

Unfortunately at the moment, it's not a safe call that any particular system is actually running a modern enough kernel imo.

getrandom() showed up in 3.17, and the latest CentOS distribution ships with 3.10. And then you've got older deployments.

You could play with autoconf hell of detecting getrandom() and compiling for it if needed, but I couldn't imagine it to be worth the effort at the moment if you're going to be maintaining a branch of fallback code anyway.

beefhash• 11 years ago

This is all fine and dandy for seeding a CSPRNG, but we still don't actually have a CSPRNG in the C stdlib or glibc. getrandom(2) seems to target exclusively getting entropy for seeding, looking at the man page ("These bytes can be used to seed userspace random number generators or for cryptographic purposes."). And of course, the syscall is very recent, introduced in Linux 3.17, coming after the kernels in Debian stable (jessie), which features 3.16, and CentOS/RHEL7, which features 3.10.

OpenBSD and NetBSD feature a ChaCha-based arc4random; FreeBSD and libbsd still seem stuck with RC4-based arc4random[1]. An equivalent for that is sorely missing on Linux. /dev/urandom requires messing with file descriptors, which you may run out of and may require error handling, plus all kinds of security precautions to make sure you're actually looking at the right /dev/urandom[2].

[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=182610 and https://bugs.freedesktop.org/show_bug.cgi?id=85827

[2] http://insanecoding.blogspot.ch/2014/05/a-good-idea-with-bad...

ReidZB3• 11 years ago

getrandom(2) uses a CSPRNG under the hood and can be treated as a pre-seeded one.

It's unfortunate that it's so recent, though.

justincormack• 11 years ago

There is an open request to add a posix_random (basically arc4random) to posix http://austingroupbugs.net/view.php?id=859

frankzinger• 11 years ago

Previously: https://news.ycombinator.com/item?id=7359992

antirez2• 11 years ago

The root problem is that cryptography is not usable without some understanding. In the latest days we developed an addiction for "best practices" like "use bcrypt", "use /dev/random" or alike which in the field of cryptography are not enough: understanding is the only possible path. Actually, this is true for most things, it just shows up more obviously in the field of cryptography because of the security concerns. So we should replace "use bcrypt" with "Read Applied Cryptography & other books".

ReidZB3• 11 years ago

The average guy doesn't have the time to read enough about cryptography to have the sort of knowledge you're wanting. This very article debates an understanding created by incomplete and poor cryptographic knowledge.

If there's anything I've learned by studying cryptography, it's that the average person needs to invest significant resources to become a pseudo-expert in cryptography - all before they can make any meaningful decision.

Applied Cryptography is not a terribly good starting point for cryptographic understanding these days. That's just another facet of difficulty: where does one even start? Do you want a theoretical baseline understanding? Do you want the high-level, quick-and-dirty overview which only gives a summary? (That'll make it hard to make real, informed decisions...)

If we really want to make cryptography accessible to many developers, the best solution is for the cryptographic community to make our libraries and interfaces better. At the same time, there comes a point where a developer has to stop and say "this is beyond what is standardized in ${widely used library}; we need to hire a cryptographer." In that sense, we need to instill a better anti-crypto ethic.

antirez2• 11 years ago

Basically to become an expert cryptographer you need a math degree and ten years of experience, so this is out of question indeed. What I'm referring to is to get enough information in order to understand the big picture: what is a stream cipher, what a block cipher, a cryptographic hash function and its main properties, how many of those primitives are kinda equivalent sometimes and you can use one to create another, the tradeoff between speed and security (and how number of rounds effect the security of crypto building blocks), analyzing simple algorithms in order to really understand why it is so hard for you to create something secure, secure PRNG generation and weak PRNG generation (and how to break a congruential linear generator), algorithms like DH, RSA, basic knowledge on number theory, and so forth. This will not make you an expert, but will give you enough understanding in order to actually undetstand why a rule or a best practice is used and when it is safe or not to break it.

About starting point, this is incredibly sad but true: there is no Applied Cryptography of 2015. The book is at this point in some way outdated and no replacement exists, however what you can do is to read it, and then to read the documents that there are around to get updated information. Also there are now the online courses on cryptography that really help. This may look like an overkill, but at this point crypto is everywhere and is the foundation of most things secure, so it is a requirement of everybody involved with computer security.

noamyoungerm• 11 years ago

The average person doesn't have the time needed to learn how to build a house, either. As a society, we solve this problem by delegating the construction industry to a group of people who specialize in it, and whom we can trust to do things properly.

There's no way around the fact that security is complex. If it's too time consuming for the average developer, then they shouldn't be doing it at all.

jfoutz2• 11 years ago

I think for ordinary programmers, stories of failures might be more effective. Case studies, perhaps even a bit embellished to make them more compelling. Something along the lines of Kurzweil's cuckoo's egg. Except shorter and more recent.

I think what you're getting at is a specific technology won't save you, it's the security mindset that's needed. Convincing programmers there really are bad people who want to pick apart your systems is the problem. Once their convinced, once they take security seriously, they'll do better.

They might start out doing a terrible job, but with the security mindset, they'll improve. They'll seek out problems and solve them. Rather than pretending it's not an issue, or blindly apply security secret sauce like "use bcrypt"

wglb• 11 years ago

What is Kurzweil's cuckoo's egg

Do you perhaps mean Cliff Stoll?

1_player• 11 years ago

Is there anyone that, with full access to the machine/kernel, has managed to predict the output of random/urandom?

I know that a PRNG is predictable if you know all the input variables, and the code for it is publicly available, but has anybody in practice been able to exploit that?

EDIT: that's an honest question. I'd like to read a paper about that.

vbezhenar• 11 years ago

Java is known to read /dev/random when dealing with SecureRandom class. In particular it might cause extremely slow starting time for Tomcat on fresh virtual machine. There's parameter "-Djava.security.egd=file:/dev/./urandom" and I always felt unsafe using it. Thanks to this article, now I won't regret it even theoretically.

Fun thing is, if you pass "/dev/urandom" to this parameter, Java will read /dev/random anyway. May be that was a wise decision 20 years ago.

Myths about /dev/urandom

Related Stories