So, you want to crypto

http://blog.existentialize.com/so-you-want-to-crypto.html

by bqe • 13 years ago

162 73 13 years ago

blog.existentialize.com

theboss• 13 years ago

TL;DR - If you want to do crypto then learn crypto.

If you want to learn crypto and do crypto then certainly start with this. Then, when doing crypto...practice. Build it and reach out and ask for help and talk to people who know what they are doing and learn from them. Ask them about problems you encountered and ask them about the best ways to solve them...otherwise you will continue to make the same mistakes.

phaus3• 13 years ago

Do not let users use your product until it's been vetted.

Its OK to let them use it so you can have a large user-base to test with, you just need to explain to them that it isn't proven secure. As in, explicitly tell them that they are under no circumstances to use it with sensitive information.

Playing around with cryptography is the only way to learn it, you just have to remember to tell people that playing is exactly what you are doing.

EpicEng2• 13 years ago

But how many people are going to be interested in a crypto product which is unproven? Sure, perhaps a few technical types will give it a whirl, but most people just want something that works. If I'm in the market for something like this I'm certainly not going to use a product with a big disclaimer stating "Hey, we don't really know if this works yet, but help us test it out!" This is mission critical stuff, not some new URL routing framework.

phaus3• 13 years ago

How many people would have been interested in a digital currency when it was worthless and not accepted as a form of payment anywhere?

Perhaps the general population won't be interested, but techies are often willing to play around with unproven technologies.

tptacek7• 13 years ago

Two things.

First, while "playing around with cryptography" may be the only way to learn it, building cryptographic systems is just about the worst way to learn. Professional cryptographers start by cryptanalyzing targets and use that experience to inform their future designs. On the other hand, veteran implementors who have never taken the time to learn how to break crypto turn out protocols and designs that are repeatedly broken. You can see that right now with TLS and the TLS working group, which still hasn't fixed MtE block ciphersuites because veteran implementors can't get it through their heads that MtE is a design flaw.

Don't learn by building. You need to learn by breaking.

Second, more than one project has done the tightrope walk of telling their users "this isn't really safe" but then misleading (innocently or not) non-savvy users into trusting them. One project made it clear that their system wasn't "ready" to defend against nation-state adversaries... but then suggested that maybe it would be good enough for journalists, and even promoted it at an event for teaching journalists cryptography. It was later comically broken.

Be honest with yourself. Crypto doesn't get beta-tested into resiliency. Strong systems start out strong. If you're building something because its your dream to thwart the NSA, don't kid yourself into thinking that you'll get there by first protecting people's Warcraft clans.

phaus3• 13 years ago

I get that you need to learn how to break software as a major part of the learning process. I also understand that a broken system often can't be fixed, and that if your system breaks you might have to start from scratch. But does one go from becoming an expert at breaking systems to automatically churning out a secure cryptographic system? Or does one become an expert at breaking and then build a bunch of flawed crypto systems, as the learning process continues?

Maybe the process is just counterintuitive, but it seems that no matter how skilled someone might be at breaking systems, that building a sound system on the first try would still be incredibly difficult. I know that crypto systems aren't built iteratively, instead I'm talking about going back to the drawing board each time. Would there be value in that, or would it be a waste of time? I'm not yet an expert (and I am well aware that you are), though I aspire to be one day. Perhaps you would be kind enough to elaborate?

sillysaurus2• 13 years ago

Don't learn by building. You need to learn by breaking.

The best way to do this is to do the Matasano crypto challenges. The challenges are designed to get progressively more difficult, and since the goal of most of them is to break something, that means you'll learn how to employ progressively more sophisticated attacks. They're also great for a newbie, because in one challenge you'll implement something that seems impervious to attack, then in the following challenge you'll attack and break it, often via an unexpected attack vector. (The padding oracle attack comes to mind.)

You'll come away with an understanding of AES in ECB CBC and CTR modes; HMAC; timing attacks; attacks on RNGs; attacks on hashes; and a lot more. That's just off the top of my head.

For me, there were a dozen "aha!" moments in the first 30 challenges. Each of those moments now live with me and inform my future decisions. They will also make you much less confident in your ability to design secure cryptosystems, which is good.

cperciva• 13 years ago

You can see that right now with TLS and the TLS working group, which still hasn't fixed MtE block ciphersuites because veteran implementors can't get it through their heads that MtE is a design flaw.

Can't get it through their heads that MtE is a design flaw? You're more generous than I would be. At this point I'd assume that everybody knows that MtE is a design flaw, and anyone designing a protocol which uses MtE has made a deliberate decision to design a weak protocol.

yeukhon• 13 years ago

I think we should clarify that building can always be breaking. You can implement a script that automate the breaking process (think fuzzer). I can fire up Python interpreter and start playing around with AES encoding with ECB and then figure "shit, ECB is not semantically secure". I can have a damn vulnerable web app and web server. I can implement them, break them and improve them if I am really that dedicated.

I can also learn by reading standard and read how people implement in projects like Firefox or Chromium. And ask questions. I think we should be careful with the meaning of "learn by breaking" because obviously only a handful of people can really break cryto system. And average developers like me are probably better off just learn and break.

Steps like that are probably my way of getting into security engineering and cryto engineering (at some point). I guess I am a different audience, I am not here to break RSA, I am here to learn how real systems are implementing RSA, TLS and see whether those implementations have flaws or not.

greenyoda• 13 years ago

I think this quote from the article perfectly sums up the dangers of amateur cryptography:

"Cryptography isn't something you can iterate on until you get it right, because you'll never know if you do."

ReidZB3• 13 years ago

I feel like it's quoted from somewhere else, but the reference escapes me. At any rate, it's a great quote.

People who try the iterated design approach are especially frustrating. It ends up becoming a game of whack-a-mole with vulnerabilities, wherein an experienced cryptanalyst will point out an issue, the designer will say "oh! of course! let me apply a patch!", and then this continues to infinity. (This scenario doesn't necessarily indicate a bad approach, but it's certainly a symptom of iterated design.)

There's a particularly lovely story in Schneier's "Memo to the Amateur Cipher Designer" [1]:

A cryptographer friend tells the story of an amateur who kept bothering him with the cipher he invented. The cryptographer would break the cipher, the amateur would make a change to "fix" it, and the cryptographer would break it again. This exchange went on a few times until the cryptographer became fed up. When the amateur visited him to hear what the cryptographer thought, the cryptographer put three envelopes face down on the table. "In each of these envelopes is an attack against your cipher. Take one and read it. Don't come back until you've discovered the other two attacks." The amateur was never heard from again.

Part of what makes it so frustrating, though, is that usually we want to be genuinely helpful. Building cryptosystems is fun (dangerously so!), and it's really crappy to end up saying "just scrap the whole thing" or what have you. But if you want to keep your sanity...

[1] https://www.schneier.com/crypto-gram-9810.html#cipherdesign

jiggy20112• 13 years ago

Surely the correct answer is "just use keyczar"? At least 99% of the time.

tptacek7• 13 years ago

Most practitioners would recommend Nacl now.

jiggy20112• 13 years ago

Nacl or libsodium?

ReidZB3• 13 years ago

Keyczar --- and other cryptographic libraries, mostly --- don't solve the issue of protocol design. They do let you choose the primitive you want, but it's still up to you to pick a mode of operation, make sure you use authentication (AEAD mode / MAC), initialize IVs/nonces/counters correctly, etc.

NaCl is very much a step in the right direction, but it's not an end-all solution either. Key management alone is a huge issue, for example, but virtually no library helps you out there beyond providing a basic toolkit. Of course, key management is very much in the realm of policy, so that's to be expected.

rnicholson• 13 years ago

Both Applied Cryptography and the Handbook of Applied Cryptography are great resources, although they're a little dated now. ... Step one is to read Cryptography Engineering. This is not optional. Read it. It is a fantastic book that details how to use cryptographic primitives.

It seems kinda superfluous to mention Applied Crypto when the real reco is to read Cryptography Engineering. I'd almost wonder if it would be better to direct people away from Applied Crypto...

Personally, I found Applied Cryptography to be so-so at best. Practical Cryptography was a breath of fresh air in comparison.

helper• 13 years ago

Yes. Recommending Applied Cryptography is usually a warning sign that the person doesn't know what they are talking about. In this case the rest of the advice is reasonably sound for an engineer that wants to start learning the fundamentals of modern cryptography.

andrewcooke3• 13 years ago

article mentions nothing-up-my-sleeve numbers, so a topical reminder that the permutation for md2 (and rc2 apparently) is still unexplained (despite being "derived from pi") - http://crypto.stackexchange.com/questions/11935/how-is-the-m...

for all you conspiracists - this was designed by rivest, the r in rsa, now famous for cooperating with nsa... (i don't really believe that the permutation is a backdoor, but i would like to know how it's derived - rivest is famous for elegant algorithms, and for the life of me i can't find a simple, neat way to get those numbers from pi)

tptacek7• 13 years ago

A world full of brilliant cryptographers outraged at the NSA, trying to get the NSA off the IETF crypto review board, working on publishing results about NSA-sponsored crypto... and you want to talk about the MD2 and RC2 constants? What's the largest system that ever relied on MD2? Let's start there.

andrewcooke3• 13 years ago

i know you have the intellectual curiousity of a sausage, but some of us are simply interested. we don't all need to be motivated by logic.

how would you derive it? tagging pairs of hex digits and sorting doesn't cut it. would rivest have used a hack that's a biased shuffle? i doubt it.

and downvotes. happy xmas!

pbsd2• 13 years ago

There were certificates (including a root CA) using MD2 until recently. MD2 itself was only retired in 2011 [1].

https://www.rfc-editor.org/rfc/rfc6149.txt

betterunix• 13 years ago

If you want a more "theoretical" look at the theory, Introduction to Modern Cryptography by Jon Katz and Yehuda Lindell is a great book. Also good (but my copy had many printing errors) is Foundations of Cryptography by Oded Goldreich.

ReidZB3• 13 years ago

Yes! This is exactly what I was going to post. The article's recommendation to read Applied Cryptography and the HAC to "learn the theoretical background" left me dejected, since neither is particularly that great in the area of theoretical underpinnings. (The HAC is a reference book, for Chrissake!) Both are great books in their own right, but they're not what I'd recommend for the theoretical background.

Katz and Lindell's Introduction, on the other hand, is absolutely fantastic for the task (this was its design goal...). It introduces theoretical cryptography from the bottom-up and uses it to motivate the various primitives and constructions from the applied realm. It's really a great mix. The book has become my go-to recommendation for those who are serious about cryptography but have had relatively little exposure to it. It also doesn't assume the reader is an expert in all things computer science, which is nice.

Goldreich's Foundations of Cryptography is more of a treatise on theoretical cryptography... it goes much deeper and starts out assuming the reader is pretty familiar with concepts from theoretical computer science and probability theory. The optional sections of Katz and Lindell's work end up being the opening chapters of the first volume --- and they're not optional. Block ciphers aren't even treated until the second book. It's a seriously theoretical series, which makes it great in its own right, but I would postpone reading it until well-after Katz and Lindell's book. (And a book on computational complexity, at minimum, for those not familiar with it.)

ztnewman• 13 years ago

Don't listen to idiots who tell you otherwise.

Real mature.

tptacek7• 13 years ago

Did you have an actual opinion about what he was saying in the essay, or do you just want to talk about how he chose to write it?

plg2• 13 years ago

The Matasano crypto challenges are a great place to start getting your feet wet and your hands dirty.

http://www.matasano.com/articles/crypto-challenges/

Myself, I'm trying them in ANSI C

alinajaf7• 13 years ago

This is a good choice. One of the challenges had me stuck for months because I hadn't realised that you really need fine-grained control over bitshifting that e.g. Ruby doesn't appear to give you. Taking twenty minutes to re-write my solution to that challenge in C sorted it out straight away.

plg2• 13 years ago

I'm presently stuck on #6 in problem set 1...

wrs• 13 years ago

Well, it would be, except the pipeline seems to be clogged. I'm waiting weeks and months to get a response. Which is frustrating because the early problems are already really interesting, and I want more!

milhous2• 13 years ago

I'm taking an Intro to Crypto course this spring. What's interesting is that it's offered through the Math department, and assumed it was a CS class.

We'll be using this text:

http://www.amazon.com/Introduction-Cryptography-Coding-Theor...

Is this any good? Apparently a best seller in the "Software Coding Theory" category on Amazon.

kbhomes• 13 years ago

I had this same textbook for the Crypto course I just completed this semester. It's a very good textbook, in my opinion, as the descriptions and examples are really informative. Usually if I couldn't get the material through my professor's lectures, it was sufficient to look it up in the book. However, we did only briefly touch on cryptographic hashes and only a little on Legendre and Jacobi symbols, and not at all on the elliptic curve and other special topics towards the end of the text, so I can't comment on those.

The book does very good job of talking about different algorithms and concepts, often times with a very brief historical introduction, and includes thorough descriptions of various popular/important attacks of those concepts. In general it's a book I'd recommend for an introduction to cryptography. You also learn a fair introductory bit of number theory which I really enjoyed.

I also met Dr. Washington, one of the co-authors of this book, who was a very pleasant and energetic person who really enjoys the topic of cryptography.

By the way, where are you taking this course?

milhous2• 13 years ago

Thanks everyone for their reviews. Glad to hear this isn't a POS text.

I'm taking this at Millersville University as a once-a-week, 3 hour evening course. I'm a Physics and CS major, and am taking it as an elective to get a Math minor.

With all the NSA and crypto news these days, it sounds like a great time to learn about the fundamentals of crypto. And I'm curious if there will be actual programming involved because to my knowledge, there aren't any prereqs for it, not even an intro to programming course.

jknightco• 13 years ago

I took an Intro to Crypto course offered through my University's Mathematics department and we used that exact text. I'm the kind of person that requires a very good text in order to do well in a course, and the book in question was of high enough quality that I did quite well.

Although I thoroughly enjoyed the Mathematical focus of the course, my particular teacher spent a good deal of the course discussing semi-related Number Theory topics at the expense of introducing some of the more interesting cryptography concepts. That said, we still learned the theory behind the important ones: DES, AES, RSA, and ElGamal, in addition to "classical" forms such as Caesar, Vigenère, and Hill.

Another downside of the course being a Math course and not a CS course: we used Mathematica for all of our "programming" projects, which felt awkward and clunky. I remember coding my solutions in Python first, making sure they were correct, and then translating them to Mathematica after the fact.

Good luck in the course! It'll be hard work, but its absolutely worth the effort.

claudius• 13 years ago

Mathematica can feel a bit clunky at first, but if Python comes with batteries included, then Mathematica brings its own power plant…

arghnoname2• 13 years ago

Dr. Washington (one of the authors, I believe the other was a grad student of his) seems to be a really decent guy and cares a lot about teaching. He goes to middle and high schools fairly regularly to introduce kids on these topics.

That concern is reflected in the textbook and his course. It's a very approachable treatment and it seems like a lot of emphasis is placed on making it fun.

The downside to a math listed CS crypto course (as it is where Washington teaches) is that there may be less emphasis on flaws that occur in actual real word implementations. A professor in the CS department also teaches crypto at Maryland and actually uses a different book (one he wrote). My understanding is that there is a bit more attention paid to implementation there, but that's coming second-hand.

dobbsbob• 13 years ago

Most of Matthew Green's course is avail online http://spar.isi.jhu.edu/~mgreen/650.445/650.445__Main.html

derefr• 13 years ago

And don't make your cryptography project sound like snake oil. Saying military grade encryption or N-bits of security makes you sound like you don't know what you're talking about.

Interesting to contrast this with patio11's statement from just a few days ago (https://training.kalzumeus.com/newsletters/archive/sco_remin...):

People are better at remembering images than they are remembering claims or facts. "256-bit SSL encryption" is a true fact about your software product, but for most customers it goes in one ear and out the other. "Bank-grade encryption" is an image -- people can envision the vault -- and is vastly more likely to be recalled favorably when someone is worried about security.

alinajaf7• 13 years ago

For Joe Public, you say "Bank Level Security", for hackers you drill down into the details. There's no reason why your marketing material can't give the visual security imagery that people want and then walk through the exact countermeasures you're taking on your security page.

N.B. Patrick also says to use your powers for good rather than for evil :)

haberman4• 13 years ago

I'm curious to hear people's thoughts about git. Git is "crypto" to some extent, Linus does not appear to have tons of crypto expertise, and it uses SHA1 as a MAC AFAICT (which according to tptacek's earlier comment is invalid). And yet I've never heard about attacks on its crypto.

This was interesting for me to think about because it seems like a counterpoint to the article, in that it is a very successful project that came about in a very "quick and dirty" way as opposed to starting with formal protocol design.

I see that Linus disclaims the idea that SHA1 is about security: "Git uses SHA-1 in a way which has nothing at all to do with security.... It's just the best hash you can get.... It's about the ability to trust your data. I guarantee you, if you put your data in Git, you can trust the fact that five years later, after it was converted from a hard disk to a DVD to whatever new technology and you copied it, five years later you can verify that the data that you get back out is the exact same data you put in."

But it seems like avoiding attacks like this must also be a goal: http://lkml.indiana.edu/hypermail/linux/kernel/0311.0/0621.h...

alinajaf7• 13 years ago

AFAIK the only "crypto" in git is GPG used to sign tags.

The content addressable data store where all the objects are kept is basically a filesystem where every filename is the SHA1 of its contents.

If you were to generate an object that was a SHA1 collision of an existing object and inject it via a commit (without access to filesystem, otherwise the point is sort of moot) then git won't overwrite the original object with that SHA1[1].

Maybe there's some other mechanism in Git that you're referring to that uses SHA1 as a MAC that I'm perhaps unaware of?

[1] http://stackoverflow.com/questions/9392365/how-would-git-han...

haberman4• 13 years ago

Git assumes that a matching SHA1 means that the content is equal to the original content. Is that not crypto? For example, if you sign a tag, it appears to sign the SHA1 of the associated content.

This is definitely outside of my expertise, so I'm sure that my understanding is incomplete. The larger questions for me are:

- if git's SHA1 content-addressable design is not crypto, how do you distinguish crypto from software like git that uses cryptographic primitives for useful purposes?

- is a project like git a safe/sane thing for a non-cryptographer to design and implement? If so, why do all the warnings in this article not apply?

rcxdude• 13 years ago

git's SHA1 is useful in one sense cryptographically: If you have a sha1 hash, then it represents the entire state of the repository and its history, so if you have a copy of the hash you can verify that any given copy of the repo has not been tampered with (but a way to generate collisions would subvert this). This has been used in a few cases when repository servers have been broken into (and a similar feature in bitkeeper allowed the detection of an attempted backdoor insertion in linux).

josephlord• 13 years ago

You can use hashes for security but you can also use them in non-security critical systems to get a unique (with a very high probability) number for use as an ID. Git uses them for this I believe.

Attacks like mentioned can be prevented by a number of mechanisms and checks for changes. If a hash being secure helps that is great but that isn't why git uses hashes. Or by separately signing commits.

git does use crypto but reuses existing crypto (https and ssl to communicate between repos and someone mentions you can use GPG to sign commits) rather than developing it's own secure protocol.

eliteraspberrie• 13 years ago

A Git commit history is a Merkle tree. The hash is not used for authentication.

Nursie4• 13 years ago

Ok so I do want to crypto and (to the best of my ability) I already do. I follow best practices, read about the subject matter, did coursera's crypto 1 (and where the hell is pt2? 1 was awesome!). I use established algorithms and I use, well audited implementations etc etc. where available.

I have a question about MACs. We're using HMAC based on SHA256 with 32-byte keys on our new system, but our security architect only wants us to send and verify 4 or 8 bytes of the MAC output. Am I wrong to be suspicious of this? It massively reduces the number of bits an attacker has to guess or calculate, though at 8 bytes that's 128 bits so not exactly a quick brute-force...

DennisP• 13 years ago

Part 1 begins again in January and part 2 begins right after it's done.

Nursie4• 13 years ago

I hope so! By then I will have been waiting about two years!

(Can't really complain though, it's excellent and free)

arghnoname2• 13 years ago

It definitely does reduce the strength of the MAC, but it is okay if your security requirements require it. Keeping in mind that some generic birthday attacks already reduce HMAC strength to n/2 bits (IIRC), and SHA-256 has you down to 128 bits of security (with an online attack though).

Truncation is mentioned in RFC 2104. I quote:

5. Truncated output

   A well-known practice with message authentication codes is to
   truncate the output of the MAC and output only part of the bits
   (e.g., [MM, ANSI]).  Preneel and van Oorschot [PV] show some
   analytical advantages of truncating the output of hash-based MAC
   functions. The results in this area are not absolute as for the
   overall security advantages of truncation. It has advantages (less
   information on the hash result available to an attacker) and
   disadvantages (less bits to predict for the attacker).
   Applications of HMAC can choose to truncate the output of HMAC by
   outputting the t leftmost bits of the HMAC computation for some
   parameter t (namely, the computation is carried in the normal way
   as defined in section 2 above but the end result is truncated to t
   bits). We recommend that the output length t be not less than half
   the length of the hash output (to match the birthday attack bound)
   and not less than 80 bits (a suitable lower bound on the number of
   bits that need to be predicted by an attacker).  We propose
   denoting a realization of HMAC that uses a hash function H with t
   bits of output as HMAC-H-t. For example, HMAC-SHA1-80 denotes HMAC
   computed using the SHA-1 function and with the output truncated to
   80 bits.  (If the parameter t is not specified, e.g. HMAC-MD5, then
   it is assumed that all the bits of the hash are output.)

Nursie4• 13 years ago

Thankyou, that's extremely useful. Birthday attacks I had thought of, did not know sha256 was effectively 128 bits. Will dig into the rfc and other stuff and see if I can make a case for longer (maybe 16 byte) field.

I know some of the older MAC techniques (ANSI X9.19) turn out to actually aid key recovery if you use shorter MACs, which is odd...

lazyjones• 13 years ago

This is a condescending blog post by someone with an (apparently) much weaker crypto background than the telegram people he is ranting about. Of course it's much easier to post something like that than it is to actually get a rock-solid implementation at the first attempt - and we can safely assume that the telegram people do not need such advice.

Would not read again.

sidcool• 13 years ago

The author seems quite pissed at the state of crypto in the world, and he's definitely trying to help. I like the general language of the post. Good work and keep it up!

berrypicker• 13 years ago

In college cryptography was my main interest, but it was mostly theoretical (math) and little programming, which meant I was in fact useless when it came to practice because I had no experience in implementation and (I found) there are so many unknowns that one of the most important things is experience in implementing stuff in/on a specific language/platform.

I have signed up to the Coursera course and hope to brush up on basic topics and start doing more advanced crypto.

cconger• 13 years ago

I love this article. It takes a pro-active, how to proceed attitude at the same time laying out the classic pitfalls that exist. This is the tone I wish to have at all times instead of the cynical one that I undoubtedly adopt.

So, you want to crypto

Related Stories