Every Byte of a TLS Connection Explained and Reproduced

https://tls.ulfheim.net/

by aberoham • 8 years ago

1185 103 8 years ago

tls.ulfheim.net

jugg1es• 8 years ago

So does this happen for every single request? or is this based on a single session.

sgwae• 8 years ago

This is what happens under the hood of every ping request.

noselasd• 8 years ago

ping would refer to https://en.wikipedia.org/wiki/Ping_(networking_utility) in this context, which is quite unrelated to TLS.

deaps2• 8 years ago

A ping is very much different. A ping is (typically) simply an ICMP Echo Request, (not TCP, thus no TLS, etc). The receiving device, if accepting echo requests and configured to reply with echo replies, then responds with an ICMP Echo Reply - or some device in the middle (or the device itself could respond with an ICMP unreachable, or some other response - or quite simply drop the ICMP Echo Request entirely and silently).

*Edited an incorrect UDP reference out based on the below comment.

Brotkrumen• 8 years ago

Per session.

logicallee5• 8 years ago

Unrelated but: based on your understanding of establishing a TLS session with a server, and then traffic through that connection, do you think the new Gmail user interface for the web (desktop) is sufficiently speedy?

The reason I ask in this thread is that this thread treats some of the low-level minimum traffic necessary between clients and servers.

syncsynchalt17• 8 years ago

The two "application data" records at the bottom are what would wrap around a request/response for an established connection - you can see how much is added to make the strings "ping" and "pong" (though some of that is padding to expand it to a multiple of 16 bytes).

And much of the rest of the records might go away completely if the session were resumed from the client's memory (this wasn't demonstrated here).

rqs• 8 years ago

Explained in detail, also the web page is well-built.

Thank you.

rajeshamara• 8 years ago

Yes very useful. Thanks for your effort.

hathym• 8 years ago

nice work!

henrychongs• 8 years ago

Very detailed explanation. Useful even for a refresher on the mechanics. Well done!

pcunite• 8 years ago

Nice, could have really used this a few years ago when I was making my own HTTP server implementation. Could someone make a PDF of this? Any C++ and C# examples of this?

syncsynchalt17• 8 years ago

I might post a PDF later (as the content becomes final), or make a printable view. In the meantime try this:

    # in your javascript console, paste this:
    [].forEach.call(document.querySelectorAll(".record, .calculation"), function(el){el.classList.add("selected")});
    [].forEach.call(document.querySelectorAll(".record, .calculation"), function(el){el.classList.add("annotate")});
    [].forEach.call(document.querySelectorAll("codesample"), function(el){el.classList.add("show")});

Then you can print the page to PDF.

rohansingh• 8 years ago

Beautiful. A few years ago, were implementing some TLS handshake stuff at Spotify (https://github.com/spotify/ssh-agent-tls) to be able to use your SSH key as a client-side cert for HTTPS connections.

The first couple drafts took forever to figure out, and I got a bunch of stuff wrong. This guide would've saved a ton of time back then.

bungie4• 8 years ago

Beautiful.

I recently had to implement two way auth over 1.2 and this would have saved much hair pulling. (and who does TWO way auth over an MPLS connection. Turns out, us).

pimlottc• 8 years ago

I feel your pain. Two-way TLS is a funny thing, it's supported by the standards and even most implementations but its actual use is minuscule compared to "normal" one-way TLS, so much so that it's hard to find documentation even acknowledging two-way TLS exists, let alone how to use it.

And don't get me started about the hassles of obtaining signed certificates that are actually usable for client auth...

tialaramex5• 8 years ago

And don't get me started about the hassles of obtaining signed certificates that are actually usable for client auth...

What sort of clients were you authenticating? The Web PKI needs to be trusted by random people from the whole world, but most mutually authenticated systems have a relatively small number of clients which are known to the server operator out-of-band. So probably the Web PKI is not the right choice. Instead you (the server operator or some neutral facilitator if it's a group of providers operating services for the same clients) should operate a CA for this purpose, not piggyback on the Web PKI.

One reason not to use the Web PKI if you aren't actually part of the public Internet is that we, to put it bluntly, don't give a shit about people who do that. Running a PKI is expensive (not just in dollar terms, it needs a bunch of smart, motivated people who are morally upright or it's worthless), and this one is ours, so it obeys our rules.

If you have your own PKI (or just one CA) you set the rules. Fifty year certificates for 1024-bit RSA? Why not. A current passport photograph baked into every certificate? Sure. Want the issuer to mint the keys and keep a copy? Do as you please. All those things are prohibited in the Web PKI.

taftster• 8 years ago

Most two-way implementations are generally privately controlled. You can act as your own certification authority and sign both the client and server certificates for a private communication.

Totally curious, it sounds like you were going for a traditionally signed certificate approach. So you had clients that you didn't "own" -- normal Joe Public -- using your service? If so, that is definitely way outside the norm. If not, why didn't you just sign your own certs?

munchbunny• 8 years ago

For client auth, assuming you control the server, would it be easier to just issue your own and install the root on the server?

tetha• 8 years ago

Would you be interested in some articles about his? A lot of our infrastructure bases trust around self-signed CA chains and mutual TLS authentication in different configurations, mostly involving elastic stacks, vault, nomad, consul - and we're migrating a lot of the cert handling to be fully automated. We're also extending this to a number of our java applications as well.

I might be able to shed some light on there, as this topic isn't actually that hard. It just requires some very careful thinking.

the_clarence3• 8 years ago

That's what the Noise protocol framework is for.

DontSueMeBro• 8 years ago

This is very nice. I think the green on green and blue on blue highlight are a little hard to see.

syncsynchalt17• 8 years ago

Agreed - the color choice is one of the things I never really felt happy with (I'm not a frontend guy).

thejoeflow• 8 years ago

Kind of ironic that their website is using an invalid security certificate?

int0x802• 8 years ago

It looks pretty valid to my browers (chrome/firefox)?

tialaramex5• 8 years ago

The site is currently presenting a Let's Encrypt certificate issued just under 24 hours ago. There have historically always (for reasonable definitions of always) been valid certs for this site, although of course I can't tell from here that they were always properly installed.

Likely explanations for your experience:

1. Your clock is wrong. If your system currently thinks this is Thursday 11 October for example, that's a problem, 'cos this is Friday 12 October.

2. There's some subtle configuration error on their server (seems unlikely as it looks to be just a generic AWS setup) that results in the wrong certificate being presented.

3. Your OS or browser trust store lacks the root CA "DST Root CA X3" operated by IdenTrust. If you didn't deliberately choose to do this, you should investigate as most likely you aren't getting important security updates.

All three causes can often by diagnosed by closely examining the detailed error reported in a browser e.g. SEC_ERROR_EXPIRED_CERTIFICATE

y4mi• 8 years ago

https://news.ycombinator.com/item?id=18201211

it mightve been invalid while he set up his cdn / reverse proxy?

int0x802• 8 years ago

Thanks, very useful.

kodablah3• 8 years ago

There's a lot of practicality I learned from reading the Go source and implementing my own (e.g. always an array of a single value of 0 for compression method).

Now do a version with DTLS (one more message type, couple more fields on existing types, and logic concerning retries). Also, now do a TLS 1.3 one.

syncsynchalt17• 8 years ago

Yep, reading the Go source is what inspired me to do this, it's still very compact and readable.

I originally used an AEAD cipher but found it was impossible to demonstrate on the command line (openssl enc refuses to do AEAD because it can't confirm the authentication in a streaming context).

A friend asked me to demonstrate ALPN in this but as I looked into it I found it was distracting, as there's already so much going on and any new feature required digression. Maybe next time!

As for 1.3 my next project was going to be implementing it rather than documenting it. Just a throwaway implementation, nothing you'd want to use.

kodablah3• 8 years ago

As for 1.3 my next project was going to be implementing it rather than documenting it. Just a throwaway implementation, nothing you'd want to use.

Since you read the Go source, you might like [0]. I will say I personally think Go could have done better. I think it's too compact, too hidden, too disorganized, too underdocumented, and too inflexible/non-extensible. I began to pick some of it apart for a DTLS impl I started at [1], but have put on temporary hold yesterday due to other work obligations.

0 - https://github.com/cloudflare/tls-tris 1 - https://github.com/cretz/go-dtls

syncsynchalt17• 8 years ago

Link to the go source: https://golang.org/src/crypto/tls/

syncsynchalt17• 8 years ago

Author here - I was going to publish this today but it leaked out ahead of time. Enjoy!

EDIT: I'm putting a CDN in place.

randlet• 8 years ago

Thanks! This is really a fantastic resource. Something like a TLS connection is one of those things that seems really intimidating but when you see it laid out like this it makes you feel like 'I can understand this!'. Great work.

bebop• 8 years ago

Great illustration, I am going to share this with the rest of my team. One thing that I would find interesting is how x509 client certificates fit into the negotiation. I know this is optional so I don't know how it would fit into your flow easily though.

syncsynchalt17• 8 years ago

Yes, I thought about explaining client certs and others (the first request I got was to add ALPN to the connection), but there's already so much to talk about in even this simple connection that I thought it would detract from the document as a whole by making it even longer and denser.

I didn't even get a chance to explain the (normal, server-side) x509 certificate signing much because it just kept taking over the document. TLS is complex enough that just explaining the happy path is 400kb of HTML.

zzzeek• 8 years ago

very nice! first thing I noticed is that at https://github.com/syncsynchalt/illustrated-tls/blob/master/... the content and samples are hardcoded into HTML - might it be nice if this was generated from some kind of JSON file or similar such that the approach you have here could be generalized to support any network protocol someone might want to annotate with descriptions?

syncsynchalt17• 8 years ago

Yes, if I do this again I will definitely generalize it into a content generator.

(Isn't that what every site turns into eventually, a custom CMS?)

As it is it's all tcpflow, hexdump, and vim.

jor-el• 8 years ago

Kudos! This a great work. I remember few years back when I was exploring TLS, there were no such resources and it took many months of trial and error to get some reasonable understanding. I tried doing a similar thing, but was only able to do for couple of packets [0]. Surely such an illustration will go a long way to help newbies to understand a complicated protocol.

[0] https://serializethoughts.com/2014/07/27/dissecting-tls-clie...

mito88• 8 years ago

A canadian?

Nice.

rustcharm• 8 years ago

Can you do this for DTLS, too? :=-)

syncsynchalt17• 8 years ago

This is the third request I've gotten for DTLS. Where are you weirdos coming from? :)

TimJYoung• 8 years ago

I did a presentation at our local software meetup on this topic a month ago, and wow, I could have really used this. It's way clearer and more organized than my muddled slides.. :-)

Very nice work !

toast0• 8 years ago

This is wonderful!

I may make a version where the bytes used for lengths are highlighted, since it feels like so many bytes are lengths; look at the SNI extension, which has three 16-bit lengths, I know why they're there, but SNI probably shouldn't be a list, and even if it was a list, an extension that consists solely of a list has a list of the length of the extension, you shouldn't need two bytes for that, and if we recast sni into just a type and a string, the string is clearly going to take the rest of the extension length, so it doesn't need a two byte length either.

syncsynchalt17• 8 years ago

Agreed, and I think some pain must have gone into those redundant length bytes. The way they've done it makes it very easy and natural to extend any part of any record later, but it gets ridiculous when you have to document them.

userbinator3• 8 years ago

Moreover, from a security perspective since we're talking about TLS, "overspecified" and/or "redundant" lengths are just begging to be made inconsistent and a source of vulnerabilities.

the_clarence3• 8 years ago

Usually if you mess up something there your implementation just doesn't work.

taco_emoji2• 8 years ago

Site's timing out and google cache is giving me a 404...

mirages• 8 years ago

HN hug of death ? Page is timeouting

kalimatas• 8 years ago

The irony is: I get "Your connection is not secure" when trying to open the link.

syncsynchalt17• 8 years ago

I've uncapped MaxRequestWorkers while I wait for the Cloudflare queue, it should work now (I believe you were seeing an error related to TLS timeout).

ct520• 8 years ago

As someone that frequently deals with large scale integration failures I greatly appreciate this site. Thanks syncsyncchalt!

quadyeast• 8 years ago

I have no need to look at that page at length ... but I am. This is very well made and interesting.

gbajson• 8 years ago

Has anyone of you seen such beautiful explanation for other protocols (TCP, 4-way handshake)?

syncsynchalt17• 8 years ago

They're in text form, but I've always loved the clarity of W. Richard Stevens' (RIP) books on TCP/IP.

He did a great job of demonstrating every byte and jitter on the wire and how it related to the underlying BSD TCP/IP stack.

Xeanort• 8 years ago

You can usually use Wireshark to find out the meaning of each byte for internet connections (and images), it works great.

tialaramex5• 8 years ago

However, Wireshark can mislead you if you don't understand what you're looking at.

It's a bit like having a low-level debugger. If you're happy with low-level C, and you're looking at a debugger and it says variable 'k' which you know is a uint8_t which loops from 0 to 5, currently has the value 65 you should say to yourself. "Hmm, I bet that the compiler used the same place to store variable 'c' that's a single byte from a text string, so this is just the capital letter A and the compiler has realised it doesn't need to store the value of k even if it's technically in scope..." rather than "OMG my loop variable somehow massively exceeded its expected range, maybe cosmic rays have damaged the RAM".

With TLS for example if you give it a whole TLS 1.2 sesssion, Wireshark will say oh, this is TLS 1.2. Fine. But if you show it only a TLS 1.2 connection that failed, Wireshark will say "Oh, this is TLS 1.0". Why? Well, the low-level protocol has been bodged over the years because of crappy middleboxes, so Wireshark doesn't actually know for sure, and rather than say "I don't know yet, I need to see more of the connection" it says TLS 1.0

This can be a problem because you'll get amateurs saying "Our system can't talk to your server because you only do TLS 1.0" and you say "No. You are wrong" and they say "Look, here's a Wireshark trace" and sure enough Wireshark is telling them it's TLS 1.0 because their system has disconnected early (e.g. because they disabled all the crypto algorithms you allow), and so Wireshark wasn't sure and labels it TLS 1.0 rather than TLS 1.2

This is going to happen again with TLS 1.3. TLS 1.3 deliberately says "Hi I'm TLS 1.2" (middleboxes again) and so that's what Wireshark will report (until you get a newer version that knows to look inside the supported_versions extension field for the version) and so you can bet that amateurs are going to say "Your service only does TLS 1.2" when actually their connection failed for some reason and they don't understand how to read the Wireshark trace.

e12e2• 8 years ago

I'm reminded about the STEPS report of using ascii art from the rfcs as a DSL for describing tcp:

Writeup/comment: http://www.moserware.com/2008/04/towards-moores-law-software...

Comment thread: https://news.ycombinator.com/item?id=846028

The vpri texts: http://www.vpri.org/writings.php

Appendix e and section "A Tiny TCP/IP Using Non-deterministic Parsing" of "STEPS Toward The Reinvention of Programming: First Year Progress Report, December 2007.":

http://www.vpri.org/pdf/tr2007008_steps.pdf

sigi45• 8 years ago

Cool! Would you be able to add a ascii view beside it?

kodablah3• 8 years ago

Not trying to come off trite, but the RFC [0] has a simple ASCII diagram of the message flow and the structures that follow are fairly easy to read. Granted you have to hop to a couple of other RFCs to understand extensions and maybe even real world impls to understand some changes (e.g. no time in the random block of client hello), but it's worth perusing if you're interested.

0 - https://tools.ietf.org/html/rfc5246#section-7.3

syncsynchalt17• 8 years ago

I think they mean a side-by-side ASCII interpretation like that supplied by `hexdump -C` - where any alphanumeric character is printed as itself and every other character is printed as a "."

It gives a nice strings-like view of raw data.

syncsynchalt17• 8 years ago

I was afraid that it wouldn't fit and might be overly complex for what was gained (there's not much text in there). However there's a PCAP at https://github.com/syncsynchalt/illustrated-tls/tree/master/... if you'd like to load it up in wireshark or tcpdump.

nanidin• 8 years ago

tls.ulfheim.net - "This site is blocked due to a security threat."

ulfheim.net - no problem

At least according to my megacorp threat filter. I have never actually seen something blocked before, and it's a shame because the page would be great to share with my team.

jms703• 8 years ago

This is one of the major problems with threat feeds.

syncsynchalt17• 8 years ago

Wonder why that would be, tls.ulfheim.net is a CNAME to ulfheim.net, they're just different apache vhosts. Same cert (using SAN).

The only things I can think of:

  - it doesn't like the hostname (tls?)
  - the hostname is new, and has no reputation
  - too much h4cking content

Obi_Juan_Kenobi2• 8 years ago

So the record header has 2 bytes for the payload size, and the handshake header has 3 bytes. Am I correct in thinking that the 3 bytes of the handshake header is superfluous? Isn't payload size limited to 2^16 bytes

zamadatix• 8 years ago

I think this is explained by the optional compression which is handled at the record layer.

parrik• 8 years ago

nice (y)

sizzzzlerz• 8 years ago

very fine work. the data flow and annotation are clear and concise. good job

lxe• 8 years ago

I love this. Excellent work. Would be cool to see a step-by-step niblets of client/server implementation of each step.

truth_seeker• 8 years ago

Thank you for sharing.

phaedrus• 8 years ago

I love the way you present the data (where clicking on the hex values of the bytes gives their explanation). At work I work on software that communicates via RS-232 serial packets; I should make something that turns logs of the packet data into HTML files with an interface like this!

umanwizard3• 8 years ago

As a crypto ignoramus - Why is the random data from each side necessary? Why can’t things just be encrypted with the PreMasterSecret directly ?

erseveR• 8 years ago

It's about the nonce / initialization vector. Basically it is used inside the cryptos primitives to add some entropy to the encryption itself and prevent a various ranges of attacks. A good introduction to crypto is "Introduction to modern cryptography" imho

tialaramex5• 8 years ago

Involving random data gives everybody who gets to pick the random data (so in TLS that's both client and server) a freshness guarantee.

Because the other party needed to know you'd picked this particular random data to make the keys, the messages from them encrypted with those keys couldn't possibly have been pre-recorded / replayed.

In the ephemeral Diffie Hellman modes both parties contribute to the key anyway so this isn't as important, but with old school RSA the random values are the only thing preventing Replay attacks.

TLS 1.3 capable servers also scribble "DOWNGRD" in part of the random field if a client message says it can't do TLS 1.3. If a TLS 1.3 client sees that unusual "random" choice it knows bad guys tampered with the connection (attempted a downgrade attack). If bad guys just change the values, they won't match between client and server and the connection aborts. Older clients think nothing of the unusual random value and carry on as before.

userbinator3• 8 years ago

I haven't looked at the spec in detail, but does this mean that random generation has to specifically exclude that "sentinel value", lest it accidentally occur?

amenghra• 8 years ago

http://www.networksorcery.com/enp/Protocol.htm has a bunch of network diagrams for various protocols.

This website lays things out really nicely, would love to have more protocols :)

blazespin• 8 years ago

I remember implementing ssl in java when I worked for Netscape back in the day. Nostalgia rush looking at this. Great stuff

userbinator3• 8 years ago

The parties had agreed on a cipher suite using ECDHE, meaning the keypairs will be based on a selected Elliptic Curve, Diffie-Hellman will be used, and the keypairs will be Ephemeral rather than using the public/private key from the certificate.

I think it's important to mention that even with ephemeral cipher suites, the server's ephemeral public key is signed using the server's certificate private key and verified by the client, since otherwise one would be able to MITM the key exchange.

syncsynchalt17• 8 years ago

It’s in the Server Key Exchange record (from memory) but i probably didn’t explain it well.

max23_• 8 years ago

Good work on the illustration.

Using Diffie-Hellman to generate a shared key with each party's private key and the other party's public key is the part that amazed me most when I was trying to understand the handshake back then.

kapad• 8 years ago

There's a typo in the server keys calculation. It should read "server" and not "client" IMO.

based2• 8 years ago

https://www.reddit.com/r/netsec/comments/9nk764/the_illustra...

Every Byte of a TLS Connection Explained and Reproduced

Related Stories