Ethernet History Deepdive – Why Do We Have Different Frame Types?

https://lostintransit.se/2024/08/21/ethernet-history-deepdive-why-do-we-have-different-frame-types/

by un_ess • 2 years ago

175 83 2 years ago

Ironically, this version of the header published in 1980 is what we still use to this day.

IMHO Ethernet is one of the of great examples of backwards compatibility in the computing world. Even the wireless standards present frames to the upper layers like they're Ethernet. It's also a counterexample to the bureaucracy of standards bodies --- the standard that actually became widely used was the one that got released first. The other example that comes to mind is OSI vs DoD(TCP/IP).

betaby• 2 years ago

OSI is very widely used, most of the large ISPs use it. End consumers are just unaware of that fact. See https://en.wikipedia.org/wiki/IS-IS

tialaramex• 2 years ago

I think this is a stretch, like saying the X.500 Directory System is widely used based on the fact that PKIX is technically adapting X.509 and thus your TLS certificates depend on the X.500 directory system. End users aren't just "unaware" that it's actually the X.500 system, it functionally isn't the X.500 system, PKIX mandates an "alternate" scheme for the Internet and the directory called for by X.500 has never actually existed.

Likewise then, IS-IS is the protocol that OSI standardized, but we're not using as part of an OSI system.

Hikikomori4• 2 years ago

Would call it stretch to say it was widely used by ISPs. Some old ones may still be using integrated IS-IS as their IGP (early OSPF had scaling issues and complicated solutions for that), but that's nothing like widely using the ISO stack. But they might have used IS-IS it to route NSAP in their network at some point in time to manage ATM-era equipment, the ISP I worked still had some ctunnels for that purpose, I doubt they still have them.

rjsw2• 2 years ago

OSI was usable as a wide area network before TCP/IP was.

fanf2• 2 years ago

[citation needed]

aylons• 2 years ago

It's also a counterexample to the bureaucracy of standards bodies --- the standard that actually became widely used was the one that got released first.

Sounds like a cautionary tale: whatever gets released first will stick. If you make a blunder, generations will have to live with it (like IPv4).

dale_glass2• 2 years ago

I wish we could have another and bump the packet size.

We're at the point where we can have millions of packets per second going through a network interface, and it starts to get very silly.

It's at the point where even a 10G connection requires some thought to actually perform properly. I've managed to get bottlenecked on high end hardware requiring a whole detour into SR-IOV just to get back to decent speeds.

supahfly_remix• 2 years ago

I wish we could have another and bump the packet size.

The clock precision (100s of ppm) of the NIC oscillators on either side of a network connection gives a physical upper limit on the Ethernet packet size. The space between the packets lets the slower side "catch up". See https://en.wikipedia.org/wiki/Interpacket_gap for more info.

We could use more precise oscillators to have longer packets but at a more expensive cost.

monocasa• 2 years ago

You don't need that as much on modern protocols. The point of 8b/10b or 64b/66b is that it guarantees enough edges to allow receivers to be self clocking from the incoming bits being more or less thrown directly into a PLL.

zamadatix4• 2 years ago

Realistically I think we would be fine to make packet size significantly larger than Ethernet would currently allow if we really wanted. E.g. Infiniband already has 1x lane speeds of 200 Gbps without relying on any interpacket gap for clock sync at all. Ethernet, on the other hand, has been consistently increasing speed while decreasing the number of bits used for the interpacket gap since it's less and less relevant for clocking. Put a few bytes back and you could probably do enormous sizes.

tlb• 2 years ago

I don't get how that limits the packet size. If a sender's clock is 500 ppm faster than an intermediate node's, you need 500 ppm of slack. That could be short packets with a short gap, or large packets with a large gap.

Ethernet specs the IPG as a fixed number of bits, but it could easily be proportional to the size of the previous packet.

api• 2 years ago

MTUs are one of the eternal gremlins of networking, and any choice of MTU will almost certainly be either too large for the present day or too small for the future. 1500 was chosen back when computers ran at dozens of megahertz and it was actually kind of large at the time.

Changing the MTU is awful because parameters like MTU get baked into hardware in the form of buffer sizes limited by actual RAM limits. Like everything else on a network once the network is deployed changing it is very hard because you have to change everything along the path. Networks are limited by the lowest common denominator.

This kind of thing is one of the downsides of packet switching networks like IP. The OSI folks envisioned a network that presented a higher level interface where you'd open channels or send messages and the underlying network would handle all the details for you. This would be more like the classic analog phone network where you make a phone call and a channel is opened and all details are invisible.

It's very loosely analogous to CISC vs RISC where the OSI approach is more akin to CISC. In networking RISC won out for numerous reasons, but its simplicity causes a lot of deep platform details to leak into upper application layers. High-level applications should arguably not even have to think about things like MTU, but they do.

When higher level applications have to think about things like NAT and stateful firewall traversal, IPv4 vs IPv6, port remapping, etc. is where it gets very ugly.

The downside of the OSI approach is that innovation would require the cooperation of telecoms. Every type of connection, etc., would be a product offered by the underlying network. It would also give telecoms a ton of power to nickel and dime, censor, conduct surveillance, etc. and would make anonymity and privacy very hard. It would be a much more managed Internet as opposed to the packet switching Wild West that we got.

gizmo686• 2 years ago

Most high level applications do not deal with any of the low level details. They open a channel by specifying a hostname and a port number, and are given a reliable bidirectional byte stream from the network layer.

As far as most applications are concerned, the hostname is just a string that is interperated by the network layer. Be that through a DNS lookup, or parsing as an address native to the underlying network protocol.

A minority of applications get fancy and request a datagram oriented link, which the network layer also provides (with an admittadly small limit of 65KB that leaks over from that.

Few applications ever go deeper than that.

leoc• 2 years ago

(Intentional) jumbo frames at layer 2 and expanded MTUs at layer 3 are certainly available (as you may know). In fact it seems (I am, it should be obvious, not an expert) that using jumbo frames is more or less the common practice by now. There does in fact seem to have been some standards drama about this, too: I can't find it now, but IIRC in the '00s someone's proposal to extend the header protocols to allow the header to indicate a frame size of over 1500 bytes was rejected, and nothing seems to have been done since. At the moment it seems that the best way to indicate max. Ethernet frame sizes of over 1500 is an optional field in LLDP(!) https://www.juniper.net/documentation/us/en/software/junos/u... and the fall-back is sending successively larger pings and seeing when the network breaks(!) https://docs.oracle.com/cd/E36784_01/html/E36815/gmzds.html .

scifi• 2 years ago

I'm not certain what my point is, but I wanted to mention that jumbo frames don't work over the Internet. More of a LAN thing.

kstrauser2• 2 years ago

The common advice I've heard for jumbo frames is not to enable them unless you can do it for every devices on your LAN, and even then it's probably not worthwhile outside specific situations like a separate iSCSI network or such.

I just now ran iperf3 from my Mac to my Synology without jumbo frames:

  [ ID] Interval           Transfer     Bitrate         Retr
  [  7]   0.00-10.00  sec  10.0 GBytes  8.61 Gbits/sec    0             sender
  [  7]   0.00-10.00  sec  10.0 GBytes  8.60 Gbits/sec                  receiver

Given how rarely I actually care to saturate the 10Gbit link, I'd rather use the slightly hypothetically slower default settings that are highly likely to work in all scenarios.

myrandomcomment2• 2 years ago

The use of jumbo frames is NOT normal and is only used in very specific setups. In general something like a storage system that is isolated to its own layer2 network. At some point your jumbo network has to hit the reset of the network and packet fragmentation is done in software via the CPU which is very expensive and not linerate. The normal outcome is you break your network.

toast04• 2 years ago

I mean, larger packets (and working path MTU detection) could be useful, but with large (1500) byte packets and reasonable hardware, I never had trouble pushing 10G from the network side. Add TLS and some other processing, and older hardware wouldn't keep up, but not because of packetization. Small packets is also a different story.

All my hardware at the time was xeon 2690, v1-4. Nics were Intel x520/x540 or similar (whatever SuperMicro was using back then). IIRC, v1 could do 10G easy without TLS, 8-9G with TLS, v3 improved AES acceleration and we could push 2x10G. When I turned off NIC packetization acceleration, I didn't notice much change in CPU or throughput, but if packetization was a bottleneck it should have been significant.

At home, with similar age desktop processors with @ dual core pentium g3470 (haswell, same gen as a 2690v3), I can't quite hit 10g in iperf, but it's closeish, another two cores would probably do it.

In some cases, you can get some big gains in efficiency by lining up the user space cpu with the kernel cpu that handles the rx/tx queues that the NIC hashes the connection to, though.

dale_glass2• 2 years ago

I discovered that putting a 10G interface into a bridge implies a very significant slowdown. Linux has to do stuff on the CPU to do the bridging, so that turns off a large part of the card's acceleration.

That's not a good thing for a server that runs a bunch of VMs.

Fortunately SR-IOV exists, but it seems a tad silly to me that I have to do all this weird PCIe passthrough stuff just for this. It's nice, don't get me wrong, but a bit too exotic for what should be a simple setup.

immibis• 2 years ago

Maximum packet size is already configurable on most NICs. 9000 is a typical non-default limit. If you increase the limit, you must do so on all devices on the network. https://en.wikipedia.org/wiki/Jumbo_frame

imoverclocked• 2 years ago

I’ve seen Cisco gear that supports 8192… that was fun to figure out with a separate network team :) “yup, we’ve enabled jumbo frames!”

throwup238• 2 years ago

I wish we could have another and bump the packet size.

That's why I'm in full support of a world ending apocalypse that allows society to restart from scratch. We've made so many bad decisions this time around, with packet sizes being some of the worst.

amelius• 2 years ago

Maybe we can then also redefine pi as 2*pi, while we're at it.

thfuran2• 2 years ago

As long as we also make sure electrons are positively charged this time.

mannyv• 2 years ago

One thing that isn't mentioned is that the physical layer at the time was 'flat' ie: a network had a shared wire. That means bus arbitration (to prevent collisions) was a big deal. Token ring solved that by passing tokens, which presumably guarantees latency. I believe Ethernet just raised a line high, and it was up to everyone to respect that.

Of course that changed when switches came out. I have a 10/100 hub in a closet somewhere for debugging, since it's nice to not have to remember how to get into a switch and set the monitor port.

Token ring equivalents are still used in lots of places. From what I remember cable modem data is basically token ring off of channel 0 (though that may not be accurate anymore).

akira25012• 2 years ago

I believe Ethernet just raised a line high, and it was up to everyone to respect that.

It's actually much simpler than that. When you transmit you also listen. If what you hear is not what you sent, there is a collision, and you backoff.

jhayward• 2 years ago

To be specific, it's even more basic than that. For 10Base-5 and all the coax ethernets, it was "if there's more energy on the wire than you are transmitting, a collision is present".

FuriouslyAdrift2• 2 years ago

Exactly. CSMA/CD (carrier sense multiple access with collision detection)

londons_explore2• 2 years ago

I wish layer 2 and layer 3 were 'refactored' to force all links to be point to point, which they effectively are in the modern world. When was the last time you saw ethernet frame collisions because you used a hub not a switch?

We'd get rid of the idea of a broadcast domain. We'd get rid of Mac address and ARP. Switches and routers would become the same device. We'd just use ip addresses for routing, and the 'next hop' would always be the opposite end of the link you sent a packet over.

The world would be a simpler place, and no functionality would have been lost.

FuriouslyAdrift2• 2 years ago

All wifi is a giant collision domain. Also, each segment of a wired network is a collision domain.

What you are describing is more in line with MPLS or Infiniband.

I agree with you frustration. I prefer to design networks that start routing right at the access port or even using an agent, virtual network port, or VPN endpoint at the client or application (like QUIC), but that is very expensive from a resource standpoint.

IPv6 is also another way to get closer to what you are describing.

In my perfect world, we'd move to something like a mashup of MPLS and HIP (https://en.wikipedia.org/wiki/Host_Identity_Protocol)

If want to study something more "routed" and more point to point, look at private mobile networks (5G).

What we don't want is more layers of abstraction... that's making every slow, brittle and impossible to troubleshoot.

fsckboy• 2 years ago

Also, each segment of a wired network is a collision domain

huh? where "segment" means where you are using a hub not a switch? cuz that was a long time ago

adrian_b• 2 years ago

WiFi uses a different protocol than classic Ethernet, with "Collision Avoidance" instead of "Collision Detection". The reason is that one WiFi station cannot know what sources of radio interference exist at the other stations of a network, because it may hear only a part of them at its location.

So all remnants of the original Ethernet could be removed from wired Ethernet, which does not need layer 2 protocols, while keeping adequate layer 2 protocols for wireless communications. Besides WiFi, there are also long-range point-to-point wireless links, where directive antennas are used at both ends. For these, there is no difference from wired links, so they do not need layer 2 protocols.

addaon2• 2 years ago

When was the last time you saw ethernet frame collisions because you used a hub not a switch?

10base-T1S is just beginning to ramp up in the automotive industry, which modifies the super-successful 100base-T1 to be cheaper by (a) allowing cheaper PHYs; (b) allowing cheaper endpoints due to the lower data rate to handle; (c) allowing lower-spec single twisted pair wiring; and ... (d) allowing multi-drop. This is intended to allow Ethernet to push down into the space that CAN-FD is currently occupying, and looks likely to succeed, at least in some niches.

londons_explore2• 2 years ago

10base-T1S

I think that standard is a huge mistake... 10Mbits isn't enough for a modern vehicle (no cameras, radars, screens etc). Many sensors alone can push megabits, and in the modern world engineers want to send their data json formatted not with bitfields.

Instead they should have used an cdma-like design with the physical being a 2 cent microcontroller for things like bulbs and micro switches. Then, for things like cameras which require more megabits use a 30 cent microcontroller with a higher chip rate, all transmitting in the same bus and using code division to avoid needing to worry about scheduling.

jpm_sd• 2 years ago

I think this is a terrible idea. Daisy chain multi drop is so last century. Switched point to point networks are so much better!

kjs32• 2 years ago

So it's like RS-485, but more complicated?

ninkendo3• 2 years ago

A really interesting article covering this: “The world in which IPv6 was a good design” https://apenwarr.ca/log/20170810

It talks about how when IPv6 was being designed, they wanted to do exactly that: drop most of the layer 2 stuff, abandon the idea of a bus network, make everything point-to-point, all switches would be L3 routers, etc.

Search for “What if instead the world worked like this?” for the relevant part.

My question though, is how would IP assignment work for each of the intermediary devices between me and (say) my ISP’s gateway? My computer is plugged into a switch right now, which is plugged into another switch, which is plugged into my router, which has a point-to-point link to the ISP gateway. Would my router get a /64, then delegate a /68 to the next “router” (ie. The physical thing I currently call a switch), which would delegate a /72 to the next one, etc? How would it determine the optimal IP allocation? What if there’s a cycle? Aren’t we sorta reinventing spanning tree at this point? (I’m genuinely curious about this, because I don’t really grok all of the implications of an “everything is L3” world like this.)

p_l6• 2 years ago

For v6-specific world, scoped addresses and scoped multicast are explicitly for that purpose. You do not need to hierarchically subnet each following router, you just need to be able to express "next hop" for the subnets you need to route towards.

You use link-local autoconfiguration, and use appropriately-scoped multicast addresses to ask "all-nodes" or "all-routers", making autoconfiguration a breeze compared to v4 world. In v4 world a similar setup is also possible, though specific details of the setup differ, and you have to setup addresses manually for each p2p link.

thaumasiotes• 2 years ago

Would my router get a /64, then delegate a /68 to the next “router” (ie. The physical thing I currently call a switch)

This is another weird thing about networking. As far as I've been able to learn, a "router" is a device with two ports that handles transmission of data between those ports, whereas a "switch" is a device with more than two ports that handles transmission of data between those ports.

But nobody would ever care about that distinction.

toast04• 2 years ago

If you force all links to be conceptually point to point, you probably make it harder to do some things. Already 1G and higher force full duplex, and 100base-TX full duplex is very common. I've still got a couple 10baseT half duplex devices though.

I have redundant internet/nat routers at home (overkill!), and they communicate amongst each other to decide which is active and which isn't, but either way, the active one ARPs for the router address with 00-00-5E-00-01-01 as the mac address. The rest of the network just sends off-network packets to 00-00-5E-00-01-01, and failover happens because switches figure out what port is currently using that address.

I share a different mac address for the upstream connection, which is PPPoE (sadly), but same deal --- when failover happens, the new computer starts using the address and everything figures it out, because stations are allowed to move to different ports by design.

zamadatix4• 2 years ago

You can pretty much 1:1 what you describe in the redundancy case with IP, just replace the "relearn which port the MAC address associated with that IP is on" with "relearn which port the next hop for that IP is on".

Things tend to get a little messier than people expect in figuring out the "what values do I use for the point to point links and how do they get assigned" step of things, though there are some clever answers there too.

db48x• 2 years ago

Actually, I know of one large system that heavily relies on having racks and racks of servers all located in the same broadcast domain. It makes the networking a bit more complicated, but in turn the software is a lot less complicated. It’s a decent trade–off.

MisterTea• 2 years ago

That sounds a lot like ATM where you called a machine and received a point to point pipe. Though you had to call first unlike Ethernet where you fling packets into the ether at will. ATM over SONET is used heavily in teleco but is on its way out in favor of OTN and Ethernet.

myrandomcomment2• 2 years ago

Too expensive to do in an ASIC. There is a reason the MAC table is bigger than the routing table on a chip, because it is cheaper. Think of an ASIC as a box that can be divided up in to smaller boxes that are an index. The total number of boxes is limited by the size of the ASIC. The bigger chip, the more box and greater the cost. To do MAC forwarding it takes 2 boxes. To do routing it takes 5. To do an ACL match it takes 14. This is the reason OpenFlow never really worked on switches at scale. What you are asking for is someone simpler to MPLS and that is an expensive feature die size wise. I have highly oversimplified this post, but it is mostly correct at a 1000ft level.

miohtama• 2 years ago

The story is a bit like the Xkcd classic

https://xkcd.com/927/

Looks like it took some years for one standard to prevail. Also TCP/IP was not clear winner in the early days.

Ethernet History Deepdive – Why Do We Have Different Frame Types?

Related Stories