5.5 mm in 1.25 nanoseconds

https://randomascii.wordpress.com/2022/01/12/5-5-mm-in-1-25-nanoseconds/

by asicsp • 5 years ago

359 120 5 years ago

randomascii.wordpress.com

I thought the later 360 CPUs were curiously designed. They had the 360 GPU integrated into the die, and had circuits that purposefully slowed things down to emulate separate chips.

https://en.wikipedia.org/wiki/Xenon_(processor)

fredoralive• 5 years ago

Presumably to ensure behaviour as near to identical as possible with earlier multi chip systems. Because they're fixed targets developers can end up coding right to the limits of console hardware, and end up with lots of reliance on exact timings etc.

Xbox 360 was getting into the era where consoles had OSes and things should be abstracted through DirectX but I guess Microsoft decided the safest thing to do was create a clone with artificial slowdowns rather than risk games pushing the limit breaking.

Nowadays consoles can change CPU / GPU microarchitectures seemingly without problems...

Ciantic• 5 years ago

I got a copy of the detailed descriptions of the Xbox 360 CPU and I read it through multiple times

This way of learning is not taught often enough, we learn it in school when learning another language (e.g. repeating words over and over). But in sciences and engineering the repeated reading of the material is not usually encouraged or talked about.

It would be interesting to see what the material was, it's not linked, but it must have been really detailed.

bostonsre3• 5 years ago

I think it depends alot on your rate of digestion (or understanding? can't find the right word...) of a piece of material. Being dropped in an alien book can make it tough to grasp everything on first pass. If you already have a lot of knowledge that links to the topics being discussed it is a lot easier to grok it in one pass.

Tepix• 5 years ago

I used to read a lot of books when learning about computer science topics. I'm sure i've read the Camel book at least 4 times over several years. A well written book like that can really do much more than just provide a reference.

A lot of documentation these days is mostly references that you look up when you need the nitty gritty details and the bigger picture is not documented as well.

123pie1233• 5 years ago

this is my standard way of learning stuff,

I don't repeatedly read books just to memorise or remember stuff, but I repeatedly read books because you learn and fully digest certain concepts which are then needed before moving on to other concepts

JasonFruit• 5 years ago

This is exactly it for me: on your first reading, there are underlying concepts you kind of get, but don't fully understand until you grasp what's built on top of them. Then, with that fuller grasp, you can go back and increase your understanding of the underlying material, which lets you understand the superstructure better — and the effect persists for several iterations if the topic is involved enough.

Short version: don't be too concerned if you're not completely confident on the details the first time through, because it will make a lot more sense next time around.

ehnto5• 5 years ago

I think rote memorization is less useful in engineering than critical thinking/problem solving, that is probably why. It's not really important you know X value of Y part by heart, you need to know how to get X value of any possible part you encounter.

To learn a language, you can't problem solve your way to the solution, you either know it or you don't, and when you use it in practice (in a conversation) you can't look up what you need to know on the spot.

im3w1l• 5 years ago

Memorizing facts is necessary to make them accessible to your critical thinking and problem solving though.

Like it's much easier to find a connection between two things you know about compared to two things you could look up.

wpietri3• 5 years ago

We're not talking about rote memorization here. We're talking about deeply studying something.

In the mid-90s I set up my then-employer's first WAN. Cisco 2501 with T1s for in-city links and frame relay for inter-city/international. As part of our Cisco contract, we got a full manual set, which was maybe 24" of shelf space. In my spare time over a couple of weeks, I sat down and read the whole thing. Then I went back and actually studied the parts that were relevant, mostly by reading and re-reading until it all made sense to me.

On top of that, I added a lot of practical experimentation with the gear. But foundational for me was really understanding the perspective of the people who made it.

kryogen1c• 5 years ago

I think rote memorization is less useful in engineering than critical thinking/problem solving

i think this is exactly the right take in entirely the wrong direction.

proficiency and mastery are about committing more and more complex things into sub- or un- concious action; muscle memory as it were. grandmaster chess players dont think about solving puzzles that stump novices, they absorb whole board states and many/all permutations instantly. BJJ black belts dont have to think about arm bars, they anticipate complex possibilities of moves and countermoves that a novice simply cannot fathom. this is as true in engineering as any other field. a simple litmus test: is there a difference between a college grad and a 40-years experienced engineer? of course there is

I think GPs point is exactly right. there is a certain lost magic to simply smashing your brain into something and absorbing as much as possible. big eyes, big ears, ego to attempt but humble to fail. one might call such a person a hacker.

edit: see The Art of Learning, Waitzkin

https://www.amazon.com/gp/aw/d/0743277465/

Cthulhu_2• 5 years ago

It depends, as always. A CPU is a whole unit, where (I think) it's worth knowing all the details. A programming language's standard library would be more about having a high level overview of what is available, and not a rote reading of all APIs.

...Unless you're into that kind of thing, and I do believe it will help "pick the right tool for the job", instead of writing your own or picking a random library off the internet. Maybe I should make a habit of it, I'm quite invested in Go at the moment.

On the other hand, probably a bigger factor is 'arsedness'. I don't have the attention span or interest to read something back-to-front, or maybe not anymore? I don't know for sure. But there's not many things that grip me enough to really draw me in.

...except writing shitpost comments on HN.

erosenbe0• 5 years ago

Yes, I teach my students to always at least look at the overview and see what is available and the taxonomy or ontology!

Don't treat all sources as a mere dictionary of self-evident structure. Quitting just because the blurb of immediate interest has been rendered leaves out key opportunities to learn what is available.

foobarian• 5 years ago

Specifically for CPU manuals, detailed reading used to pay off especially back when every cycle counted. You would spend days rereading tucked away sections describing some weird extension or edge case hoping to harness it for your game/program and get more speed out of it.

scoutt• 5 years ago

repeated reading of the material

I taught myself almost everything I know, and repeated reading is the method I usually use. I don't memorize, but instead I attack any material to learn by "reading sweeps" (this only applies to topics I know nothing about).

I first read all the material and, even if I don't understand a word about anything, I keep going until the end. Then I read everything again from start to end and here I can definitely say that I'm starting to understand. Then again, and this time I allow myself to go back and clarify something I didn't understand. When I feel confident, I try to implement something with the reading material at hand.

I can't recall any material that I recently learnt that way but "The definitive guide to the ARM Cortex M3" by Joseph Yiu is a book I remember (back in 2007 when I started with Cortex microcontrollers).

tomcam7• 5 years ago

Impressive. I’m just now learning how to learn that way, and I’m at retirement age. My way is much less efficient, because I stop and research each thing I don’t understand until I feel comfortable moving on. Highly inefficient, I now believe. My kid does what you do and learns way faster than I ever did.

billti• 5 years ago

That the approach I've settled on that works best for me. I used to get frustrated that there wasn't "one source" of information that could take me on the journey from novice to expert, and that there was "so much to choose from" on most topics.

Now I happily browse through as many sources as I can find, noting which I think are high quality even if above my head, and absorbing what I can for my current knowledge level. Then I'll revisit the good stuff and scan it to pick up the next layer deep, and repeat over select material until I feel I have what I need.

The other thing I do is make flashcards as I go (I use an app called Quizlet, but I know there are lots out there). Especially with a new subject, just basic terms can be overwhelming at first. I capture want I feel is important in a question & answer format, and just fire it up when I have to a few mins to run through some flashcards (e.g. waiting for a meeting, sitting on the lav, etc.). It seems to work really well for me.

brucedawson5• 5 years ago

The material was all proprietary/NDA and I don't think it's ever been shared publicly, sorry.

The main point I wanted to make with how I became the CPU expert was not actually the "reading multiple times" in order to deeply understand it (although that was important) but rather the fact that I chose to become the CPU expert rather than having it assigned to me.

It was one of many times that I became the acknowledged expert of some area, and perhaps therefore in charge of that area, just by working hard in that area until I was "obviously" the expert.

zeusk• 5 years ago

Yeah, I definitely echo this.

I work on Display team in Windows and in past worked mostly on IddCx and SoftGpu but recently went through some VESA docs and chimed in my opinion during some meetings. Now I'm leading DisplayPort reliability efforts with graphics IHVs

Tuna-Fish• 5 years ago

Somewhat similar is Grace Hopper explaining the nanosecond: https://www.youtube.com/watch?v=9eyFDBPk4Yw

discreditable• 5 years ago

I am blown away by the wafer of Xbox 360 CPUs hanging on this guy's wall. That was a kingly gift!

phkahler• 5 years ago

> I am blown away by the wafer of Xbox 360 CPUs hanging on this guy's wall. That was a kingly gift!

Yeah, the only place I've seen a full wafer on display is a display case in the lobby of a chip company office building. I wonder how often some process goes out of whack and they get an entire known-bad wafer that still looks great?

londons_explore• 5 years ago

You can buy complete wafers on ebay for a few bucks, so I'd guess it's pretty common.

I think many of the alignment steps during manufacturing involve making a wafer, seeing how misaligned it is using an electron microscope, adjusting all the machines, then making another wafer. Repeat until the wafers start working.

bostonsre3• 5 years ago

Anyone know how much that would cost roughly? Wondering if they had a big reject bin that needed to be disposed of.

andreareina• 5 years ago

Guessing that the cost for a wafer at a state-of-the-art process has gone up over the years but not that much, in the vicinity of ~$10,000?

https://twitter.com/chiakokhua/status/1306437988801486848 sourced from

https://cset.georgetown.edu/wp-content/uploads/AI-Chips%E2%8... p45

rasz• 5 years ago

https://twitter.com/Siliconinsid/status/1480216470374207500

"Wafer of the Sony PlayStation 1 CPU. While decaping a Sony CXD8606Q, I was very lucky to be able to match the die to an unidentified wafer I had in my collection. The 6“ wafer could accommodate a total of 200 individual dies."

marcan_428• 5 years ago

The speed of light story is a fun conjecture, but given the discrepancy (30cm vs 5.5mm), I don't think that's it. Light travels slower in wires but not that much slower, and increased design clocks don't explain the discrepancy either.

I think it's more likely that this is an artifact of the interconnect design. For example, the interconnect may be designed as a tree, where the path from core 0 to the L2 only visits a branch but the path to core1/2 has to traverse the root of the tree. If each stage adds a clock cycle, there's your 4 clock cycles: one to go to the root, one back down to the other side, and then the whole thing in the other direction for the reply from L2 with the data.

If the interconnect had been a ring bus instead, you'd find that the latency from cores 0 and 2 is the same, but higher for core 1 - assuming no other endpoints in the ring, but that's obviously not the case, since I/O needs to be on the bus too. It's possible he got the core numbers wrong in the die shot, and core 0 is the bottom left core. Then if the ring goes clockwise L2 - core 0 - core 1 - core 2 - I/O on the top right - L2, that's how you end up with core 0 being closer to the L2 in terms of clock cycles than the others, and the observation would be consistent with this design too.

123pie1233• 5 years ago

this is an interesting video (to me anyway!!) on the speed of electrons vs fields

The Big Misconception About Electricity.. https://www.youtube.com/watch?v=bHIhgxav9LY

Dumble• 5 years ago

https://www.youtube.com/watch?v=2Vrhk5OjBP8

jaywalk• 5 years ago

That video is hilariously wrong, which seems to be a common theme for Veritasium lately. I can't tell if he legitimately believes what he's saying, or if he produces these seemingly-correct-but-not videos to drive up engagement and views.

krupan2• 5 years ago

ElectroBOOM replied to this: https://youtu.be/iph500cPK28

AshamedCaptain3• 5 years ago

The speed of light is a useless metric when measuring propagation delay of a signal in a wire since there is load of factors that limit it much more significantly, such as the width of the conductor / capacitance / resistance, the strength of the driver, etc. Starting point: https://en.wikipedia.org/wiki/Elmore_delay

And if you are going to do anything useful (e.g. toggling the signal), the max frequency at which you can do it is even lower than the limit set by the propagation delay.

In any case, even TFA says the reason pretty explicitly: it's actually 4 pipelined stages, thus 4 clock cycles delay. Probably it was way too much power+area to drive the distance in one cycle.

entangledqubit• 5 years ago

Random historical context (IIRC), in the early 2000s (around when the chip discussed was made) chip designers were dealing with the transition from gate/switch delays being a dominant factor to signal wire propagation delays becoming very significant. This was a side effect of the improving processes making smaller chip features.

extropy• 5 years ago

Interesting. From what I could gather, latency in ICs is dominated by the time it takes for a voltage to build up at the sink, which is dominated by the resistances and capacitances along the way, makes sense.

krupan2• 5 years ago

In chips because of the layers of conductors and insulators you are actually charging a capacitor, not just sending a signal down a lone wire.

AshamedCaptain3• 5 years ago

That's how all conductors work, not just "in chips due to layers of conductors and insulators". The thing is that in most situations you just model the conductor away as a "perfect ideal wire" and everything usually just works since you don't care about these small delays and/or you have very high power. But when you enter sub-nanosecond and/or sub-mW territory...

kens2• 5 years ago

Yes, a long bus is like a resistor-capacitor delay circuit. You can speed it up by using a bigger driver for more current, but that takes up more space and adds its own delay. I expect that the bus in the Xbox is split in two, with a second driver in the middle. This makes the first half faster, but the second half of the bus probably takes another clock cycle, which would explain the observed behavior.

brucedawson5• 5 years ago

Reading from L2 requires sending down a request and then waiting for the response. This makes me realize that the actual speed is 5.5 mm in 0.625 ns because it's a two cycle delay in each direction. I'll update the article.

But, this request/response means that a ring bus seems unlikely because a unidirectional ring bus should make the sum of the request/response times identical for all CPUs, if I'm understanding correctly.

kens2• 5 years ago

You mentioned a document with a detailed description of the internals of the Xbox 360 CPU. I don't suppose that document is available to the public?

marcan_428• 5 years ago

I was thinking bidirectional ring bus in my model; indeed if it were unidirectional you'd end up with the same result for all CPUs.

FWIW, the Cell used a bidirectional ring bus network, and it's of the same generation and happens to share technology heritage with the XCPU... (very similar PowerPC core) :-)

BeeOnRope• 5 years ago

Speed of signal propagation though wires on modern chips is much slower than the speed of light. A bit less so on an old Xbox chips, but still a big difference.

On a modern core the delay is enough that you can't come close to reaching from side one side of a core (not to mention CPU) to another in a single cycle.

dlemire• 5 years ago

Do you have a measure of this speed?

xyzzy21• 5 years ago

So first, there are two types of velocity when it comes to signals on a transmission line which is what metal lines on chip are: phase velocity and group velocity.

Phase velocity is close to the speed of light. It can be reduced by dielectric materials parameters so it's often lower than c.

https://en.wikipedia.org/wiki/Phase_velocity

Group velocity is often much slower. Information transfer can never be faster the group velocity, rather than the phase velocity. So when it comes to "on chip communication" it is group velocity that matters. Group velocity depends on many factor of transmission line design such as line thickness, line width, dielectric thickness, etc.

https://en.wikipedia.org/wiki/Group_velocity

cmpaul2• 5 years ago

Technical aspects aside (and congrats on the achievement!), I love the implications of the author's "Aside":

1. An otherwise inconvenient situation (snowpocalypse) was turned into a learning opportunity. I would've probably turned it into a Netflix binge.

2. People took the time to detail the CPU design in a document. I'm always looking for examples showing the benefits of documenting design (I'm an EM) and it's rare to find one that changes someone's life trajectory. :)

tomcam7• 5 years ago

Agreed. I’m a pathetically slow learner so situations like #1 were always an essential part of how I got ahead of the curve. #2 is very cool also... but what is an EM?

cshokie• 5 years ago

Based on the context most likely “Engineering Manager.

brucedawson5• 5 years ago

2004 was a bit early for a Netflix binge. Plus, the power was out, so there goes that plan.

The documentation that I read showed the pipelines in great deal, but they were badly presented such that the flows of data and time were obscured. My main contribution was to straighten out and simplify the diagrams, and then animate them. Well, and then explain them. It was definitely a CPU that required that level of detail.

tom-thistime4• 5 years ago

Sounds like mostly gate delay? I'll share my ignorance. For metal traces on silicon the physical "speed of light in this material" propagation delay should only be around 9-10 ns/m. Call it 10 ps/mm, about 55ps in 5.5mm. Or else if the trace has high enough resistance (maybe not metal), another thing to worry about is charging the line capacitance. That is longer for longer lines, but it is a different kind of delay from speed-of-light.

nashashmi• 5 years ago

I have seen these hang on someone's wall. Because they made the ATI cpu. I thought the whole thing was one big cpu. I was not impressed.

Now I am! 15 years late.

mattnewport• 5 years ago

I was working on the central technology team at EA during the run up to the launch of the Xbox 360 and some of us got early access to help prepare for the new architecture.

I was blown away by the amount of technical detail we were getting access to and how clearly explained it was. Much of that was thanks to Bruce Dawson. I remember devouring everything I could find that he'd written and it was a fascinating crash course in CPU architecture for a relatively young software engineer at the time with little hardware background. Thanks Bruce!

mring33621• 5 years ago

All I want to say is that the XBOX 360 was one of the best consoles EVER! I still happily play games on mine.

ridaj• 5 years ago

https://en.wikipedia.org/wiki/Signal_velocity - my understanding is that the speed is driven by interactions between the trace and its surrounding material, which creates capacitance and inductance along the path

jmrm2• 5 years ago

At that distance, depending of the frequency of those buses, it could have some distortion and some latency due to being a transmission line. If this happened, the team that made that CPU would add some kind buffers at the receiving side to get the data at a correct time, and that could be most of the induced lag.

Adding that to the fact that the three cores have a shared common bus, and there are gone to be some kind of signalling about who is using it (that we don't know), this could be reasonable.

The Xbox 360 CPUs works at 3.2 GHz, that's 0.3125 ns per cycle, and four cycles make those 1.25 ns. I won't be surprised if those who made the CPU designed to have 4 cycles of delay to cores 1 and 2 to be sure there isn't any kind of glitch reading the data from the port.

SeasonalEnnui• 5 years ago

You meant 0.3125ns per cycle, the rest of your comment triggered a thought:

It's common in FPGA/microcontroller designs to have 'input synchronizers' on inputs from long digital lines (on any input where you can't guarantee synchronicity) to prevent metastability problems, especially when crossing clock domains. I think that's likely the case here, there are both long lines and different clock domains.

jmrm2• 5 years ago

Oops! Totally true: 0.3125 ns per cycle

tomcam7• 5 years ago

The post itself was a pleasure to read. The comments are a mini education on how to learn. I strongly recommend reading through both!

rdlw• 5 years ago

This post should be retitled to "11 mm in 1.25 nanoseconds", the blog post has been renamed to that as per this update:

"...So the 5.5 mm actually only takes 0.625 nanoseconds each way, the signal actually goes 11 mm in 1.25 nanoseconds, so the original title of this blog post is a lie. Sorry!"

rq1• 5 years ago

Isn’t the discrepancy due to https://en.m.wikipedia.org/wiki/Space-filling_curve ? (Aka Peano curves)

5.5 mm in 1.25 nanoseconds

Related Stories