Reboot Your Dreamliner Every 248 Days to Avoid Integer Overflow (2015)

https://www.i-programmer.info/news/149-security/8548-reboot-your-dreamliner-every-248-days-to-avoid-integer-overflow.html

by pjmlp • 8 years ago

356 280 8 years ago

i-programmer.info

jaclaz3• 8 years ago

Provided that the guess is correct:

A simple guess suggests the the problem is a signed 32-bit overflow as 2^31 is the number of seconds in 248 days multiplied by 100, i.e. a counter in hundredths of a second.

The question that comes to my (perverted) mind is what is the counter for, or more strictly why it needs an accuracy of 1/100th second?

If it is related to a "periodical" action (a time interval) it makes little sense to have that degree of precision, and on the other hand, if the precision is needed, why not calculate it from a base point?

I mean, if it is related to "boot" time, I presume that noone would ever use a counter, rather a log of some kind with a timestamp and calculate (properly) the time elapsed ...

masklinn• 8 years ago

Linux's "jiffy" defaulted to 100Hz in 2.4 and older, and to 250Hz since 2.6.13.

adrianN2• 8 years ago

I'd assume the software is not event driven but cyclical. In each cycle you gather inputs, compute, produce outputs. You need a cycle counter because event validity and response time limits are likely defined in terms of cycles (e.g. peripheral sensor A has to report at least once every three cycles or sound alarm). Bad things happen when the cycle counter overflows.

na852• 8 years ago

At the very least, the HUMS and CV/FDR systems need sub-second precision for maintenance and crash investigation reasons.

As for why they don't calculate it in a particular way, probably just an amateur mistake.

Lots of aerospace-related applications are moving towards the methodology where you cook up a model in matlab and then hit the AutoCode button. Things like this are the reason why.

gargravarr• 8 years ago

The systems are controlling generators, which produce AC current. It's not unreasonable to think the controllers are monitoring the AC waveform, which has a frequency of 50-60Hz. Ergo, if the controller is checking the generator is producing the expected waveform (because a lot of sensitive things depend on the waveform being accurate), it makes sense that it could be using a 100Hz polling interval, which is where the counter comes into play.

It's possible that it's using a system-wide timer for convenience since the embedded hardware is very limited compared to a full computer, where spinning off a separate timer is trivial. When the requirement for different timers hit the program designer's desk, they probably took the most precise use case and designed one timer around it, and neglected to take into account the overflow.

MrBuddyCasino• 8 years ago

Aircrafts operate at 400Hz to reduce the size and weight of transformers, motors and power supplies. Its less efficient than 60Hz, but planes aren't that long anyway.

jaclaz3• 8 years ago

Yep, but purely anecdotal data, I once was involved in the construction of a (small) hydroelectric plant (I was responsible for the construction/building, not for the electrical parts, our construction company had a couple electric partners for that).

For some reasons the tender was for the building and plant, but the client made a separate tender for the actual turbines.

Though I tried (vainly) to convince the manufacturer of the turbines to use a controller from the same firm we had as partner, they decided to use "their" way.

In theory not a problem as the turbines and their controllers would "talk" to the external control system (SCADA, etc.) through a "standard" RS422 connection.

After a couple days of testing, it was clear that the SCADA was going periodically "beserk".

Though not at all my field/responsability, I was willing to have the problem solved and after 3 (three) days of the engineers from the turbine manufacturer and from the control system manufacturer finding nothing (and BTW largely failing to communicate between them) I started looking with them, one by one, at the signals that were exchanged on the connection.

It was evident that there was some form of overflow, the on-turbine controller sent way too many data and "clogged" the receiving part.

There was a sensor for rotation speed and another one for oil pressure that were polled (and sent data) at a rate of 1000 per second.

The turbines were of the Pelton type, with an external (large) flywheel, and it took (literally) tens of seconds since you stopped the waterflow to have the actual wheel slow down and after several more tens of seconds stop, and viceversa once it got to the intended speed, it had in practice no variation.

So you had these two sensors polled 1000 times a second to measure something that would change - maybe - only after several seconds intervals and that could be as well corrected with a delay of tens of seconds.

Reducing the polling rate from 1000 to 10 times a second (still way overkill for the proper functioning of the system) all errors went away.

It came out that these sensors were a new type/make/model, the first to be capable of Khz polling, and the turbine engineers set them to the max only because they could.

howard941• 8 years ago

Anectdotal like my brother, an interactive inverter product I developed sampled the 60hz voltage and current at 19.2khz to more accurately determine RMS voltage and current in the face of some really nasty distortion. It was probably overkill but another guy on the team who worked on revenue-grade watthour meters thought it was just fine.

clarkmoody• 8 years ago

If the clock is being used for real-time dynamic model propagation, the 100 Hz speed is enough the capture the physical phenomena in the system, such as control system response. 10 Hz is a bit too slow, and 1 kHz is overkill.

During my graduate work in aerospace engineering, 100 Hz was pretty much standard for sensor filtering and simulation work.

ocfnash• 8 years ago

The article finishes with the rather vague and somewhat terrifying assertion that:

"It is estimated that the Airbus A380, comparable in complexity to the Dreamliner, has more than 100 million lines of code."

moreira2• 8 years ago

The thing is, do they just count all the lines of code in all the libraries they use, as well as the OS (all of it compiled from source, I would guess)? I imagine they do, since that's the only reasonable way that 100 million lines of code could be reached.

sbradford2613• 8 years ago

So on a project like this, you do not use many standard libraries. It is usually explicitly stated that you can't use anything from the std c library and such.

Also source lines of code are usually calculated as lines of code that need to be verified. So if a line of code is in the software, it needs to have a test that covers that line of code.

mehrdadn• 8 years ago

Why wouldn't they count everything? Does a line of code stop being a line of code if it's written by someone other than you?

logfromblammo2• 8 years ago

Interesting that they have so many, when we frequently compare the use of lines of code as a software metric to kilograms of aluminum in an aircraft.

ainar-g3• 8 years ago

IIRC, 64 bits give you 200 years when you're counting nanoseconds. Why on Earth would they use a 32-bit integer? I doubt this was some kind of microoptimisation. My bet is on some sort of legacy component that is 64-bit-o-phobic.

DocTomoe• 8 years ago

Of course, in security-relevant areas, prooven legacy code always trumps the new code. Noone wants to lose an airliner because Phil right out of a CS course positively needed a ruby interpreter in the main flight computer...

To understand this mindset, have a look at http://www.stroustrup.com/JSF-AV-rules.pdf

sbradford2613• 8 years ago

My favorite recommendation to people. Take a C or C++ program they have written and compile it with full MISRA. Definitely makes things more interesting.

danbruc2• 8 years ago

200 years sounded low to me and so I did the math and it is indeed pretty spot on. 63 bits, i.e. signed integers, will last 292.27 years and you obviously get twice that if you are not interested in negative numbers.

ainar-g3• 8 years ago

Right, thanks. My estimate was closer to 200 because I was thinking of Go's time.Time.UnixNano method[1], which returns undefined results when used before the year 1678 or after the year 2262. But yeah, if you count since the system restart, it's bigger.

[1] https://tip.golang.org/pkg/time/#Time.UnixNano

ajross2• 8 years ago

Why on Earth would they use a 32-bit integer?

Why stop there? Why on earth would anyone use an integer instead of a double, given the inherent risk of truncation error? Or for different arguments: Why on earth would anyone use a 16 bit wchar_t? Why on earth would anyone make char unsigned (or signed)? Why on earth would anyone put the little end first?

Machines are machines. They have fixed representations for different types, with tradeoffs. And you have to pick one. And the thing about timeout handling specifically is that everyone along the path from the timer driver up through the app needs to agree on the precision needed, or you'll get an overflow condition.

Arguments of the form "Bugs are bad and we shouldn't write them" have not historically helped with improving software quality.

_verandaguy• 8 years ago

I think the comment you're replying to makes a solid point, still.

If this bug were to appear every 200 years, that's substantially longer than the lifespan of any single airframe currently in existence (and nearly twice as long as the existence of powered flight) -- and if a Dreamliner were to actually survive that long (most likely as a historical artifact doing heritage flights, maybe), doing this kind of reset would just be part of the routine of getting the "old frame" up and running, not unlike timing certain steps of a WWII fighter plane's manually rather than having them done automatically.

Without saying that increasing the width of that variable is the _optimal_ solution, in this case deferring the error leads to safe and predictable operation over the airframe's nominal lifespan.

For what it's worth, quadrupling the variable's width to a int128_t would mean you can store over 10^22 years at nanosecond granularity, effectively future-proofing this bug out of existence, and I'm reasonably sure a modern system should be able to spare 3 extra bytes.

titzer2• 8 years ago

"Well, I figured....that bridge is gonna fall down in 200 years no matter what, so I went ahead and used rubber bands to hold the suspension lines in place. Only cost me $2!"

No, just no. Don't be a hilljack. Build something to spec. 32 bits is clearly not to spec.

cpeterso• 8 years ago

Why on earth would anyone use a 16 bit wchar_t?

Fun fact: the C standard does not specify the width of wchar_t. While MSVC uses 16-bit wchar_t (UTF-16), gcc (even MinGW on Windows) uses a 32-bit wchar_t.

CountSessine• 8 years ago

They have fixed representations for different types, with tradeoffs. And you have to pick one. And the thing about timeout handling specifically is that everyone along the path from the timer driver up through the app needs to agree on the precision needed, or you'll get an overflow condition

I agree, but the difference between 32-bit counters and 64-bit counters as a precision refinement is very special. Upgrading from 32-bit counters to 64-bit counters, even when counting nanoseconds, lifts you out of human timescales. No electronic system has ever or will ever need to maintain a microsecond count for 600000 years, for example.

Dylan168076• 8 years ago

Why on earth would anyone use an integer instead of a double, given the inherent risk of truncation error?

Depends on what you're doing.

Why on earth would anyone use a 16 bit wchar_t?

That's a very good question! Either use something that can hold at least 21 bits, or use UTF-8.

Why on earth would anyone make char unsigned (or signed)? Why on earth would anyone put the little end first?

These have nothing to do with space, what weird analogy are you trying to make?

They have fixed representations for different types, with tradeoffs. And you have to pick one.

For most purposes, there is zero benefit for going too small. Pick a number that can't break under the use it's getting. If a fixed-size number can't do that, change your algorithm.

And the thing about timeout handling specifically is that everyone along the path from the timer driver up through the app needs to agree on the precision needed, or you'll get an overflow condition.

Making sure your data types are compatible should be one of the easiest pieces of analysis you're applying to your safety-important code.

Arguments of the form "Bugs are bad and we shouldn't write them"

Yeah, this isn't that at all. This is "think about the limits of your data types as you pick them". A computer being left on for a year should be an expected use.

kilo_bravo_3• 8 years ago

I was in training brushing up on my embedded programming skills, which thankfully I don't have to use anymore, and the instructor told a story about a bug in a missile that was on the verge of delaying its deployment due to tests failing.

While on the bench and being subjected to a bunch of tests to validate the seeker components against simulated targets and countermeasures, the flight control surfaces would start spazzing out after a few minutes in the test jig.

The problem had something to do with IR (or something) tracking and attitude control, with the tracker rolling over the odometer and sending spurious data to the flight guidance system which caused the tests to fail and would have lead to an in-flight breakup of the missile real-world.

The onboard hardware was highly resource constrained, and engineers and developers worked for weeks trying to fix the problem, going as far as contemplating a complete redesign of the seeker system.

Then somebody pointed out that between the time of launch and the missile's two end states: impact with the target or fuel exhaustion-- was less than 60 or so seconds.

The only reason the bug was causing problems was that tests were being run back-to-back-to-back in order to speed up things, and the seeker subsystem was being powered on for way longer than it ever would be in the real world.

A missile shouldn't have to be on for more than a couple of minutes, and a Dreamliner shouldn't have to be on for more than 248.55 days so I'm willing to bet they stuck some ancient 8-bit micro in there, reused ancient code that was already tested, and called it a day.

Nimelrian• 8 years ago

a story about a bug in a missile that was on the verge of delaying its deployment due to tests failing.

A Patriot Missile in Saudi Arabia wasn't launched to intercept a SCUD, which then killed 28 soldiers. The reason was poor handling of rounding errors [1]

The maiden flight of the Ariane 5 rocket resulted in a RUD due to an integer overflow [2]

[1] http://www-users.math.umn.edu/~arnold/disasters/patriot.html

[2] http://www-users.math.umn.edu/~arnold/disasters/ariane.html

Symmetry2• 8 years ago

Well, the sort of engineer working on this might very well have gotten their start in environments where they were restricted to 5k of RAM. In that case it's easy to paint yourself into a corner by choosing integer representations that are larger than you need.

nofunsir• 8 years ago

64-bit-o-phobic... GreenHills INT178, until recently.

stephengillie2• 8 years ago

When a city bus needs to reboot, the driver will park the bus first. How do they reboot a plane in the air?

abstractbeliefs• 8 years ago

They don't, they reboot it on the ground. The implication is that airlines need to find at least one hour in each 248 days to service this bug.

This shouldn't be a problem, but in the highly optimised logistics of air travel, finding a solid hour to just reboot the GCU is expensive.

stephengillie2• 8 years ago

Having seen a similar bug on humble Netscalers, and found a much shorter reboot quite difficult to schedule, I do believe the operational challenge here.

ubernostrum2• 8 years ago

Even the "minimal" A-check process happens often enough to avoid a 248-day limit, and typically grounds an aircraft overnight.

People really do not realize how much maintenance commercial aviation involves.

taneq• 8 years ago

Despite the name, there are occasions when an airbus is actually a groundbus. :)

lucb1e4• 8 years ago

I know it's a joke but the mental picture of an airplane with its wings removed showing up at a bus stop, dropping in for a broken bus, is very amusing. Would probably be national news and good PR if they actually pull this off, might be doable with a short enough plane. One can dream...

okket• 8 years ago

Doesn't routine checks include complete reboots? They should happen more often than every 248 days.

ginko• 8 years ago

Yes, having to do a power cycle along with your routine checks seems like a non-issue.

raverbashing2• 8 years ago

There are several types of routine checks, I think Boeing calls them A/B/C/D (with subtypes, depending) Airbus has a different nomenclature but the idea is the same.

Not sure when a reboot might be done. Maybe on a B or a C check, a C is every 18mo apparently https://www.airliners.net/forum/viewtopic.php?t=753393

tonysdg• 8 years ago

You would certainly hope; and yet, long uptimes unforeseen by designers can have devestating consequences:

A government investigation revealed that the failed intercept at Dhahran had been caused by a software error in the system's handling of timestamps. The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters...the Scud impacted on a makeshift barracks in an Al Khobar warehouse, killing 28 soldiers.
As a stopgap measure, the Israelis had recommended rebooting the system's computers regularly.

https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha...

oftenwrong• 8 years ago

more detail on the patriot missile failure:

https://web.archive.org/web/20100702180720/http://mate.uprh....

saint_abroad2• 8 years ago

Obligatory xkcd: https://xkcd.com/1495/

"Why everything I have is broken"

na852• 8 years ago

Looks like we killed their server. Anyone have a cached copy?

bcaa7f3a8bbc• 8 years ago

Here we go.

https://archive.fo/dLsMd

raverbashing2• 8 years ago

Funny how they didn't catch that in the apparently deep code reviews and complicated processes they have in the aerospace industry.

sbradford2613• 8 years ago

"One interesting fact is that the FAA claim that it will take about one hour to reboot the GCUs - so there clearly isn't a reset button."

I am incredibly surprised by this, most higher level flight control systems have power up requirements in the seconds. Then lower level actuator controls or engine controls have power up requirements in the milliseconds.

cm21872• 8 years ago

What do they do if there is a temporary power loss in mid air?

sbradford2613• 8 years ago

So that depends on if you mean a power source temporarily goes down or if a box experiences a power blip. So in the first case each box usually has 3 different sources of power, something like the generators on the engines, then backup on the 24 volt bus, then finally a backup battery.

In the second scenario it depends on how long the blip is. Usually there are holdup requirements that a box will not reset if power is lost for x amount of time. If power is lost for longer it will save state and leave itself in a state that will come back up quickly.

What I am thinking is happening in this case is that some value is getting messed up in NVM and it must be reset by the maintenance crew, so the "reset" they are talking about isn't just rebooting after the error occurs. But if you reboot before the error happens the NVM doesn't get messed up and the value is just updated with the correct number.

ethbro• 8 years ago

https://en.m.wikipedia.org/wiki/Ram_air_turbine

andyjohnson02• 8 years ago

Anecdote:

A few years back I was travelling on a Virgin Pendolino train (an Alstom Class 390, probably) in the UK that was having a few problems. Speed changes were causing a lot of juddering, and as I remember the interior lighting and air conditioning was being unreliable. After limping on for a while, the train stopped at a small non-scheduled station and the crew announced that we'd be stopped there for a while, and not to be alarmed if the lights went off (it was night). This duly happened, the ventilation flans stopped, the interior doors all moved to the open position, seat booking indicators turned off, etc. After a pause that was probably a minute or two, everything came back to life and we resumed the journey.

Basically, they rebooted the train. It was interesting to realise that such a large, complex, fast-moving machine could be reset like that, with several hundred people embedded inside it.

tialaramex• 8 years ago

On trains with the original version of the UK's TPWS protection system you have to reboot them if somebody mistakenly leaves them on top of the TPWS transmitter when shutting down, for example stopping slightly short of the usual position in a terminus station. In this state when the train boots up TPWS is in an error condition, so you have to override that, move it a few metres, then reboot it to get rid of the error. Newer models recover without being rebooted.

Failing to reboot caused at least one accident because the TPWS was just left overridden and so wasn't protecting anything.

Since a series of UK rail disasters involving trains whose safety systems were switched off, running in passenger service without the safeties is prohibited.

odorousrex• 8 years ago

The smallest unit of time the FAA uses in their requirement documents is "one man-hour".

It could take 5 minutes or less to boot, but the FAA will still say it takes an hour.

tim333• 8 years ago

This report actually sounds a little worse

the FAA's new rules require operators to reboot the plane's electrical system every now and then because "all three flight control modules on the 787 might simultaneously reset if continuously powered on for 22 days." The effect of this simultaneous reset "could result in flight control surfaces not moving in response to flight crew inputs for a short time and consequent temporary loss of controllability."

Hope they've fixed that one now - also from 2016 https://www.popularmechanics.com/flight/airlines/a24151/boin...

kokey• 8 years ago

I remember an issue with the Oracle DB client libraries for Linux (RHEL 2 or 3 I think) where if the system had something like a 160 day uptime any software using the library to connect to the DB would just hang. I saw the effects with strace where it would just get stuck in a loop doing some time related syscall. We managed to get an unofficial patch from Oracle that I could wedge into our servers, since reboots were under strict change control. I remember this issue came up when I was being interviewed for a position at the Guardian a couple of months later and they seemed amused by what I had to do to fix it since they also encountered the same issue but fixed it by making sure they reboot more than twice a year.

andyjohnson02• 8 years ago

The space shuttle had a related problem, although not caused by overflow. Some parts of the STS's avionics used a clock that would reset to zero at 00:00:00 on 1st January, while other components had clocks that would continue to count up. If a shuttle mission spanned the new year boundary then systems would panic if they could no longer agree on the time.

The only reference I could find to this is https://abcnews.go.com/Technology/story?id=2699091&page=1

nimbius2• 8 years ago

I read the article from tip to tail and as a professional engine mechanic, It reminds me of Fords "myford touch" platform.

the system was rolled out across newer fleets without much testing, and in some models controls practically every single feature in the vehicle from climate control to the onstar SOS. There was a recall for platinum model F150 trucks because the system could glitch out after so many hours of continuous operation and trigger a fault in the 4 wheel brake force distribution system. This in turn either completely arrested the brakes, or caused them to quietly apply themselves at around 15%...you couldnt undo this unless you pulled the fuse. Even worse, collision detection would be disabled because the system thought you were aware of a potential crash and were braking.

If a certain bluetooth phone were paired, it could cause the trailer load and position sensor to erroneously predict grade downshifts. The result was that incoming calls on the highway would either wreck the rear differential or put the truck on the side of the road.

pjmlp2• 8 years ago

On the latest ADB Podcast, they mention that certain car infotaiment systems just go "blue screen" if you feed them a more recent Bluetooth version than what they have hardcoded.

http://androidbackstage.blogspot.com/2018/08/episode-97-blue...

colejohnson66• 8 years ago

Why is the infotainment system even on the same computer as the brakes!?

Sileni• 8 years ago

The trend towards monolithic hub systems is baffling. Didn't we learn those lessons while we were struggling with mainframes and timesharing? You don't lock critical systems, ever. You don't even directly interact with them, you send them messages and if they don't receive them they should continue to function as the operator would expect. The collision avoidance system failing to function correctly is no where near as dangerous as arbitrarily applying the brakes.

java-man• 8 years ago

incompetence.

let me explain. attaching a non-essential sub-system with a large attack surface to a mission-critical network is not malicious (I hope) it is incompetence. I would accept a unidirectional serial link with a validated protocol as a sole connection between the two.

I think it is the same kind of thinking that goes into other bad ideas: flickering stop lights, red turn signals. The auto makers just don't really care or think. This is something I don't understand.

chooseaname2• 8 years ago

Either the engineers thought it was a good idea or the engineers could not convince management that it was a bad idea.

Someone12344• 8 years ago

It is cheaper.

dtech• 8 years ago

Modern electronic cars use a single bus (CAN bus) that connect all electrical systems. The brakes, the engine, the wipers, the stereo...

Luckily it is a very well-tested system, it was one the first commercial use cases for QuickCheck [1], but still...

One of the bugs that John Hughes found was an integer overflow that transformed a lowest-priority message into a highest-priority message. [1] https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quviq...

logfromblammo2• 8 years ago

Because car companies have always built cars, not secure networks. Those executives have never been to DefCon; they have no idea what level of risk they face from the unsecured electronics components in their cars. They are not just unaware that putting critical systems and non-critical systems on the same bus is a bad idea, but they are also completely ignorant of the difference between real-time systems and the operating systems on their phones and computers.

They need to hire some folks with aviation backgrounds, who can explain to them why the plane does not fall out of the sky when the in-flight entertainment system chokes on a scratched DVD. Even when they get it wrong, it is still less wrong than the auto-makers.

[Edit:] Aviation people are the only outsiders they're likely to listen to.

They're not perfect. But they are better. They at least have an awareness that security is an issue, even if the ways they handle it are... well... let's just say they're not ideal.

Even though the tech folks who frequent this site are knowledgeable, the people who build stuff don't always respect our expertise. Sometimes they don't even realize that our expertise might apply to their problems. This is how we get black-box electronic voting machines and CANBUS2 and wi-fi light bulbs or security cameras that inadvertently open a back door into your LAN.

jgrip• 8 years ago

In most cars this is not the case. For example, in my 2005 Opel Astra (Saturn Astra for US) there are three buses. High-speed CAN used for critical systems, Low-speed single wire CAN used for other vehicle systems and a mid-speed CAN for the entertainment and climate control stuff. All traffic between these buses are "firewalled" by CAN bridges that should only forward relevant frames between the nets.

VW for example have a firewall in front of the OBD connector, only allowing traffic for the diagnostic addresses to pass.

However, I expect that in newer "cloudy" cars, they need so much data that these "firewalls" have become very permissive. Remote start via Apps, triggering signal horns from the Internet, OnStar telemetry reporting etc.

Traditionally the car makers have been completely terrible at tech security but they are slowly improving on this front. In fairness they've also been to some degree hampered on this front by regulations protecting local small garages, stating that the diag stuff cannot be locked down too hard.

LeifCarrotson• 8 years ago

The other comments have effectively lambasted the manufacturers for their incompotence and foolishness in putting these on the same system, so I'll skip that bit.

The practical reason is that the big central touchscreen is a convenient place to put stuff. On an old truck, you selected 4WD by moving a big transfer case lever, which indicated you were in 4WD by its position. No computer to get in the way. Slightly newer trucks have physical buttons with lights that connect to electrical solenoids, often directly. Tractors have brake pedals for each wheel, but old trucks don't have a similar braking force distribution system - you got locking diffs or dragged all the brakes and hoped it helped. With a touchscreen and a computer in place, it's easy to make compelling sales pitches for putting complex features there instead of on expensive, complicated physical controls.

Complicated on a per-unit basis, that is - I understand that a computer has far more moving parts than any plastic-and-wires assembly, but when it works it's easier to draw buttons and graphics than to make mechanical actuators.

Whether or not the processing happens on the same CPU or communication happens on the same physical bus doesn't matter that much when you have the ability to make selections that affect the engine and drivetrain on the main display.

macintux• 8 years ago

John Hughes likes to talk about the CAN bus when discussing Quickcheck and test generation.

Learning about the crazy complexity and potential for unforeseen interactions made me very unhappy when security researchers remotely controlled a car on the highway as a publicity stunt. There’s simply no way they can know with absolute certainty what might happen.

ryanmarsh• 8 years ago

That’s horrifying. Are there any comprehensive write ups on this you can share? I’d like to read more about this.

vvanders• 8 years ago

That's pretty spectacular, I'd love to read some more sources on it.

Makes me glad Tesla did their stuff the way they did. You can reboot the cenelter console and/or days while driving and not much happens other than A/C turning off for about 20s.

antongribok• 8 years ago

Tesla really scares me.

They released an OTA update to the brakes with less than 2 weeks from first reports to code being pushed to the entire fleet of cars on public roads.

Someone12344• 8 years ago

Tesla pushed out a software update to AutoPilot (Lane Centering) that caused a fatality[0], I hardly think they should be held up as an example of good software hygiene.

[0] https://www.mercurynews.com/2018/03/29/tesla-crash-victims-f...

amluto• 8 years ago

I’m very glad that Tesla’s instrument cluster is semi-independent. Because mine crashes every few minutes while driving due to a “high priority” firmware regression.

Someone12344• 8 years ago

Chrysler has had similar issue[0] this last year, their Pacifica Minivan had a software crash that would cause the engine management computer to crash and the vehicle to completely lose power while operating (inc. at highway speed). In effect you'd have complete loss of electrical power and even power steering, brake assist, and hazard warning lights(!) would be lost.

Took them six or more months to fix it as they weren't able to reproduce the software glitch.

[0] https://www.nytimes.com/2018/01/12/business/chrysler-pacific...

DoofusOfDeath• 8 years ago

I've noticed the occasional posting in HN from people who aren't [aspiring] professional software developers, which surprises me.

If you don't mind me asking, what brings someone in your line of work to a site like this?

[edit: I just realized that ^ sounds like a tragically bad pickup line.]

soared• 8 years ago

~30% of submissions are software-dev-only (new typescript 3.0 release), but most submissions are just high quality, interesting posts. And the level of discussion is excellent - no memes, no jokes. Its like the opposite of reddit.

zitterbewegung2• 8 years ago

HN is not a monoculture the appropriate stuff on the site can interested anyone and no one is required to be a software developer to get on this website.

nimbius2• 8 years ago

our heavy diesel emissions performance machine runs Linux, and after a vendor training on how to use it I became totally fascinated with it. A lot of Linux professionals like HN a lot, and the articles are a good break from the norm on lunch break.

Im learning python, too! :)

seem_2211• 8 years ago

The comments are informative, there's no politics, it's more interesting than the news. I'm a sales recruiter, so I don't read the in the weeds software articles, but there are still 2-3 good articles on the front page at any one time.

okmokmz• 8 years ago

A lot of the content is still relevant for those in other IT roles, or even just hobbyists. I'm a security engineer rather than a dev but I still find many of the posts on the site useful and/or interesting

glup2• 8 years ago

There's also a ton of academic lurkers -- I've always though that this due to the facts that 1) the border between academia and industry is pretty porous in technical fields (super obvious in ML research, but I think holds more generally) 2) academics borrow---and contribute to---the same set of methods and technologies 3) most individuals have an interest in complex systems (and their hilarious, tragic faults) such that they find articles like this one interesting.

I think it's also useful to look at preceding communities like Slashdot — in that case it started out pretty tightly scoped for Linux enthusiasts, gaming, programming, and the then-nascent social internet but then the cohort of regular readers and contributors got older and they found themselves in policy, academia, and managerial business roles; as long as people remained involved in the community, the content of the site expanded to reflect the (extremely rapidly increasing) role of tech in these domains. (Obviously, Slashdot suffered from several ownership changes, a lame commenting system, and most everyone moved on to other venues).

cli• 8 years ago

I've been reading this site since 2009 and can barely write anything more complex than Hello World. The guidelines on submissions say nothing about software engineering, and the definition of hackers as written by Paul Graham does not limit it to software engineers.

engi_nerd• 8 years ago

I've been lurking here for 10 years. For most of that time I was employed as an electrical engineer.

Why should people who are interested in high quality technical discussion surprise you with their presence?

"Hacker" used to mean a lot more than "software developer".

tejtm• 8 years ago

you had me at "bad pickup" :)

analog31• 8 years ago

I came across HN by accident, and what keeps me coming back is the interesting, informative links & threads, and the quality of discussion. Threads about workplace issues are of interest to me, as are general scientific and social topics.

I'm a "scientific" programmer, meaning that I use programming as a problem solving tool, but I don't write commercial software for a living. I turned away from that path in the early 80s.

ggcdn• 8 years ago

Big factor for me is the higher-quality discussion compared to say, Reddit.

MiscIdeaMaker99• 8 years ago

Not everyone in the software world is a developer.

saudioger• 8 years ago

Uhh that's kind of terrifying. I'm due for a new car after 10 years, and I think I might specifically try to avoid the systems that centralize things like breaks with entertainment systems...

lanius• 8 years ago

Yikes. I'm not a prepper, but there is something to be said for older vehicles without this unnecessary computer nonsense.

thrillgore• 8 years ago

Why aren't these systems just plain air gapped?

jacobush• 8 years ago

This is just ... outlandish. You couldn't make this shit up with a straight face.

samstave• 8 years ago

Question from the ignorant:

What % of a 24-hour day is any given commercial airliner in service, vs. sitting completely turned off in a "parking" spot?

sitkack5• 8 years ago

Most modern things don't ever "turn off".

kleff• 8 years ago

Wait, so they don't turn off the planes when not in use?

mnw21cam• 8 years ago

This kind of plane isn't "not in use" very often. In fact, any time it's not flying people around is costing the airline loads, and they try to minimise that.

ceejayoz• 8 years ago

Plus, when it's not flying passengers, it's being fueled, cleaned, maintained, and inspected. All of those are easier if there's power and systems online.

mack1001• 8 years ago

Doesn’t this call for an OSS vehicle platform that can be reviewed by thousands of contributors to avoid these type of issues?

mannykannot• 8 years ago

I see that the "many eyes" hypothesis lives on, despite the accumulation of contrary evidence.

imtringued• 8 years ago

Well usually the difference between open source and proprietary is that the proprietary vendors are slow to fix security issues because it doesn't increase their profit. In some cases they even try to hide the fact that the software is insecure or even sue the person who reported the vulnerability. Meanwhile most OSS software gets fixed as soon as the vulnerability is found.

tdubhro1• 8 years ago

Unfortunately, evidence [1] suggests that the rate of bug detection by reviewers does not scale linearly with the number of reviewers, adding more than 4 reviewers uncovers bugs at a much lower rate [2].

However I absolutely agree that better / more extensive / diligent code reviews are part of the solution to improve code quality and eliminate these kinds of defects. It's tough to create the right kind of incentive structures for reviewers internally at a company; maybe the future will have specialist firms that provide review-as-a-service for a fee, or perhaps firms could trade review-hours (all under strict NDAs I'd imagine).

[1] Glass: https://books.google.gr/books?id=3Ntz-UJzZN0C&pg=PA174&redir... [2] https://en.wikipedia.org/wiki/Linus%27s_Law

paultopia• 8 years ago

OMG, that's terrifying. United Airlines flies a lot of those. United doesn't, as far as I can tell, do maintenance until something breaks.

ubernostrum2• 8 years ago

United doesn't, as far as I can tell, do maintenance until something breaks.

In order to be allowed to fly in the US, and other countries, you better believe United does maintenance.

This is one of my favorite reddit comments ever, and goes over aircraft maintenance from a car analogy, explaining what you'd have to do to a car in order to meet the mandatory maintenance standards imposed on airlines as a matter of law:

https://www.reddit.com/r/aviation/comments/3v1hzj/how_do_old...

jaclaz3• 8 years ago

JFYI, I often use as a reference the actual VW Owner's manual for the Beetle, you can find a copy of the 1952 version here, relevant pages are 7 and 8 "Operating Instructions":

https://www.thesamba.com/vw/archives/manuals/1_52bug.php

https://www.thesamba.com/vw/archives/manuals/1_52bug/3.jpg

https://www.thesamba.com/vw/archives/manuals/1_52bug/4.jpg

but at least up to the '70's the recommendations were more or less the same.

sseth• 8 years ago

Thanks for the great reddit link. It is the best way to explain this!

slavak• 8 years ago

I very much doubt that is true. They wouldn't exist as a commercial carrier for long if they did this.

SmellyGeekBoy2• 8 years ago

People keep using this word "terrifying" despite air travel being the statistically safest form of transport by orders of magnitude...

marijn• 8 years ago

Your options are to increase the number of bits used, which puts off the overflow, or you could work with infinite precision arithmetic, which would slowly use up the available memory and finally bring the system down.

Yeah, no. Doubling the amount of bits to 64 while keeping the same precision gets you about 3 billion years worth of time, which is probably enough. And I'm going to leave calculating how much time it'd take to fill up any reasonable amount of memory with a single arbitrary-precision integer as an exercise to the reader.

ainar-g3• 8 years ago

Even if you do use arbitrary precision arithmetic and count nanoseconds, the heat death of the universe is more likely to occur before your number takes 1KiB of RAM.

gvx• 8 years ago

If my calculations are correct, 520 bits should be enough to count in units of Planck time until the heath death of the universe.

bunderbunder2• 8 years ago

Probably the deeper problem with using arbitrary precision arithmetic is that you end up with a variable-sized datatype, which I believe means at least a modicum of extra hassle & complexity for any language that the control software is likely to be written in. And less predictable timing, which might be a big no-no if this is something that needs to be used in timing-sensitive places.

I'd much rather take the 64-bit int, myself.

manicdee• 8 years ago

Are you assuming an update to a number will use the same memory as the original number, and that the original memory will be released?

shrimp_emoji• 8 years ago

Always buy in to 64. What was our ancestors' problem?[0]

0: https://en.wikipedia.org/wiki/Year_2038_problem

codeulike• 8 years ago

Storage space was the problem, used to be expensive.

daemin2• 8 years ago

I don't think this cases needed to use 64bits for the clock. Because realistically I would hope that the plane would be serviced more than every 248 days. Therefore each service the computer could be reset or go into a self-check which would reset the computer and therefore prevent the overflow.

beached_whale• 8 years ago

At 4 billion increments per second, it takes over 70 years to overflow a int64

Symmetry2• 8 years ago

64 bits is certainly the solution I would go for as well. Arbitrary precision numbers are going to involve heap allocation which is something I would normally like to avoid entirely in a system like this.

joe_the_user• 8 years ago

The thing about the error is that it's very standard sort of error but an error which apparently formal methods didn't catch.

I've just been looking up TLA+ (model verification language) and I'm not sure how it would deal with avoiding overflow; it's approach is proving that a system will always be in a required state but avoiding overflow in most practical approaches involves effectively staying far enough away from the overflow condition that it never appears (as the other comments here go into). Done right, overflow will never be impossible, just unlikely.

https://en.wikipedia.org/wiki/TLA%2B

chrisacky• 8 years ago

This is a little familiar with the rocket failure at Dhahran[1] of 1991 resulting in 28 deaths. The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.

Two weeks earlier, on February 11, 1991, the Israelis had identified the problem and informed the U.S. Army and the PATRIOT Project Office, the software manufacturer. As a stopgap measure, the Israelis had recommended rebooting the system's computers regularly. The manufacturer supplied updated software to the Army on February 26.

[1] https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha...

mabbo• 8 years ago

While it is a good lesson from a software perspective- your bits are going to overflow, make your software handle that gracefully- I've always been a bit uncomfortable with the blame for deaths being placed on that bug.

The bug didn't kill anyone. The scud missile fired with intent to kill did. And of course, it was fired because it was the Gulf War and countries were attacking each other. Blame who you like for that situation. But all anyone talks about is how it was the Patriot Missile bug that lead to the deaths.

The software bug failed to prevent the deaths that were going to happen anyway. Lessons learned, yes, but I'd hate to imagine a programmer somewhere living in guilt over it.

brianwawok• 8 years ago

The software bug failed to prevent the deaths that were going to happen anyway.

It MAY have failed to prevent the deaths. I don't think Patriot has a 100% success rate. If the normal success rate of that intercept was 1%, 50%, or 99% - it changes the wording a bit I think?

briffle• 8 years ago

I agree. Really there are two alternatives to what happened.

1. The patriot battery does not have the bug, (or was recently rebooted) and now has a BETTER chance of hitting the scud missile. (but not guaranteed) People may or may not die.

2. The patriot battery is turned off, or not there, and nothing stands in the way of the scud missile. People most likely die.

M_Bakhtiari2• 8 years ago

We are issuing this AD to prevent loss of all AC electrical power, which could result in loss of control of the airplane

So the 787 doesn't have a mechanical backup for the flight controls? So much for the Boeing fanboys talking about their mechanical yokes. Through realistically that says very little about actual reliability and safety, just look at the 737 rudder issues.

gsnedders• 8 years ago

Even the 777 has very little in way of mechanical backup (it has a couple of spoilers and the horizontal stabilizer with mechanical linkage).

The main difference is the duplicated controls in the cockpit are mechanically interlinked, so moving one will always move both.

I believe the 787 is Boeing's first (commercial) aircraft with no mechanical backup, but there's not been much for decades.

supernova87a• 8 years ago

When you say "mechanical", in a 787 or any large airliner it doesn't mean straightforwardly what it might mean in a small plane.

In any large airliner the pilot's controls are indirectly linked to the flight surfaces by either hydraulic or electrical systems. There's no way a person could produce the forces needed to handle a plane that large adequately.

So I suppose it depends on the specific system, but if electrical power were to be totally lost, perhaps they would be dead in the water. (air)

As is my rudimentary understanding.

foldr• 8 years ago

https://en.wikipedia.org/wiki/Ram_air_turbine

M_Bakhtiari2• 8 years ago

In any large airliner the pilot's controls are indirectly linked to the flight surfaces by either hydraulic or electrical systems.

I understand the 777 and 787 are special cases, but my impression of the rest of their line was that the flight controls are directly linked by control cables and are assisted by mechanical-hydraulic servos, a bit like hydraulic power steering on cars. So with a total loss of electrical systems they would still have normal primary flight controls.

Of hydraulic servos can fail (for example the aforementioned 737) and even triple hydraulic systems can be taken out by something like the uncontained engine failure of United Airlines Flight 232.

I tend to think that electrical and electronic systems can ultimately be more reliable and maintainable, but I worry that the safety culture is much worse among the designers of such systems than their mechanical equivalents (no wonder, we've been tinkering with the latter for thousands of years and the former only a few decades).

baybal2• 8 years ago

Remember the Ariane rocket - also a piece of verified code running in verified OS, running on verified MCU, made in verified silicon, but the obvious flaw was still missed.

The problem with formal verification that for a complex system, the amount of constraints goes over the roof, and it is no longer possible to humanly understand if one of them makes sense in complex.

It might very well be possible that something "register A should never overflow when input B is below value C" was put in validation rules, but nobody gave importance to understanding what it was. Or worse, somebody was simply lazy to change it, fearing it will upset 20 some even more obscure validation constrains.

lucb1e4• 8 years ago

Site does not load for me. Found this one: http://archive.is/dLsMd

saint_abroad2• 8 years ago

(Signed version of) the 497 day bug [1] strikes yet again.

[1] https://news.ycombinator.com/item?id=3231781

yellowapple• 8 years ago

In this case, does it really matter? I'd be very surprised if it's running nonstop for more than a couple days, let alone almost a year. Or is this some system that normally doesn't power down when the rest of the plane does?

racingmars• 8 years ago

Planes like the 787 are probably very rarely completely powered down. At the gate, they're on ground power and there are probably some power buses left energized even if the plane is going to sit unused until the next day. These generator control units (since the article says they take an hour to boot up...running through a ton of tests, I guess?) are probably energized even when most other aircraft systems are turned off. If the aircraft is being towed to a hangar or away from a gate for parking, they probably want its lights on while in motion on the field so will stay powered from battery or fire up the APU (auxiliary power unit) before disconnecting from ground power to tow it around. When it makes it to the hangar, perhaps it gets plugged back in to ground power. So it's conceivable that this particular component could remain powered up continuously for a looong time.

That said: as others have stated, some forms of maintenance would power down the plane completely, and these likely happen more often than this overflow bug occurs. If I remember from the articles when this bug was first discovered, it was discovered through simulation or specific testing and not because an operator encountered it, which would imply that yes, operators do "reboot" the whole airplane often enough where this doesn't naturally happen.

nashashmi• 8 years ago

My office building has a computer system for elevators that directs the traveler upon selecting the level to an elevator preselected with the destination floor. Every month or so, the system begins to slow down or lag, e.g. user enters floor, system pauses (during morning rush hour this causes delays), flashes to the selected elevator, and returns back to the user screen. The longer the interval since reboot, the longer the pause. Obviously there is some sort of memory corruption in the system causing a buffer overflow and the routine programmed in the software is not clearing it. So a manual reboot is required to make it work as normal.

nradov2• 8 years ago

Those symptoms would be more likely caused by a memory leak and/or fragmentation, not by memory corruption or a buffer overflow.

foofoo55• 8 years ago

Related article from 1986, predicting such bugs, and listing a few:

http://articles.chicagotribune.com/1986-12-14/news/860403047...

08-152• 8 years ago

I wonder why the GCU needs a time counter in the first place and how it is used so that the whole controller shuts down on overflow. I bet, 16 bits would be enough if handled properly.

sbradford2613• 8 years ago

Software with airplanes is usually periodic and not event driven. So every 10 ms or some interval of time it will compute all of the control laws and monitors and such. So that 32 bit counter is most likely the period counter or a maintenance interval counter.

08-152• 8 years ago

Yeah, makes no difference.

I imagine, the software needs to remember some state variables with time stamps, to ramp power up at a certain rate or to implement PID control of the power level. Crucially, it never needs an absolute time stamp, it only needs to know how old the data is.

The most simple(!) approach is to use an unsigned integer as timer/counter and let it overflow all the time. Age is computed by unsigned subtraction, ignoring wraparound. With a 16 bit timer and 10ms resolution, ages up to almost 11 minutes can be represented. Why would a turbo-generator even need to remember what it did 11 minutes ago?

In other words, the mistake is probably not the narrow counter, it's the signed arithmetic and the subsequent failure when it computes a negative age. Someone else said "they probably stuck a crappy old 8-bit micro in there." That would surprise me---people programming 8-bit micros tend to know how to use unsigned arithmetic where it makes sense.

Tempest1981• 8 years ago

This one was 216 days -- which seems stranger than 248.

Crucial m4 -- 5184 Hour Bug https://www.storagereview.com/node/2676

nmg• 8 years ago

Naive question: Would a reasonable way to avoid this scenario be, to increment a secondary counter when the primary counter reaches max-1, then reset the primary counter to zero?

sbradford2613• 8 years ago

A couple different ways to avoid the issue:

1: Size the type used to store the value so that it cannot overflow even in corner cases such as a plane being on for several months. Ex. going from a 32 bit int to a 64 bit.

2: Have a flag that you set when the counter overflows, so you can calculate the actual time. Given the length of time that planes are on this would give you enough time.

3:Lessen the precision of the counter (Probably not an option since the precision is usually a requirement)

ascar2• 8 years ago

This is mathematically no different than using a larger int. Your second counter would store the more significant bits.

lucb1e4• 8 years ago

What if the secondary counter overflows? And should all software now check two counters when they want to know the actual time?

Just use a 64 bit integer, at least for this purpose (since this apparently needs only 10ms precision): that is twice as large as a normal integer, so basically the same as using two counters, except you don't need extra logic in your code.

x0x0• 8 years ago

Prolly process.

IE a global review of every counter in the code, and again for each commit, of the max run time encountered by any code.

Plus also prolly running in sim with eg 1 million x time.

dsfyu404ed• 8 years ago

Do civilian aircraft not have requirements for maximum reboot times?

Naval vessels usually have sub-minute times from power restored to shooting back and they're legacy code and legacy hardware nightmares. A 787 doesn't have layers upon layers of legacy code that most naval applications do. While the stakes are definitely a little lower you'd think they would be able to do a full reboot in less than the time it takes to fall out of the sky by a comfortable margin.

sp527• 8 years ago

While this is pretty sketchy, I DO like the design principle of ephemeral subsystems. It just makes sense to assume a service can go down at any time for any reason, from its conception, and bake that into how it's built. You could argue periodically restarting things is just a natural extension of that.

stcredzero• 8 years ago

A company I worked for licensed a managed language VM, which was being used to operate an airport people mover, which was crashing after just shy of 50 days. (Crashing software-wise, not physically. It would just awkwardly stop.) It turned out to be an integer overflow. If this was a 32 bit register for milliseconds, what would be about right.

leephillips• 8 years ago

"You may be used to rebooting a server every so often to ensure that it doesn't crash because of some resource problem"

Is this something that people have to do? I maintain a few Linux servers, and I never need to reboot them. They only go down when the hosting company needs to do some kind of hardware maintenance.

linker3000• 8 years ago

That was once the reason put on a support ticket for a Windows 2000 Server which I reported as running slowly - something like: "Server had been running for over 30 days - needed reboot". I suspect this was down to the bespoke app, and not Win2K.

tantalor• 8 years ago

will go into failsafe mode and the plane will lose all electrical power

Sounds real "safe".

jjwiseman• 8 years ago

When this became known, I asked on the aviation stackexchange about whether there were any circumstances under which it might be expected for a 787 to remain powered on for 248 days: https://aviation.stackexchange.com/questions/14494/in-what-c...

The answer seems to be that it is very unlikely, but not impossible.

rosege• 8 years ago

Too bad it was seem very strange if you asked the pilots before boarding just how long it had been since they rebooted it.

kwhitefoot• 8 years ago

Your options are to increase the number of bits used, which puts off the overflow,

Just adding one more byte puts the overflow off for almost 200 hundred years, should be long enough.

The question that comes to my mind is: now that they know that Boeing is sloppy are they going to thoroughly audit the code to see if any other overflows are lurking in it? And will they do the same to Airbus? And if not then why not?

plg• 8 years ago

Has this issue been fixed or do they still have to reboot every 248 days?

kbutler2• 8 years ago

On a recent family trip a few miles from our destination, all my vehicle's dash controls went out, then reappeared with a charging system error indicator.

Everything seemed fine - I watched battery gauge and hoped I'd make it. When I got to the destination, I stopped and restarted the engine, and everything looked fine, and the charging system indicator went back to normal.

I noticed afterward that the "Engine Hours", which had been getting close to 10,000, was now in single digits. No other internal counters were reset.

I wondered if it was an overflow condition, but it appears more mundane - many vehicle owners report seemingly random resets. The surprising thing seems to be that it hadn't reset before getting close to 10,000 hours!

kw71• 8 years ago

Wow, that car was driven a lot. All my 20+ year old cars have less than 4000 h.

kbutler2• 8 years ago

Or did their counters reset? ;-)

30 odometer miles per engine hour seems about average (to one significant digit - varies with proportion of highway vs city vs idle hours). That would suggest all your 20+ year old cars are under about 120K miles? Or they have a very high mix of freeway miles. Either way, they're not likely getting the average 10-15k miles per year.

Conversely, that rough evaluation is making me question whether my recollection of nearing 10K hours was correct - the vehicle is under 200K miles, which would suggest <20mph average.

mvpu• 8 years ago

Forgive me if it's a stupid idea - but why not shutdown these plane computers every night when they aren't in service and boot them again in the morning before the first flight? Let all the counters reset, all memory creep and leaks go away, start fresh....

jhpankow• 8 years ago

This will increase the failure rate due to added thermal/electrical stresses caused by the constant power cycling.

sitkack5• 8 years ago

Your options are to increase the number of bits used, which puts off the overflow, or you could work with infinite precision arithmetic, which would slowly use up the available memory and finally bring the system down.

Pedantically true for an infinite universe, but merely moving the counter to 64 bits would give one 2^32 * 248 days, lets say we only use 30 bits, we still on the order of a billion years for 4 extra bytes.

King-Aaron• 8 years ago

Reminds me of "Memory leaks on missiles don't matter"

https://twitter.com/pomeranian99/status/858856994438094848

ayy_lmao• 8 years ago

To be honest this isn't quite a big deal. Why would anyone let an airplane running for days continuously.

Just a quick calculation. If it takes 248 days = 248 * 24 * 3600 seconds to go to 1 to 2^32, then the sensors have a sampling time of 5ms so it takes a measurement at a 200Hz frequency. Not related to anything but it's nice to know I guess.

This is quite old news though. I remember it was mentioned in last year's CS50 lecture about integers.

Reboot Your Dreamliner Every 248 Days to Avoid Integer Overflow (2015)

Related Stories