Microsoft Exchange stops passing mail due to bug on 1/1/22

https://old.reddit.com/r/sysadmin/comments/rt91z6/exchange_2019_antimalware_bad_update/

by technion • 5 years ago

711 356 5 years ago

old.reddit.com

alblue• 5 years ago

TL;DR Microsoft released a virus definition file with a timestamp/serial 220101001, and is unable to parse it into a (signed) long value since the largest value is 2,147,483,647 and it’s possible that there’s a factor of 10 in the deserialisation logic.

ginko• 5 years ago

But 220,101,001 isn't larger than 2,147,483,647.. Did they get the number slightly wrong?

cnorthwood• 5 years ago

It also included a serial number following the date part of 001 (also 002, etc)

Cycl0ps• 5 years ago

There's three 0s before the last 1. That pushes it over the edge of what a long int can handle

Flocular• 5 years ago

The error message " Can’t Convert “2201010001” to long." makes it seem like that was the timestamp/serial used (Would be yymmddHHMM format). No factor 10 in deserialisation needed for an overflow.

f0e4c2f7• 5 years ago

Maybe we should just leave email off for 2022.

skofgar• 5 years ago

This would be a great New Years resolution. No emails.

bpodgursky• 5 years ago

You won't like what comes next.

wrycoder• 5 years ago

Purchase orders via Twitter?

influx• 5 years ago

Much prefer e-mail, which is async over messaging which isn't. Also, my e-mail has programmable filters...

k8sToGo2• 5 years ago

Why is messaging not async? No one is forcing you to reply right away.

AnIdiotOnTheNet• 5 years ago

Honestly I'm getting to the point where I'm beginning to think computing was a bad idea.

formerly_proven• 5 years ago

Sand was never meant to do this. Every day we stray further from the light.

wedn3sday• 5 years ago

I think 90% of society would probably be better without having easy every day access to advanced computers. Let the HPC people keep running their physics and chemistry experiments, but maybe we dont all need to have 10 computers surrounding us 24/7.

speedgoose• 5 years ago

The average quality of life and life expectancy improved a lot thanks to computing. I'm sorry you don't like emails though.

vsareto• 5 years ago

Delete all business applications but keep video games.

ryanelfman• 5 years ago

This resonated so hard for me. Sounds like depression

IntelMiner• 5 years ago

There's a joke I heard a long time ago that I always found amusing, though a lot of my friends I think are sick of

"If you went back 300 years and told people you taught rocks to think by blasting them with lightning, they'd burn you at the stake as a witch"

duxup• 5 years ago

I really like this idea, until I consider all the other options.

thanatos519• 5 years ago

... so if they test things, they don't do it in an environment that resembles the world of their victims^Wcustomers?

Krasnol• 5 years ago

I assumed it's normal now to outsource beta testing to the customer.

hughrr3• 5 years ago

Being a victim is exactly what I felt like when I was an MSFT customer. I shall remember that one :)

fragmede2• 5 years ago

For all the hate the cloud gets because of cost/whatever, I'm really glad to have a SaaS provider for email right about now.

belter2• 5 years ago

Many eons ago...Around 10 to 15 years ago...Seen a similar serialization and date processing bug with a Java based Enterprise Integration Platform used by some of the largest companies in the world.

Several European Central Banks were unable to process transactions ;-). Can't provide more details, to protect the innocent. Since then, it has been a standard within the Testing group of that vendor to always have a running platform setup with a date 6 months ahead ( clock sync from a different source ). Something I added to my own Software Engineering and Testing standard practices.

karmakaze• 5 years ago

I came to ask if anyone was aware of systems that automate tests of dates in the future to find it already answered.

Something I added to my own Software Engineering and Testing standard practices.

Noted.

trembonator• 5 years ago

Cmon, it’s Microsoft! You shouldn’t need to do that!

hughrr3• 5 years ago

Isn’t this basically fuzzing?

Making sure it works at the business domain level is another thing entirely.

game_the0ry2• 5 years ago

A couple of years ago, I did work for a major airline where the test suites that would test date processing would fail on every leap year (Feb 29). So they decided not to run tests or deploy on Feb 29th.

jlkuester7• 5 years ago

Lol, reminds me of of the large EMR platform I used to work on that (until very recently) could not handle the "fall back" DST event (thanks to local date encoding in very parts of the system).... So the recommended 'fix' for hospitals was to just turn it off (their entire electronic health record) for the duration of the DST changeover... I personally believe that a non-trivial amount of people die due to DST and similar time related bugs...

Hamuko2• 5 years ago

Our unit tests break twice a year, starting a couple of days before DST change. I'm sure it could be solved, but since the DST change happens on Sunday and the CI builds fail on Friday, no one has really bothered.

mdavidn2• 5 years ago

One project had tests fail every Saturday and Sunday, near the organization’s fiscal week boundary. Ever since then, I have strived to hard-code all dates in test suites, choosing dates several decades in the future.

treeman79• 5 years ago

We doing a midnight deploy with 40 people. After boss explained process. I asked him, so are we deploying on the first midnight or the second?

Long quiet pause, then a series of curses. So we deployed at 2am instead.

unbanned• 5 years ago

Is it someone's responsibility to periodically use the environment as real users?

Without this, there's no value in running a duplicate environment like this. Automated tests won't capture human interaction perfectly...

rbanffy6• 5 years ago

You can automate at the UI level and run that set of interactions (as well as screen captures and image matching if needed)

nurettin• 5 years ago

Reminds me how Homakov forward-tested github issues by sending a message from 3012 (it worked)

Here's the original link https://github.com/rails/rails/issues/5239

Unfortunately, they fixed the date and his bender account got renamed.

benttoothpaste• 5 years ago

Long should be a 64 bit type on 64 bit machines, no?

stkdump• 5 years ago

No, Win64 is an LLP64 platform.

bithavoc• 5 years ago

Expanding on the topic, https://devblogs.microsoft.com/oldnewthing/20050131-00/?p=36...

breakingcups• 5 years ago

technion3• 5 years ago

As far as I can tell, there's currently no official acknowledgement of this issue.

It has affected 100% of the Exchange servers I'm overseeing. In every case, a complete halt to mail was corrected by a fix only escribed on Reddit and Twitter.

bazzargh• 5 years ago

There is one here https://techcommunity.microsoft.com/t5/exchange-team-blog/em... though you'd be hard pressed to recognize it as being this bug, since it doesn't mention the error message that everyone has seen, and the instructions for workaround are not as explicit and detailed as I'd want if I was paged for this.

YXNjaGVyZWdlbgo3• 5 years ago

That is what happens without QA, people saving timestamps as int.

Forge36• 5 years ago

How often does QA adjust time on their machines? Could be an easy test case/setup step for an automated suite

YXNjaGVyZWdlbgo3• 5 years ago

Since the year 2k craze it's actually a rather standard practice in QA, at least on our team, to increment in decade steps up to 2038.

littlecranky67• 5 years ago

kind of offtopic: I recently learned google uses a "Leap Smear" instead of using leap seconds. To avoid errors and outages by having a minute with 61 seconds, they just speed up their time on a single day to match that extra second eventually.

PascLeRasc• 5 years ago

That’s really interesting, where did you hear that? I’d like to read more if you have anything

mikehollinger• 5 years ago

https://developers.google.com/time/smear :-)

ed25519FUUU2• 5 years ago

I’d you’re using Google’s public timeservers you’re also using leap second smearing.

https://developers.google.com/time/smear

db48x2• 5 years ago

Phone numbers are not numbers. Zip codes are not numbers. Model numbers are not numbers. Characters are not numbers. Which part of this progression escapes you?

Seriously, if you can’t add, subtract, multiply, and divide them, then they aren‘t numbers and you shouldn’t use a numeric data type for them. This is CS 101 knowledge. How badly does MS run their engineering?

hulitu• 5 years ago

Just look at Win 10 and at Teams for examples. The only engineering which counts at MS is the one which maximizes profits.

Demiurge2• 5 years ago

And how is this different for everyone else in the free market economy?

samstave• 5 years ago

Isn't it lovely that Bill Gates' famous words are that vaccines are the best investment he has ever made?

frays3• 5 years ago

The only reason I can think of is YYYYMMDDHHMM represented as an unsigned long is 4 bytes, while the equivalent in characters would require 12 bytes.

Is it possible that at Microsoft's scale this would actually make a significant difference?

pintxo2• 5 years ago

It probably did, 25 years ago

throwuxiytayq• 5 years ago

Not likely. Modern software developers laugh this kind of difference away, and will almost always default to taking a memory/storage space hit over other possible trade-offs (performance, correctness, convenience...)

petee17• 5 years ago

Anything you do will take 3x the memory, so yes... I cant see why anyone would store as a string internally, you have to decode into a number anyway so just extra work. Strings are only useful to humans.

Edit: reading all the comments here, I get the feeling high level programmers don't fully understand how things are represented at the low level, there is no purpose in storing a version/serial as a string.

yodelshady• 5 years ago

a simple tuple of appropriately-sized integer types would only need 8 bytes, which coincidentally is what MS will need come 2043.

That does mean rolling your own "greater-than-or-equal" operation though.

zitterbewegung• 5 years ago

The only reason I can think of is that storing a version as a number would allow you to check if your version is incompatible with another server or feature / extension. By using less than or greater than.

frays3• 5 years ago

If they were stored as characters you could still do a string comparison using greater than and less than signs and logic.

"202201010001" > "202112122359"

evaluates to True in Python, SQL and many other languages.

ilikehurdles• 5 years ago

Not necessarily.

2.2 is greater than 2.12 but v2.2 is actually an early predecessors to v2.12

tyingq2• 5 years ago

Not that it excuses the issue, but I suspect the motivation was numeric sorting to identify the newest. So it was sort of a number in that respect.

LadyCailin• 5 years ago

Not criticizing you, but no, it’s a string which happens to have a natural ordering. The fact that it happens to look like a number is irrelevant.

tomjen32• 5 years ago

But they are. Zips here in Denmark are clustered progressively, so that if you have a package going to (say) 8600 and 8704 then it makes some sense to send those orders to the same distribution node, since they will be close together, but to do that you need to treat them as numbers.

My grandparents first phone number was 13. Not because they got in super early and had the 13th phone number in the country, but because that was the 13th number for their exchange. When a bunch of exchanges were put together everybody got another number in front of theirs. Their phone number is much longer today, but it still ends in 13.

I guess you could treat the phone numbers as strings, but I don't know if you can do that with the Danish zip numbers.

pintxo2• 5 years ago

Lots of phone numbers start with a zero, converting string to numbers will most likely change the phone number by removing the information of the leading zero. Doing this in js will store it as base 8. Neither will make anyone happy.

silvestrov• 5 years ago

You should never use clustering/subtraction on Danish zip codes as a source of "how hearby". To quote https://www.postnumre.dk

   2400 Copenhagen NV (North West)
   2412 Santa Claus, Greenland 
   2450 Copenhagen SV (South West)    
   ...
   3790 Hasle (close to Roskilde)
   3900 Nuuk (Geenland)
   3992 Dog sled patrol "Sirius"
   4000 Roskilde

chiph• 5 years ago

Even in North America, you shouldn't treat ZIP/postal codes as numbers. The US added 4 digits 30-ish years ago (ZIP+4) and they're separated with a dash. You could store them as separate numbers [0] and rely on the presentation code to add the dash. But since Canada has alphanumeric postal codes (Mexico uses numbers), you're better off treating them as a string field.

The US used to prefix phone numbers with letters (like MElrose 1-2345) as an aide-mémoire during the transition to 8 digits with area codes. And you'll still see businesses advertising with letters in place of digits (like 800-GOT-JUNK which is a junk removal firm) but they never get entered into phone number fields like that. The best reason to have phone numbers as text though, is the digit groupings [1]. In North America, it's 3-3-4 but other countries group their numbers differently to help people remember them. So allowing users to enter numbers in their format you're making it easy for them.

[0] The main reason to do this is if you're in the business of sending out mailings. The US postal service gives discounts if you sort and bundle by ZIP code, and mixed ZIP and ZIP+4 values would make that hard.

[1] https://en.wikipedia.org/wiki/Chunking_(psychology)

akersten2• 5 years ago

A ZIP code is an identifier made of numerals, it is not a number in the mathematical sense. It does not make sense to multiply, add, subtract, or divide them - even though the numerals are assigned in apparently ascending order. There's nothing preventing a future decree from making one that starts with a Z, or a 0, or putting a dash in the middle of it. It is foolish to store it as an integer type.

You can cluster them for distance without storing them as an integer. Just throw them in a graph data type and compute edges having actual integral weights. The ZIPs themselves are not what you want to be doing your math on.

skrebbel• 5 years ago

Captain Hindsight strikes again!

fortran77• 5 years ago

How badly does MS run their engineering?

Bad enough to be a 2.53 Trillion Dollar company.

hiptobecubic• 5 years ago

Version numbers can of course be numbers, but the problem here is that they were constructing a number like a string from semantically meaningful constituent parts (and not testing thoroughly).

raverbashing• 5 years ago

And again, if there are no negative versions, they should have used an unsigned int, not an int.

MikusR• 5 years ago

But you can add and subtract dates.

Turbots• 5 years ago

Add March 5th to June 2nd...

db48x2• 5 years ago

You can only add a Duration to a Timestamp; you cannot add two Timestamps.

adolph• 5 years ago

The map is not the territory.

The efficacy of any computational operator on a specific representation does not impute the same for all instantiations since the semantic link between identity and representation may only exist in the mind of the programmer and not the computational device.

ajross2• 5 years ago

It's neither a phone number, zip code, nor model number. It's a version number. And version numbers don't multiply or divide, but they do compare. This kind of code has a decades-long history:

    #if BUILD_VERSION >= FIRST_VERSION_THAT_WORKS
    ...
    #endif

And in fact if you look around, almost every large C or C++ project has a WHATEVER_VERSION token (c.f. LINUX_VERSION_CODE for the kernel) that packs the version into a comparable integer literal.

The bug here is that they picked an encoding that was obviously going to overflow in a Y2.022K bug, not with the technique.

cesarb• 5 years ago

[...] almost every large C or C++ project has a WHATEVER_VERSION token (c.f. LINUX_VERSION_CODE for the kernel) that packs the version into a comparable integer literal. The bug here is that they picked an encoding that was obviously going to overflow [...]

Speaking of LINUX_VERSION_CODE, it had a similar problem recently. See the article at https://lwn.net/Articles/845120/ which was summarized in this year's LWN.net retrospective at https://lwn.net/SubscriberLink/879053/aaea44782e8c760d/ as follows:

"Sometimes the most obvious things can be the hardest to anticipate; consider, for example, the case of minor revision numbers for stable kernel releases. A single byte was set aside for those numbers, since nobody ever thought there would be more than 255 releases of a single stable series. But we live in a different world, with fast-arriving stable updates and kernels that are supported for several years. It should have been possible for us, and for the developers involved, to not only predict that those numbers would overflow, but to make a good guess as to exactly when that would happen. We were all caught by surprise anyway."

Pxtl2• 5 years ago

We need null-terminated binary-coded-decimals as a standard datatype for cases like this. "Numeric String".

ananonymoususer2• 5 years ago

Okay, so if it's null-terminated, how does one represent a zero?

NelsonMinar• 5 years ago

So.. are database IDs numbers? Because they sure are autoincrement integers in almost every DBMS. I always thought that was a mistake but it's practical.

Quekid5• 5 years ago

Yes, but as soon as they leave the realm of the DB itself, they (should) become opaque identifiers.

masklinn4• 5 years ago

So.. are database IDs numbers?

No.

Because they sure are autoincrement integers in almost every DBMS.

The autoincrement occurs before it's a database ID. The generator for database ids is a sequence, that has no bearing on the semantics of the database ids.

mind-blight3• 5 years ago

It's becoming pretty standard to use UUIDs over integers now. That provides more architectural flexibility

marcosdumay2• 5 years ago

They are absolutely not numbers.

As many people already commented, it's reasonable to encode things as numbers as long as you keep in mind there's an encoding operation with it's possible flaws.

petee17• 5 years ago

You're right but thats not the issue here, the version is structured as a number but they failed to give it a type large enough to represent it; simply a signed/unsigned screwup.

masklinn4• 5 years ago

That's not a fix, barely an improvement, really just a shitty workaround: the exact same issue rear its head again in 2043.

The actual fix is exactly what the original comment states: a version is not a number.

lolinder2• 5 years ago

The version is not structured as a number, it's structured as a string of digits that incidentally resembles a number. There are no operations that take two version numbers and return a new version number, so they are not numbers.

collegeburner• 5 years ago

Sometimes a performance reason? Number is usually smaller, faster, also better if stored in a database and more efficient if used as a primry key,

ClumsyPilot• 5 years ago

Gotta free up space for those electron apps

bob10292• 5 years ago

I have found that any time you would refer to a thing as a "Code" or "Number", you are almost certainly talking about a string. Every property name in our codebase that contains one of these terms is a string type without exception.

dannyw• 5 years ago

If it's not a float or integer, it's a string.

yholio• 5 years ago

These are version numbers. It's perfectly fine to use numbers to express version numbers, a great majority of protocols reserve numeric fields for this.

The problem here was basic overflow, they reached a version number exceeding the capacity of the numeric type.

rbobby• 5 years ago

YYMMDD + 4 digits of versions. At a glance, even a bit more than a glance, that seems fine for a long. And if 2020 worked and 2021 worked... why would 2022 break?

Lol.

I'd bet the conversation about this way back when it was being decided was "will 3 digits for daily version/build be enough" and the answer "go with 4, better safe than sorry".

snicker7• 5 years ago

Time is a number. At least, it makes sense to subtract them (I guess you can say that time is a one dimensional affine space).

kzrdude• 5 years ago

just like pointers, it makes sense to subtract them and add offsets but to straight up add them makes no sense.

pjc50• 5 years ago

Time is a number and a location. If you normalize everything to a reference time such as the UNIX epoch, you can usually do subtraction. Even then you may have got the times from a non-monotonic clock ...

bsza• 5 years ago

Time is a number, but a date is not. 211231 + 1 != 211232.

kaba02• 5 years ago

Bit of a nitpick, but due to relativity, time is not simply a number though.

pvillano2• 5 years ago

side note

some numbers^* cannot be meaningfully multiplied or divided. For example, 3^rd place, 30° F, or 37.388N,-122.067S.

Some numbers can only be added to by numbers of a different type. For example x°C + y°K, 3rd grader + year, absolute position + offset. Similarly, subtracting two of these numbers results in a different unit. e.g. position - position = offset

it would be great if someone could comment the exact terminology for what I'm trying to describe.

* Types/Units

mind-blight3• 5 years ago

I'm not sure if it's what you're looking for, but algebraic days types are a fairly common way to encode different kinds of days that have the same shape.

You may also be looking for fields: https://en.m.wikipedia.org/wiki/Field_(mathematics). You could probably also describe the relation between different number types as linear operators over a vector space

john-aj• 5 years ago

In statistics, one usually differentiates between

- nominal scales (e.g. man/woman),

- ordinal scales (elementary school/middle school/high school),

- interval scales (degrees Celsius) and

- ratio scales (human height, degrees Kelvin).

Only ratio scales may be multiplied and divided. Interval scales may be subtracted and added. Ordinal scales may only be compared (less/greater/equal). Nominal scales may only be compared in terms of (in)equality.

https://en.wikipedia.org/wiki/Level_of_measurement

jongalloway2• 5 years ago

F# has Units of Measure: https://docs.microsoft.com/en-us/dotnet/fsharp/language-refe...

There are libraries that implement this concept in other programming languages (C++, Java, C#, etc.)

samus• 5 years ago

Levels of measurement:

https://en.wikipedia.org/wiki/Level_of_measurement

Kevin5782• 5 years ago

isnt working with integers generally faster than working with strings? that might be why some ppl store phone numbers and zip codes as int

ben-schaaf2• 5 years ago

Not if the only thing you're doing with the number is converting it from/to a string...

reaperducer3• 5 years ago

that might be why some ppl store phone numbers and zip codes as int

Anyone who stores ZIP Codes as an int should have his dev license revoked. You've just corrupted the data you store for a hundred million people in the northeast.

I'm currently dealing with a situation where a system developed by an offshore team stored Social Security numbers as integers. They had no idea that an SSN can start with a zero, and didn't even do a basic web search to see what the possible range of values is before designing the database and application.

ed25519FUUU2• 5 years ago

This is so extremely common that it’s funny to me you scoff at it.

Under the hood these systems will convert to some numerical format to compare which version is newer. You store it as a (proper) numerical format or you serialize it into one later, but at the end of the day it’s a number because that’s how you compare it.

dheera• 5 years ago

How badly does MS run their engineering?

I think the real problem is ... how many good engineers and engineering managers nowadays want to work at MS?

djrogers• 5 years ago

Seriously, if you can’t add, subtract, multiply, and divide them, then they aren‘t numbers

You’ve missed a ton of other scenarios here, the relevant one being a compare operation with a < or > output. These are version numbers. Version numbers the way they’re used here increment, they are not random strings, which is what you’re effectively suggesting.

If the version numbers here were represented as strings they’d need to be converted to a number at some point to compare them, and you’d probably run into the same problem you have here.

Of course I’m sure you’d have the perfect, completely applicable, and universally accepted solution then as well…

lolinder2• 5 years ago

Comparing strings is trivial, and would work out of the box with the format that they're using (YYMMDD###).

Jenk5• 5 years ago

Version numbers

They still aren't compatible with number types. They are groupings of numbers at best.

2.9 is an earlier (not less than) version than 2.10.

3.1.6 isn't a (strict) numerical format.

5.4.11-beta1 has non-numerics.

akersten2• 5 years ago

the relevant one being a compare operation with a < or >

In Math we call that subtraction!

wolf550e• 5 years ago

ISO8601 strings are sortable and are not numbers.

pigbearpig• 5 years ago

These should have been strings, end of story. You can sort a string. Amazing to see storing a non-number as as number being defended. Truly CS101.

WoahNoun• 5 years ago

Posets aren't numbers either.

mr_toad3• 5 years ago

https://en.wikipedia.org/wiki/Ordinal_number

darau1• 5 years ago

We use exchange at work. Daily, I will utter some form of "Microsoft can't do anything right, man".

michaericalribo• 5 years ago

Well, to put on my pedantic hat and indulge my inner math major, what you describe is algebra on top of objects—you’re talking about the math that applies to numbers, not the numbers themselves. Specifically you described groups and fields and rings…the symbols themselves are numbers, but they don’t recombine with each other in the same way as other numbers.

You’re right that they shouldn’t be implemented a numeric data types.

That said, dates do have properties of modular arithmetic, so in a strange sense they are very much numbers…

dehrmann2• 5 years ago

You should try PHP. You can multiply all sorts of things!

dehrmann2• 5 years ago

I used to work on a customer API that accepted these fields as JSON. My first project was improving validation (I used JSON schema) while not breaking existing integrations. While it's true that those fields aren't numbers, I saw all those fields sent as numbers.

rbanffy6• 5 years ago

Seriously, if you can’t add, subtract, multiply, and divide them, then they aren‘t numbers

Sadly, sometimes you can, regardless of whether you should.

SAI_Peregrinus3• 5 years ago

I'd say if they don't form an ordered field, they shouldn't be called "numbers". The complex numbers really ought to be called an algebra, not numbers. They're the dimension-2 Cayley-Dickson algebra, after all!

comboy• 5 years ago

Everything ends up in your database as bytes. You can say everything is a number. It's just naming. Whether it's sane to do basic math operations on them is loosely correlated with whether user sees that number as an integer on his screen.

Most bugs are stupid and should be obvious. They happen.

You can look at your old code and say WTF, so judging decisions in somebody's else complex system without having any idea about requirements and constraints seems pointless.

hexo2• 5 years ago

Whats the point of using exchange? Why people can't just use regular mail servers? Whats different?

hahajk• 5 years ago

What are other good self-hosted email server options?

hexo2• 5 years ago

I don't know... postfix?

ananonymoususer2• 5 years ago

https://mailinabox.email/

frays3• 5 years ago

I'm more interested in hearing why these Sys Admins haven't switched over to O365 yet?

analog31• 5 years ago

Are they actually different, as in the nuts and bolts, or just different shells on top of the same infrastructure?

bluedino3• 5 years ago

Cost. Especially if you have a large amount of users, and you already have the IT staff and servers to run it on (you can try to justify it with less admin costs and less server costs but the numbers don't add up for some orgs)

As an example, I worked at a healthcare org and it would have been $70,000 a year for hosted Outlook, but only $110,000 to buy the software outright and run it for ten years (700,000 - 110,000 is a big savings)

ocdtrekkie• 5 years ago

A local Exchange server has vastly better uptime than Office 365, works when cloud services and the ISP is down, and costs drastically less. Hilariously, the standing advice has usually been for Exchange Online customers to still install and operate a local Exchange server too, which means switching to the cloud didn't even reduce your management or hosting burden at all.

Sure, you get the occasional bug like this, but it's less often than Microsoft screwing up the entire cloud.

bluedino3• 5 years ago

Calendars, push, a bunch of other stuff that's useful

It's not perfect but it's by far the best mail server for internal Email.

awill• 5 years ago

Every company I've worked at who uses self-hosted exchange uses it because that's what they used 20 years ago. I don't know companies deliberately using it.

I suppose office 365 is different. Smaller companies get a good bundle with Teams, Office, mail etc..

bogantech• 5 years ago

It ties in easily with AD, does shared mailboxes, calendars, contact lists, meetings, automatic out of office, sorting, legal hold, archiving, backup etc relatively easily without having to glue together a hundred different things.

MS will support it & offers training. It can be managed easily by the same windows admins they have managing their AD infrastructure and desktop support.

gtsop• 5 years ago

Interested to know if there are other legacy pieces of software in any industry having similar bugs

puyoxyz• 5 years ago

don’t link old reddit please, anyone who wants to use old reddit already has it set as the default, and for apollo users old reddit links don’t automatically open in apollo like normal reddit.com links

tuwtuwtuwtuw6• 5 years ago

Not sure what Apollo is, but it sounds like whatever it is it should be fixed instead of you asking people to give some alternate links to a worse version of the website.

ZeroCool2u• 5 years ago

Apollo is a 3rd party Reddit client for iOS. It probably should be fixed in Apollo. I know Sync on Android has no issue handling any of the Reddit sub domains.

butwhywhyoh• 5 years ago

Wrong, I don't use reddit at all and I prefer old reddit. It's not default for me since I don't have an account.

tuwtuwtuwtuw6• 5 years ago

Same here, but I would prefer if it linked i.reddit.com instead. Like this:

https://i.reddit.com/r/sysadmin/comments/rt91z6/exchange_201...

belltaco2• 5 years ago

https://chrome.google.com/webstore/detail/old-reddit-redirec...

iso12102• 5 years ago

So people should go out of their way to change their links to a system they don't like to appease people of some software nobody's heard of and presumably could trivially be adapted to recognise "old.reddit"?

SeasonalEnnui• 5 years ago

As a counter data point - please do link old reddit. The dark UI patterns on normal reddit are astonishingly hostile for an anonymous user.

netsec_burn• 5 years ago

If I remember correctly, HN automatically converts Reddit links to old.reddit.com (less user hostile).

tuwtuwtuwtuw6• 5 years ago

Sounds like it should convert it to i.reddit.com then since it's even less user hostile, at least in my view

technion3• 5 years ago

Can confirm, when I made this thread it didn't have the old. name in the URL.

dagmx• 5 years ago

The Apollo safari extension doesn't support it, but you can simply go to the share sheet and select Open in Apollo

jakub_g• 5 years ago

Semi-relevant: "Practical time machines" (ironically, written by MS engineer from another department)

https://textslashplain.com/2021/10/01/practical-time-machine...

linker3000• 5 years ago

We shouldn't blame Microsoft for this; I once worked in the veterinary industry, and one Practice Management System allowed you to register a pet including their real or approximate birth date, or by entering an approximate age.

If you entered a birth date, the software would calculate the pet's age, and we noticed we had the odd quirk, like a 400 year-old cat etc. I couldn't see an obvious pattern to the anomaly, and it didn't occur often, so I created a support ticket.

After a couple of days, I received a response that 'dates were hard' and we should correct the ages manually as needed.

So there you have it: Dates are hard.

(No, the bug wasn't ever fixed.)

goda90• 5 years ago

I feel like after Y2K, we should've learned to never store two digit years again. If the format they used was actually YYYYMMDDhhmm, then it would've crashed as soon as someone tried to convert it to an int and they would've fixed it without another thought. Sure dates are hard, but a few simple rules cover a lot of problems.

OisinMoran2• 5 years ago

Do you mean it would crash from that number being too big for a byte-sized int? Just checking I'm not missing anything here.

mark-r5• 5 years ago

They probably figure it will be no problem, who will still be using Exchange in 2100? Same thinking as the guys who set up all those databases in 1975. And at least once it overflows (again) it will be somebody else's problem to fix.

BrandoElFollito3• 5 years ago

So there you have it: Dates are hard.

Being an amateur developper, I never tried to touch a date in another way than though a specialized module (arrow in Python, moment or luxon in JS, ...).

I know only a few such modules so everything that remotely looks as time/date computation is a nail for my universal hammer.

joenathanone• 5 years ago

I learned this one the hard way when using jQuery Datatables, dates aren't natively supported for sorting, so had to integrate moment.js and learn the quirks of date formatting between it, Datatables, .net and sql, especially since with moment you uppercase everything MM/DD/YYYY vs MM/dd/yyyy, plus notation for AM/PM and seconds is also different.

Dates are hard.

OisinMoran2• 5 years ago

It's painful to learn of such a bug and have no insight into how it was caused. Would love to get my hands on the codebase to figure this one out!

In another universe The 400 Year Old Cat could have been our 500 Mile Email.

petee17• 5 years ago

We know how it was caused, bottom of reddit post is the update. The developers got clever and used a common method to store a version as an integer, but they never checked that they had enough space to store such a large number for the uncommon format they chose.

Its just another form of y2k, or y2038.

kingcharles• 5 years ago

Did you check with the cat to see if it was actually 400 years old?

Swenrekcah• 5 years ago

You joke, but if they had been tracking shark ages instead it might well have been!

https://www.bbc.com/news/science-environment-37047168

throwawayboise2• 5 years ago

Dates are hard, it's got to be one of the things on the lists of "things programmers don't understand."

Choices to represent dates as an integers (seconds from an epoch), or as strings, or as arrays, or as other data types all come with a lot of non-obvious consequences and edge cases to deal with. I always use a provided "date" datatype from a library, and never try to roll my own with base data types if I have any choice in the matter. And they are still hard, depending on what range of dates you need to be able to handle.

SamuelAdams• 5 years ago

Semi-related, one of my favorite posts on UTC. Seems simple to use until it’s not.

https://zachholman.com/talk/utc-is-enough-for-everyone-right

bfung• 5 years ago

Heh, if dates are hard now, just wait till 2038 (https://en.m.wikipedia.org/wiki/Year_2038_problem)

JdeBP• 5 years ago

The knowledge has indeed existed for years.

* https://news.ycombinator.com/item?id=4128208

lopatin2• 5 years ago

Sure, dates are hard. But I think it's too generous to say Microsoft shouldn't be blamed for releasing such a bug.

rbanffy6• 5 years ago

Dates are hard but Microsoft is a trillion dollar company that sells products on every locale and timezone on the planet.

Something like this is unacceptable.

raintrees• 5 years ago

Set-MalwareFilteringServer -BypassFiltering $True -identity <server name>

Then restart Microsoft Exchange Transport service.

New year: Building a postfix-dovecot server.

kizer2• 5 years ago

Seems like many bugs occur when programmers try to be clever unnecessarily. In this day and age, we don't really have to do tricks to save memory or time in application programs; it's just not worth it with modern silicon and compilers/runtimes. So many bugs could be avoided if people valued robust functionality with maybe a "boring" implementation over trivial applications of cleverness etc.

zbentley• 5 years ago

Time to trot this out again: https://infiniteundo.com/post/25326999628/falsehoods-program...

tomohawk• 5 years ago

2022 is already a great year!

tdiff• 5 years ago

So they are not checking security definition updates on some test systems before releasing? This is ridiculous.

Scarblac• 5 years ago

The old joke "Every program grows until it can send email, except for Microsoft Exchange" is still true on 1/1/22.

mitchs• 5 years ago

I'm curious, what everyone considers far enough in the future for your code. Eg. are you willing to write code that puts nanoseconds in a 64 but number? That is "only" 584 years after epoch. Would you consider that ok?

vlmutolo• 5 years ago

We're currently dealing with problems that arose from people assuming that their systems wouldn't ever be used for more than, say, 50 years. That assumption is frequently proven wrong.

500 years is pretty different. Odds are that computing will look nothing like what we know today. Our software will be long dead.

And if it isn't, there will be yet another Y2K bug in 500 years' time, and they'll solve it. Just like we keep solving it.

userbinator• 5 years ago

Did anyone notice this anomaly?

    220101001
    2147483647

Those don't line up. It seems like it's converting a larger number (10x?), but the error message mentions the above string.

neycoda• 5 years ago

This is just evidence we're in Year Zero.

Taniwha• 5 years ago

Surely one of the lessons here is that the big push to have languages detect arithmetic overflow are largely moot unless you actually do something useful when it happens (because here's a case where yes they did detect it but ....)

tluyben2• 5 years ago

Aaaa. Exchange. In another universe no one would be using it. We did a project for a large university and even MS could not make Exchange run stable (for a few 100k people); they had all servers reboot at 3 am or they would crash, lose data etc. It is old school horrible as MS used to create; they improved much over the decades but I guess they did not merge that mindset in the Exchange (or SharePoint) teams.

frupert52• 5 years ago

That's because they don't want to encourage the use of on-prem

mark-r5• 5 years ago

Instead of switching to unsigned 32 bit int to buy another 21 years, they should switch to 64 bit ints. Or even 64 bit double - inside every double is a 53 bit int hiding in plain sight! That would buy them another 90073970 years.

Microsoft Exchange stops passing mail due to bug on 1/1/22

Related Stories