XZ Backdoor: Times, damned times, and scams

https://rheaeve.substack.com/p/xz-backdoor-times-damned-times-and

by lamontcg • 2 years ago

277 170 2 years ago

rheaeve.substack.com

goalieca• 2 years ago

One of Jia’s followers on GitHub is form Eastern Europe.

Edit: it’s a meme. Apparently. I feel like an old man.

illia-git This user is suspected to be a part of an online ransomware organization. Please report any suspicious activity to GitHub security. Poland

mintplant• 2 years ago

The message is a custom status, and probably a reference to a Discord meme:

https://knowyourmeme.com/memes/discord-user-is-a-suspected-t...

juitpykyk• 2 years ago

FBI could subpoena GitHub for IP addresses.

tobyp-• 2 years ago

Considering what the account was up to I sincerely doubt it was being used without Tor or a VPN.

chgs5• 2 years ago

This article asserts some opsec failures - time zones switching when they shouldn’t etc.

It’s quite plausible that they didn’t manage perfect vpn usage every single time.

arp242• 2 years ago

Probably; but ask Ross Ulbricht how easy it is to screw that up.

The thing is you only need to make a mistake once.

gpm• 2 years ago

They probably have/will, though we are unlikely to find out what they get unless they manage to arrest/charge/try him.

Jia Tan probably used a vpn though - we know that they did for accessing IRC (source: https://boehs.org/node/everything-i-know-about-the-xz-backdo...)

cft2• 2 years ago

Most (but not all ) VPN providers keep logs and payment info that are subpoenable. You could use something like Mulvad with Lightning Network payments, but I am not sure that even that is fully anonymous.

The Witopia VPN that he used for IRC [1] is US based: https://www.personalvpn.com/contact-us/ and they don't mention neither LN payments nor not keeping logs.

1. "~jiatan@185.128.24.163" https://boehs.org/node/everything-i-know-about-the-xz-backdo...

pquki42• 2 years ago

Not going to be useful if they consistently used a VPN to access GitHub. But people make mistakes sometimes.

earslap• 2 years ago

history shows that it is generally very hard to not slip up here and there, especially if you are not expecting to be a huge target.

xuki• 2 years ago

But Sir The Calls Are Coming from Inside the House!

BlueFalconHD3• 2 years ago

this [1] but with handcuffs

[1] https://i.redd.it/obvqaa0lhn841.jpg

champtar• 2 years ago

The ssh public key might also be interesting, given the opsec failures, they might have used it for other accounts / at other providers

BlueFalconHD3• 2 years ago

If a US gov. agency is "Jia Tan" then this might not happen.

k8svet3• 2 years ago

Shame on me for not reading more before my now removed comment. Shout-out for doing this level of analysis and guessing, as with everything in this incident, fascinating and tantalizing to wonder about.

It is interesting, at least one of the things listed as suspicious is something I do with some frequency. I will sometimes work on what is logically three commits and not chunk them out until the very end.

A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?

Twirrim2• 2 years ago

A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?

I don't think much speculation is needed. RPM and deb catches every Enterprise Linux distribution, to the best of my knowledge.

rpm catches Red Hat and all derivatives like CentOS, Alma Linux etc, as well as SuSE, Amazon Linux, and even Microsoft's Mariner Linux (now renamed Azure Linux).

deb catches Debian and Ubuntu.

That's a huge global install base just through those two targets.

k8svet3• 2 years ago

Sure, but why go to the effort to exclude others? Did they think it would help avoid detection on systems that they didn't care to backdoor?

sihe4• 2 years ago

A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?

The payload only works on systems that connect their sshd to systemd-notify -- which is not true for many Linux distributions beyond Debian-based and RH-based anyway (e.g. Arch Linux doesn't do this).

optimalsolver• 2 years ago

Poor Jia. Two whole years of work down the drain. If you're reading this Jia, remember that you miss 100% of the shots you don't take. Chin up, brother.

sva_• 2 years ago

Who knows how many alter egos the individual behind that online personality has, and what other projects they poisoned through those.

rudedogg• 2 years ago

Yeah, with a 2-3 year wait to know whether you succeeded or not, I doubt you'd put all your eggs in one basket.

tiahura• 2 years ago

Don't worry, they (it's a team) also contribute to image libraries, webkit, Kubernetes, ...

krick4• 2 years ago

Yes, unironically, that, being a huge leak, really convinces me that we all have multiple RCEs on each of our systems. I mean, we already kinda knew that, of course, but even now that I'm typing this it's a bit hard for me to actually believe it. But realistically I think it's pretty much a proven fact. This level of sophistication… what did I even expect? Of course well-payed teams of high-level professionals are working in this area for a very long time by now. This is basically like professional football players competing against office workers in their late thirties who like to kick a ball on weekends. I am comforting myself by thinking that something a bit more high-level, written in python or even golang, or, oh hell, maybe even rust would be more a bit more transparent for other contributors… but even this is probably a lie. And if it wouldn't be, surely tons of super-popular low-level packages written in C/C++ are as good as incomprehensible for somebody who isn't specifically security-auditing this.

est2• 2 years ago

remember that you miss 100% of the shots you don't take

But I am sure JiaT didn't only work on the xz project?

beanjuiceII• 2 years ago

How is it down the drain? You know he didn't achieve his goals?

EasyMark• 2 years ago

yep, maybe it was a false flag to take people's eye off the -other- backdoor lol

dotps13• 2 years ago

Occam's razor says that the perpetrators likely didn't spend years building up trust to hack some early adopters before the releases got more stable.

It would be like spending hours baking bread only to eat it before it goes in the oven.

throwawayyy9237• 2 years ago

Setting aside the gravity of the situation, I'm 100% gripped by the "mystery novel" side of this. Seeing the internet piecing together the Who? the How? an the Why? is fascinating.

mavili• 2 years ago

I dont know what the XZ backdoor is about but, to assume someone putting backdoors into software would work usual working days and hours has to be very naïve. The whole premise of the article is based on a false assumption IMO.

chgs5• 2 years ago

If you’ve missed probably the largest cyber security story since stuxnet, and arguably bigger than that, I suggest you start looking at the last few days.

Start here.

https://news.ycombinator.com/item?id=39865810

vsnf• 2 years ago

arguably bigger than that

That’s a stretch. Stuxnet was the first acknowledged state cyber attack, utilized multiple zero days, and destroyed nuclear weapons manufacturing facilities. Bigger in scope sure, but bigger unconditionally? I don’t know about that.

bagels• 2 years ago

Governments employ armies of hackers who just may work usual working days and hours. I'm sure some of them don't, but some of them probably do.

thrdbndndn3• 2 years ago

I suggest not putting a UTC+8 timezone graph as header image, when the conclusion is he's likely from Eastern Europe (not saying you should put one showing EET timezone instead).

I understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't (hence why clickbait works). I also think it's a little tasteless, if not misleading.

Disclaimer: I'm a Chinese.

willsmith72• 2 years ago

how is it tasteless? the only fact is that, almost always, Jia Tan's timezone was UTC+8.

the rest is mostly speculation about where else they could have been from given holidays and times

pvg• 2 years ago

The whole enterprise is a little tasteless with a chance of worse. Not some substacker or tweeter speculating in itself - people are obviously free to let their speculation flag fly. But you start amplifying this stuff on an open internet messageboard and next thing you know, it's reddit 'found' the Boston bomber. The local downsides outweigh the gawking value and I say this as a dyed-in-the-wool internet gawker.

gmassman2• 2 years ago

It seems no more misleading than Jia Tan assuming an identity based on git commit timestamps…

ShamelessC• 2 years ago

understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't

People who don’t read the article and find themselves lacking an informed or nuanced understanding of the contents of said article are the only ones to blame here. Full stop.

That HN doesn’t have a culture of shaming those who don’t read the article is partly the fault of the rules here which finds “read the article please” to be a distraction/nuisance but don’t mind that this often results in misinformed discussion propagating toward the top of a thread.

thrdbndndn3• 2 years ago

By that logic there is also nothing wrong with clickbaity titles, which I disagree.

77pt77• 2 years ago

export TZ=Z will just tag all your commits in UTC.

Just a heads up for the next guy...

bombcar2• 2 years ago

Write a script to select a (random) time zone so all your commits are within an hour of noon local time zone time :)

widdma• 2 years ago

given that Australia does daylight savings time

DST varies by state in Australia. Western Australia (UTC+8) does not observe DST.

mik19982• 2 years ago

I don't think he was from Eastern Europe, but if you want to look at UTC+0200/+0300, in Europe this only includes Finland, Baltics, Ukraine, Romania, Moldavia, and Greece. But notably if you look a bit down it also includes a good chunk of the Middle East, including Israel.

bradleyjg• 2 years ago

What about the daylight savings time and holidays points? How do they line up for Israel?

mik19982• 2 years ago

I don't know about holidays but Israel uses the same daylight savings as the mentioned Eastern Europe countries, ie switching from UTC+0200 to +0300 at around the same time - only difference is that they start daylight savings on Friday instead of Sunday the same week.

ufo• 2 years ago

I don't know all the holidays but they have commits on Yom Kippur dates in 2021 and 2022. For 2023 it's harder to say because their commit activity is sparser.

hnthrowaway03282• 2 years ago

Israelis usually are off on Fridays so it doesn't match the git commit records.

chgs5• 2 years ago

Not sure why you’re downvoted. When you think of state actors, Israel comes very high (stuxnet etc), as well as the usual US/Russia/China/NK groups.

Unit 8200 in the IDF especially have a very notable reputation.

That’s not to say other counties don’t have capabilities (and this doesn’t look like you need the resources of a group like say the NSA or GCHQ for this particular attack - indeed it could just be a single lone wolf) but it’s noteworthy.

krick4• 2 years ago

If the only think I knew was UTC+2 (and I don't even think I know that, I'm really cautious with assumptions about what mistakes that kind of people plausibly make), Israel would be very high on the list. But FWIW, 25Dec is a Catholic holiday, and even though it's mentioned on Wikipedia page about Israel holidays, I'm not sure how official that is. I mean, it doesn't seems to be, like, a real Israeli holiday.

est2• 2 years ago

even a North Korean hacker would be plausible.

lenerdenator3• 2 years ago

“Trust no one! The minute God crapped out the third caveman, a conspiracy was hatched against one of them! ” - Gen. Hunter Gathers, OSI (The Venture Bros.)

There are parties who have effectively endless resources and motivation to mock up a false-flag event to steer the responsibility for this a certain way. They're all very, very good at covering their tracks, and even the most experienced security researchers will take months or years to just get an educated guess of who could be responsible. Trying to figure out who did this is a waste of time.

The lesson is this:

1. Never, ever install bleeding-edge software in production, for any reason.

2. Pentesters in your organization should be regularly trying to blow holes in your stack and let both you and package maintainers know the result. If they're not, they're not doing their jobs.

3. FOSS maintainers should audit the new code in each release for security issues, particularly for things that are obviously security-sensitive.

4. Donate to your FOSS maintainers; they do insanely important work.

These rules will pay off no matter who or what wants to break into systems.

This fortunately didn't get to stable. This was as close of a shave as you can possibly get without a security Chernobyl in your systems.

EDIT:

Downvote all you want; I'm right.

There are people who very much want to convince you and everyone you know that a certain party is responsible for this and every other attack you read about. They'll go out of their way to do that. There are geopolitical goals to be achieved by doing so.

We're currently talking about banning foreign ownership of an incredibly popular web app in the US right now. If you are a government or commercial entity that would benefit from such a law, how convenient would it be if there happened to be a GitHub user with a Chinese-sounding name who committed an attack vector to the project with a timestamp that looked like it could have come out of the PRC?

And let's say it is someone in mainland China. We find out they are beyond a shadow of a doubt and who they are on a personal level. Some big-time DA's office like the Southern District of New York puts out an indictment for them and requests extradition. The people who are behind this are almost certainly state-sponsored and there's no way their asses are being handed over to US Marshals, ever. You could even argue that if they managed to stumble into a situation where they could be extradited, the government responsible for backing them would "tie up the loose end" before letting that happen, lest greater knowledge of what the organization has been doing fall into enemy hands.

Besides the Bond-style intrigue, there's no practical application to the knowledge. You already know where your users are coming from, most of the time, and might be geo-blocking based on that alone. If not, you know you should be alert to other threats from that region. If you're not, you aren't doing your job.

threatofrain• 2 years ago

Never, ever install bleeding-edge software in production, for any reason.

But for any X which is mainstream today someone was always the first of X. And even then being the second or third is still cutting edge. For a certain kind of company it makes sense to play this way, but if it were everyone then we'd have a tragedy.

jamebond• 2 years ago

Tukaani is a Finnish organisation primarily contributed to by Finnish people. You look at the contributors and almost all of them have Finnish names.

Seems to check out to me.

lrasinen• 2 years ago

Noticed a 4-week empty block in August. This lines up with European holiday schedules. Less common in Finland though, we prefer July.

BluSyn• 2 years ago

For years I've had `gc` in my terminal mapped to `TZ=UTC0 git commit` for exactly this reason. No need to change system times or git settings. (though perhaps a better way in global config?)

It's not even for nefarious reasons. I travel a lot, and don't like leaking my travel itinerary on public repos.

nerdponx• 2 years ago

How about this?

  alias git='TZ=UTC0 git'

yanateras• 2 years ago

Even without the zero!

  alias git='TZ=UTC git'

sneak• 2 years ago

I travel a lot but never change my workstation’s time zone out of PST/PDT these days. That would be a manual step and I don’t use Location Services.

akira25013• 2 years ago

Time zone is purely a displayed time consideration. I'm not bothered that the display time on my computer is different from the time zone I'm actually in.

kelnos3• 2 years ago

Same. My phone automatically changes, and I'll manually change my watch, but when I travel, I leave my laptop in my home time zone. Part of it is I just can't be bothered, and part of it is I want an easy way to know what time it is at home without having to fiddle with a world clock.

SmartHypercube• 2 years ago

I have been using a small tool I made to hide the time information in my git commits: https://github.com/SmartHypercube/git-date-truncate

It not only sets the timezone to UTC, but also sets the time to 00:00. So my commits will only have date information. I think only saving the date permanently in the git commit is good enough. No need to leak the precise time I made commits.

lapcat5• 2 years ago

who regularly works in the early morning?

For a hacker, it is much more plausible to work in the afternoon and late at night

c/hacker/younger person

lucb1e3• 2 years ago

c/hacker/younger person

What is this syntax? I know s/a/b/ for sed substitution but not such a thing starting with c

someplaceguy3• 2 years ago

What is this syntax? I know s/a/b/ for sed substitution but not such a thing starting with c

It's the syntax for sed substitution for someone who mistypes, is dyslexic or does not have an excellent memory.

jtuple• 2 years ago

The "hacker" stereotype is certainly stale, but subbing "young person" (and implying not "old person") is equally silly.

I'm 41 and have been awake before 7am maybe 20 times in the past 10 years. Half the reason I still put up with computers is because it's one of the few professions where I can work a 11-7pm schedule most days and still excel.

There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.

lapcat5• 2 years ago

There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.

Oh, my sweet summer child.

Talk to me again in another decade or so.

evilDagmar• 2 years ago

No, stereotypes aside, hackers who code diligently tend to wrap their "actual work" schedule around their external obligations, and while the morning is generally filled with distractions like dealing with "morning people", after lunch (an on into the evening) there are generally far fewer interruptions. I can honestly get much more done with an uninterrupted four hours than I can in the entire eight office hours of random phone calls and emails coming in, and I am no longer a "young person", and I'll do some stuff at night just because I know it'll take half the time if I'm not interrupted and I've had a few hours to at least intermittently mull it over. If you're having peaceful mornings, I envy you.

lapcat5• 2 years ago

stereotypes aside, hackers

If we're putting stereotypes aside, wouldn't we also put aside the word "hackers"?

In the xz case, the term may be accurate, but we don't know the identity of the culprit(s) and have no empirical basis for even speculating about the work schedules of such people.

If you're having peaceful mornings, I envy you.

Well, I don't work in an office. I work alone. Like a stereotypical "hacker".

lenerdenator3• 2 years ago

Generally, anonymity in the free software sphere is a good thing: software is inherently based on accomplishment and merit, and there is no reason to know anything about a person’s identity.

This is an interesting take. Most software engineers I know love to show off their FOSS contributions. It's a place to do what _you_ want to do, free of budgets and MBA interference. I guess in certain parts of the world you probably want to keep that sort of skill set under wraps, though.

gmassman2• 2 years ago

I think personality matters a lot here. One person may flaunt their open source contributions to their colleagues because they like the kudos. Another person may quietly commit and never publicize themselves because they don’t like being the center of attention. Both can even be true of the same person at different points in their careers!

Dibby053• 2 years ago

It would be interesting to see how the commit timestamps change around time change. If he has a job in a country that observes DST they would likely show an abrupt change.

s_Hogg• 2 years ago

Saying "Eastern Europe" as opposed to "Moscow" is a little coy, isn't it? They're in the same time zone and one seems likelier than the other, to put it one way.

Edit: speaking of, these guys might also have some time pressure at the moment with regards to hacking stuff lol. Kind of fits the bill with what we know about the hack

chgs5• 2 years ago

Moscow is UTC+3 year round, not UTC+2 in winter and +3 in summer

alisonatwork• 2 years ago

Israel is perhaps more likely given daylight savings.

sureglymop• 2 years ago

Why? There would be a number of plausible location based on the time zones. Just asking why you would make that conclusion in terms of rationality.

tete• 2 years ago

What if it is an American (random other place) pretending to be an Eastern European pretending to be Chinese? ;)

If I would be pretending something I'd certainly put at least one more indirection in.

rvba• 2 years ago

Or a Chinese pretending to be an eastern european or israeli pretending to be a chinese?

In such setup nothing bad happens in such a leak.

qiqitori• 2 years ago

Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.

Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?

someplaceguy3• 2 years ago

Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?

Yes, and even worse:

Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!

No, this is a difference of almost 10 hours...

Maybe the author inadvertently switched the two analyses? Or maybe the author is just not very good with timezone math... :)

sihe4• 2 years ago

Author here. Thanks for pointing this out – we switched the examples. (although we also are very bad with timezone math at 2 am.)

Sophira• 2 years ago

I would remind people that it is not difficult at all to have someone's sleep schedule match another country, particularly when we're dealing with what seems to be a state-sponsored attack. Times are a red herring - don't be fooled.

tgsovlerkhgsel• 2 years ago

And yet in many clearly-attributed cases the times lined up with the attacking country's office hours.

wannacboatmovie• 2 years ago

Everyone here is acting as if it's impossible to script git commits or email replies as a scheduled task or even put someone on the night shift to click send - to give the appearance of being somewhere you're not.

There was WAY more than one person involved here. Every email, commit, etc. was a deliberate task that was carefully planned.

krick4• 2 years ago

Would be nice if whoever made the analysis would attach the actual data in csv or something. I mean, it's a bit handier to overview than actually cloning the repo(s) ang using git log, if I'm not gonna dive deep.

broknbottle• 2 years ago

Why not include the timestamps for their replies to the mailing list?

https://www.mail-archive.com/xz-devel@tukaani.org/

cozzyd5• 2 years ago

Do you understand the time format there? It lists things as -700 or -800 for me, is that because of my local time zone (Chicago) vs. the mailing list (Finland?).

Delk• 2 years ago

I think mail-archive.com just shows time stamps in UTC-0700 or UTC-0800 regardless of where you are. That's how time stamps appear on all of the few mailing lists I checked out there, regardless of the sender of the message.

dc-programmer• 2 years ago

Eastern Orthodox Christmas is not on Dec 25 so the argument for EE is not as air tight as the author thinks it is

cozzyd5• 2 years ago

Depends on the country. Greece, Bulgaria, Romania and now Ukraine celebrate on Dec 25.

BlueFalconHD3• 2 years ago

If some nation state actor dis this, I imagine they couldve easily modified times of their commits, takin into consideration holidaya and such. Also, nation state actors have tons of resources. They could have people waiting to make commits or schedule it with something like cron.

ufmace• 2 years ago

Probably worth considering here - if this was a state-sponsored attack, it's entirely possible (and exactly what I would do if I was running such a program) that there is one person or department actually writing the code and list messages etc, and another person or team who sits between that department and the internet and whose entire job is to ensure that everything that comes out from the development team takes place at actual times and with timestamps that are consistent with whatever story they're using for the project. Possibly even once in a while intentionally inserting a "mistake" to hint to investigators that the person's real location is another chosen place, which is also different from the real place where the project is being run.

bombcar2• 2 years ago

Exactly. Once you assume that it’s more than “a lone wolf/single guy” many more things become possible.

Including carefully curating things to appear in a location and to NOT appear to be more than one person.

dotps13• 2 years ago

Always .. however the holiday schedule thing does stand out as something that could have been missed by the orchestrators.

apantel• 2 years ago

This is the real point.

omoikane• 2 years ago

It's likely that there is a whole team behind this operation, and part of that team might include tasks such as "socially engineer XZ project to accept a maintainer of our choice."

This was suggested in a different post[1], where there seem to be newly created accounts that came out of nowhere to create a sense of urgency, which might have accelerated the maintainer role transition.

[1] https://connortumbleson.com/2024/03/31/watching-xz-unfold-fr...

via: https://news.ycombinator.com/item?id=39888854

Search for "Progress will not happen until there is new maintainer" for the relevant text.

4bpp• 2 years ago

GMT+8, the name (Mandarin/mixed-dialect first name + Hokkien surname) and most importantly the circumstance that the user was seen connecting from a likely VPN exit or hosted server in Singapore [1] are all consistent with a Singaporean identity, whether real or cover. For that reason, the opinion of [1]'s source that the name sounds fake should also be taken with some amount of salt (Mainland Chinese are not always well-informed about Diaspora Chinese communities). Worth noting that many of the holidays mentioned, likewise, are not public holidays in SG [2] - Jan 24th was, but Jan 22nd (listed as working) was a Sunday, inconsistent with a "working on regular working days" pattern most anywhere.

Would be interesting to see a greater corpus of Jia Tan's writings, especially older ones. There are fairly distinctive idiosyncrasies that may be used to distinguish Singaporeans, Mainland Chinese and speakers of various Slavic languages (and Anglo natives).

[1] https://boehs.org/node/everything-i-know-about-the-xz-backdo...

[2] https://www.mom.gov.sg/employment-practices/public-holidays

qsi2• 2 years ago

In the few commits from Jia Tan that I've seen, the English parses (to me at least) idiomatically as a native speaker from either the US or UK, and I did not encounter any signs of Singaporean English. Obviously, this is a very small sample so one cannot read too much into it.

Almost 10% of Singaporeans share the family name Tan. [1]

[1] https://en.wikipedia.org/wiki/File:Singaporean_surnames_by_f...

delta_p_delta_x• 2 years ago

the English parses (to me at least) idiomatically as a native speaker from either the US or UK, and I did not encounter any signs of Singaporean English

Singapore is an Anglophone country. Many Singaporeans are perfectly capable of writing (idiomatic) BrE or NAmE, given the medium of instruction in Singaporean education from kindergarten to university is English. Example: me.

gfaure• 2 years ago

It's also worth noting that Tan is a Mandarin Chinese surname too (譚) in addition to being a distinct Hokkien surname (the much more common 陳).

idlephysicist• 2 years ago

Something feels weird about the timezone arithmetic here, though maybe I am nuts. Take this for example:

Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.

Putting it all into UTC we get:

    23:38 - 8 = 15:38
    17:27 - 3 = 14:27

That's not 9 hours by my numbers. Or another example:

Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!

Putting it all into UTC we get:

    21:53 - 3 = 18:53
    17:00 - 8 = 09:00

I think these timezones were swapped (accidentally) by the article's author. If we do the calculation again:

    21:53 - 8 = 13:53
    17:00 - 3 = 14:00

That _is_ a matter of minutes.

sihe4• 2 years ago

Yes, thanks for pointing this out! As somebody else already speculated, we got the two examples swapped. Already updated the post.

fullstop2• 2 years ago

If you don't have your BIOS clock set to UTC and your OS is expecting it to be, you can get strange behavior like this. If they were switching between Windows and Linux, that could explain some of the offsets.

teddyh• 2 years ago

You can fix that in Windows: <https://wiki.archlinux.org/title/System_time#UTC_in_Microsof...>

sihe4• 2 years ago

Hi all! One of the authors of the original piece here. Thanks for all your comments – we picked up a lot of cool ideas for future analyses. Just to address four common objections:

- We agree it's unclear whether this was one person, or several ones. But even if it was a team, the timezone idea would still tell us where the team is located (the timings seem too much like regular office hours for a globally distributed team).

- It's also absolutely possible that commit times were changed, or commits were made/uploaded using timed scripts. Still seems like it would be hard in practice to use this to cover up a massive difference in timezone, though, because it would require adding a massive latency -- and xz wasn't developed in a vacuum, but in reaction to other online events like the filing of issues or mailing list posts.

- yes, we did get the two examples mixed up (the two times we say are 10-ish hours apart are actually only minutes apart, and the times we say are minutes apart, are 10-ish hours apart). Update is on the way.

- and finally, yes, UTC+2/3 goes beyond just Eastern Europe and includes part of the Middle East. We also clarified this in an update to the article.

1letterunixname3• 2 years ago

Giant, fragile, pointless build infrastructure is always suspicious. Just take k8s or any other Go cli utility unable to build directly as it was originally intended back when there was go get and now go install.

Simplicity and clarity translate into security, maintainability, and reliability.

kijin• 2 years ago

you will often not be able to change the time without adding latency, i.e., withholding commits and thus delaying project development -- and even then, getting it right every time is hard.

Someone with access to the right machines could easily check whether this is the case. A lot of projects have set up webhooks on github to sync new commits with their own bug tracker, for example, or testing infrastructure. Logs on these systems would tell you exactly when the commits were pushed.

fbdab1032• 2 years ago

I wonder what the "Lessons Learned" look like from the other side?

Are they whiteboarding the best strategy for distributing the timestamps of commits? Mandatory performance evaluations on all RCEs? Distribute the poisoned commits amongst more authors?

farmdve2• 2 years ago

They will likely now optimize the code to not trigger high latency and also ensure no Valgrind will find any issues.

someotherperson4• 2 years ago

Maybe something that can be done if it hasn't been done already: semantic analysis across the different platforms they communicated on, to determine if it was the same person. Also semantic analysis on the same platforms using different cohorts of time (days of the week, months of the year, quarters, whatever).

Also: analysis on the email chains they interacted with. It's a possibility that they may have responded to their own emails in their mailing lists with fix suggestions/requests for clarification etc.

someotherperson4• 2 years ago

Maybe a consideration: this isn't necessarily a nation state. It could literally just be one or more individuals trying to set up some sort of crypto heist. At this point we need some sort of new adage to the effect of "never attribute to nation states what can be attributed to crypto"

farmdve2• 2 years ago

To me this is false. They usually like quick profits. Look no further than Ethereum and defi chains like Optimism,Polygon etc. It is so easy to write a malicious contract that even I with more than 12 years in the cryptocurrency space got had and lost 1k.

dotps13• 2 years ago

The attackers would have to know the distributions used by the company they are targeting, that connections are open to the internet, that the company won't change their setup over the course of several years.

Everything you are saying is so far out of the realm of possibility that it makes me wonder if you follow this stuff at all.

If you had 2+ years of spare time, and these type of skills, the attacker would be far better off just trying to get a job inside the organization they are targeting.

someotherperson4• 2 years ago

If you had 2+ years of spare time, and these type of skills, the attacker would be far better off just trying to get a job inside the organization they are targeting.

Who says they don't? :-)

crem• 2 years ago

Here is the a distribution of commit times and days of week (hopefully I didn't mess up time zones and plot everything in UTC): https://imgur.com/a/1js5T8L

~~It has surprisingly many commits on Sundays.~~

UPD. Weekday 0 is actually Monday, not Sunday.

knorker• 2 years ago

You can commit while on a plane, with git. And on many flights even upload these changes. So that's not really a slam dunk.

Everything else: Very interesting!

XZ Backdoor: Times, damned times, and scams

Related Stories