Matrix.org Security Incident

https://matrix.org/blog/2019/04/11/security-incident/

by Perceptes • 7 years ago

74 15 7 years ago

matrix.org

arunc• 7 years ago

Why was Jenkins running on a production server?

deepwell2• 7 years ago

it was not, read again

Perceptes• 7 years ago

But it does seem to be the case that the same SSH key pair that was used to access Jenkins also provided access to the production infrastructure. Unless I'm misunderstanding the nature of the attack.

iancarroll• 7 years ago

Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure.

This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...

snailmailman• 7 years ago

i think so. here is a wayback machine link to the previous blog post https://web.archive.org/web/20190412000400/https://matrix.or...

it stated "Having fully flushed out the attacker [...]" which i guess turned out to be false :-/

also im getting invalid HTTPS certs on the blog now. for some reason im getting a cert that looks like its for github.com ?

edit: now im getting a lets encrypt cert on matrix.org, but a cloudflare SSL error page when i go to www.matrix.org ? the lets encrypt cert looks like it was just issued about an hour ago.

edit2: i guess both with and without www. are lets encrypt, but the with www. cert was issued back in february (and gives a cloudflare SSL error page), while without www. was issued today. (and gives the current hacked message)

deepwell2• 7 years ago

Here is the message of the hacker, after the matrix.org admins allegedly cleaned everything up and wrote the original blog post: https://web.archive.org/web/20190412055614/https://matrix.or...

ge0rg• 7 years ago

The most favorable reading of the current defacement page is that the attackers still controls the DNS, but no other parts of the infrastructure.

Otherwise, the page probably wouldn't run off github.

localhostdotdev2• 7 years ago

content before it gets fixed:

    Time for actual transparency.

    [list of servers, uname -a for each]

    root@[name]:/var/lib/postgresql# df -h
    [list of partitions]

    $ cat users.txt | grep [name] | head -n1
    @[name]:matrix.org|[hash]
    $ wc -l users.txt
    [~6M users]

    See you soon.

(affects whole site, even https://matrix.org, site is on jekyll BTW)

l2dy• 7 years ago

Here's the source of that page: https://github.com/matrixnotorg/matrixnotorg.github.io/blob/....

localhostdotdev2• 7 years ago

account got banned :) (it's on archive.org though)

mkj• 7 years ago

As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history

That seems like a fairly bad usability/security design?

zigara2• 7 years ago

The attacker seems to have responded:

https://github.com/matrix-org/matrix.org/issues/357 edit: just saw the rest: https://github.com/matrix-org/matrix.org/issues?utf8=%E2%9C%...

"[SECURITY] SSH Agent Forwarding

I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out.

Complete compromise could have been avoided if developers were prohibited from using ForwardAgent yes or not using -A in their SSH commands. The flaws with agent forwarding are well documented."

irgeek• 7 years ago

Assuming the GitHub issues are from the actual attacker -- and I see no reason to doubt they are -- this is very troubling:

https://github.com/matrix-org/matrix.org/issues/363

Compromise began well over a month ago

Yikes. That's a long time for a compromise to go unnoticed.

ygjb• 7 years ago

Not really, the average in the industry seems to be floating between 70 and 400 days depending on the source of your stats on the topic (different vendors and reports use different stats for this)

Matrix.org Security Incident

Related Stories