Slack's private GitHub code repositories stolen over holidays

https://www.bleepingcomputer.com/news/security/slacks-private-github-code-repositories-stolen-over-holidays/

by mindracer • 4 years ago

347 139 4 years ago

photoGrant• 4 years ago

This is how LastPass started the efficient downward spiral, no? This isn't shocking, but also is.

Is it wise to simply keep private repositories away from GitHub at this point? It seems the best way to avoid being drawn in with the rest as a target.

Internal bad actors? So location wouldn't matter?

verdverm• 4 years ago

If you read the details...

The incident involves threat actors gaining access to Slack's externally hosted GitHub repositories via a "limited" number of Slack employee tokens that were stolen.
While some of Slack's private code repositories were breached, Slack’s primary codebase and customer data remains unaffected, according to the company.

KingOfCoders3• 4 years ago

Would a hardware token have prevented this (noob question)?

throwaway010523• 4 years ago

Wouldn't private git hosting add another barrier for the attackers? In addition to the tokens, they would also need to somehow gain access to the network through VPN or through an opening in the firewall?

cj3• 4 years ago

Is it wise to simply keep private repositories away from GitHub at this point?

This sounds like the equivalent of "Use MacOS because most viruses target Windows". Which some people do, and yes it probably lowers your risk, but there are ways to use Windows (and Github) securely.

If you're really paranoid about your source code, you should probably self-host your own instance of Github/Gitlab/etc (and lock it down through VPNs, IP white lists, etc) rather than using their cloud service.

This approach also gives you first hand access to logs that would expose internal bad actors, if that's included in your threat model.

lrvick2• 4 years ago

there are ways to use Windows (and Github) securely.

No proprietary software you have no control over or ability to audit can ever be considered secure.

cadamsau• 4 years ago

there are ways to use Windows ... securely

Going to respectfully elaborate on this specific point, with full awareness that it's not core to the discussion, and that my opinion is my own and highly subjective, and that I'm not fully disagreeing but wish to point out a nuance. There are too many ways to put hidden software on a Windows system, which makes it harder to use securely than my OS of choice, macOS.

It's more difficult to audit a Windows system than a macOS system. For example, background processes can be installed on Windows in dozens of ways. Go download AutoRuns (https://learn.microsoft.com/en-us/sysinternals/downloads/aut...) and look how many tabs it has. The docs even say "You'll probably be surprised at how many executables are launched automatically!". UAC mitigates this somewhat, but it lacks granularity; it's all-or-nothing so you'll click Allow without knowing what's actually going to happen. And then you have COM registration and win32 APIs that let applications get at other windows easily. Layer upon layer of legacy compatibility make the whole system feel like a mess - at least to me.

Using macOS securely is also quite involved of course - for example auditing Homebrew's behavior when it downloads half the open-source world just for one package. But when it comes to stuff running in the background, by comparison, macOS Ventura recently added a comprehensive list of launchd agents and other daemons that run at startup; each of which can be toggled (though the UI doesn't allow you to drill into them). It's all on one pane in System Settings, built into the OS, no need for Sysinternals' separate tools.

Using macOS securely is a lot easier, in my experience. There are ways to use Windows securely, but it's a lot harder, and (I think) that makes it not worth the trouble. Using macOS securely is a lot closer to just using macOS.

That said, for any OS, it goes without saying to only install trusted software. It's possible to cleanse both OSes fully of software you no longer want, but on Windows it always feels like surgery, whereas 90% of cleaning something from macOS is dragging the application bundle to the Trash.

jerf• 4 years ago

The most basic way to analyze security is to look at it in terms of cost to penetrate security versus the benefit to the attacker. Things are not a binary "secure" or "not secure", they are secure when the expected costs are greater than the benefits, preferably by a comfortable margin.

The problem is that there are some resources where the benefit to the attacker goes up faster than the company can afford to cover, and the company has to cover all the attack avenues. You can imagine that the value of Slack's source code can be high to certain people, and it can be difficult to completely seal off something like that when source code by its nature pretty much has to be distributed to hundreds or thousands of people. (At least overall; any given source may not be that distributed, but there will be thousands with some sort of access.)

Major corporations basically have to act as if source compromise is inevitable. Best practices obviously include partitioning who can have access to what (perhaps the best practical argument against a one-company mono-repo, if de facto it has to be broken up by access permissions anyhow) and not including anything in the repo that is directly security catastrophic if it gets out (security secrets mostly), but this is still limiting blast radius rather than "solving" the problem.

It's a perfect storm of being extremely expensive to cover, in terms of money, internal process friction, and having a lot of heterogeneous vectors you need to cover, and for certain companies, being very high value to a number of different kinds of entities.

Being in GitHub specifically would only matter if GitHub was itself compromised. GitHub can not do very much about legit credentials being stolen through completely non-GitHub related means. (Most things, if not all things, that might leap to your mind will also block legitimate usage quite a bit. No matter what crazy thing you can imagine, someone's probably doing it somewhere and it's probably mission critical.)

I expect there have been a great deal more source code compromises than we've heard about.

openplatypus2• 4 years ago

What's the most interesting in this whole write-up is this: > Security update hidden from search engines

Slack is selectively and deliberately limiting public access (discoverability) to the security breach announcements.

have_faith• 4 years ago

marked with 'noindex'

How do they even provide a plausible excuse for this? very bad look.

r4vik• 4 years ago

it's totally plausible. and best practice.

If you have a localised blog for multiple regions you will probably have your main blog set up to be indexed and your regional blogs set up to be no-indexed so that Google isn't indexing the same stuff twice.

bradhe• 4 years ago

Slack aka salesforce

chasd00• 4 years ago

Right. Salesforce does a good job of keeping major security incidents out of the news. I suspect that practice also applies to all their acquisitions.

muglug3• 4 years ago

I work for Slack, opinions my own.

As far as I can tell, Slack's not trying to hide this news.

Slack communicated about the breach here: https://slack.com/blog/news/slack-security-update

There is no `noindex` tag on that page (check the source).

There is a `noindex` tag on an alternate "en-gb" version of that page: https://slack.com/intl/en-gb/blog/news/slack-security-update

This is to be expected — it's presumably SEO equivalent to specifying a canonical URL.

oxfordmale2• 4 years ago

Slack communicated the incident on the 31st of December. That day is typically used by politicians to bury bad news, as attention to news is quite low on that day. It might be coincidental, however, the little cynic voice in me is left wondering.

openplatypus2• 4 years ago

That article is NOT listed on news site https://slack.com/intl/en-gb/blog/news Yes, your direct link works, but the article is not listed.

richardwhiuk6• 4 years ago

That's not the standard SEO way of flagging a canonical URL.

wut425• 4 years ago

Very doubtful on SEO and/or alternate content. BleepingComputer also notes the August 2022 notice, which is 'noindex' on all versions.

This is most definitely to burry the notices.

Suvitruf• 4 years ago

No downloaded repositories contained customer data, means to access customer data, or Slack’s primary codebase.

tedk-42• 4 years ago

Doesn't sound too bad to be honest

BoardsOfCanada• 4 years ago

I hope microsoft can have a look and improve teams.

gchamonlive• 4 years ago

I wish, but teams is bad not because Microsoft isn't capable of fixing it.

It is bad because regardless of how infuriating and frustrating the user experience is, people are still gonna be forced to use it under the massive pressure of Microsofts weight.

blackoil• 4 years ago

There was a project to completely rewrite Teams UI. no idea where is that.

brynjolf• 4 years ago

Teams is written in Angularjs aka Angular 1 [0]

They rewrote parts of it to React when they made the weird half client for Windows 11. Not sure why those two clients haven't merged yet.

https://news.ycombinator.com/item?id=27633756

modo_mario• 4 years ago

I neither expect it nor hope so. I still remember when they bought Skype. Said they'd keep the Linux support to stem community worries then dropped it like a brick about a year after.

schnable2• 4 years ago

The model of teams is fundamentally different than Slack, "fixing" it to compete directly would be a big product shift.

nickjj3• 4 years ago

They say no customer data was compromised and are investigating "potential impact" to customers.

The potential impact here is an attacker now has access to some of their code which could let them find and take advantage of vulnerabilities resulting in customer data being accessed.

Technically "potential impact" is correct but I think companies often underplay how severe a source code leak is. It's the exact blueprint of how their app is built.

moonchrome• 4 years ago

OK so does this mean that OSS is severely vulnerable by default ?

thot_experiment• 4 years ago

I don't know, I guess if one known malicious actor having access to a piece of information is the same as everyone having access to it, then yes.

nickjj3• 4 years ago

OK so does this mean that OSS is vulnerable by default ?

No, but I don't think that's a fair comparison. An open source tool or app is open source by choice so they have an advantage of knowing what they're getting into.

In Slack's case maybe they have a bunch of undocumented APIs which are publicly accessible and now with access to the source code you know what they are, and when you hit them they result in customer data being returned. It runs in their production environment off the live site so it doesn't involve anything crazy like you needing VPN access to their DB to get production data.

That's just 1 basic example of what could happen when a private code base becomes public due to a leak. I'd like to think an open source site wouldn't do that because at a fundamental level the app is built in the open. Also, there's likely many sets of eyes from different folks looking at it from different angles.

Having undocumented publicly accessible API endpoints isn't an option in an open source world but a private code base could maybe get by with security through obscurity on a few things thinking "well, the code is private...".

hnbad• 4 years ago

In a way. The difference is that if you develop something with public source code, you're vulnerable from day zero of development. That means severe vulnerabilities are more likely to surface before the product is mature enough for those vulnerabilities to represent massive risks. OSS applications benefit from survivorship bias: the code that's there probably contains fewer vulnerabilities because any vulnerabilities that were there have likely been exploited already, allowing them to be detected and fixed.

Meanwhile if you allow a closed code base to grow and mature for years and then expose it to the prying eyes of the public, it might contain a few bangers that would have been caught much earlier if the code had been developed in public.

Of course this assumes that vulnerabilities get fixed and dev teams learn to avoid them once they've found them. This somewhat breaks down in typical plugin-heavy open source stacks where any random plugin might be developed by a 13 year old in their bedroom and no longer maintained and your flagship product now depends on it because nobody noticed its glaring security flaws. Though this can be managed and avoided somewhat by being more deliberate when picking out plugins.

EDIT: A really straightforward example is committing unencrypted secrets (API keys, passwords, whatever). In an OSS product this will likely be exploited almost instantly but this creates cultural awareness to not do that (either by learning from mistakes or by being told horror stories about those mistakes). In a closed source product this could easily go on for years with no consequences until someone exposes the code to the public.

hoseja• 4 years ago

Do they not have them anymore?

captn3m0• 4 years ago

Timeline matches with the Travis breach. So far likely impacted:

1. Slack

2. Okta (https://news.ycombinator.com/item?id=34081154)

3. Coa (https://github.com/veged/coa/issues/99#issuecomment-96169688...)

dijit2• 4 years ago

Add yesterdays announcement about CircleCI[0] and you have even more plausible options.

[0] https://circleci.com/blog/january-4-2023-security-alert/

rad_gruchalski2• 4 years ago

Wow wow wow. It would have been great for CircleCI to actually send an email with this info. I have a couple of OSS repos with them…

berniedurfee• 4 years ago

CircleCI uses Travis?

swyx• 4 years ago

are you implying that travis may have caused compromise of customer codebases? wow

nightpool5• 4 years ago

Yes, this is well known, they had a major breach earlier in 2022 that apparently leaked all of their Github tokens. This happened almost simultaneously to the Heroku leak and is thought to be connected to it. From OP's link, it sounds like they had another one recently—maybe another group re-using the first vulnerability?

wut425• 4 years ago

I was thinking of the Pytorch supply-chain issue as well: https://www.bleepingcomputer.com/news/security/pytorch-discl...

I can see someone working at Slack try a nightly, they surely work with ML.

blkhp19• 4 years ago

So, where can these be found? I’d like to dig through the code.

halukakin• 4 years ago

I hope github would start allowing all users to use ip whitelisting features. I understand it is a sell point for enterprise accounts, but tokens/passwords are not secure enough in today's environment.

rvz• 4 years ago

Like I said before [0][1], a reminder that private repositories are not private on GitHub.

A great time to self-host.

[0] https://news.ycombinator.com/item?id=34232347

[1] https://news.ycombinator.com/item?id=23102942

mytailorisrich2• 4 years ago

I don't understand why some companies host their code externally on Github.

For a software company the code repo is the #1 asset.

cj3• 4 years ago

Hate to nitpick, but is code really the #1 asset of most software companies?

if AWS/Amazon's entire source code were leaked tomorrow, how would competitors use it to their advantage?

Would it help Lyft at all if they gained access to Uber's source code? Probably not.

Is some startup going to be able to take Slack's source code and use it to build a competing service? Probably not, it would be easier to write the code from scratch and reverse engineer features.

Source code is definitely a company asset, but most companies wouldn't be incredibly damaged or threatened in any meaningful way if their source code were exposed... unless said source code exposes security flaws that are used for some attack on the service. But security through obscurity (e.g. keeping insecure code secret) is not a valid way to keep things secure.

londons_explore2• 4 years ago

There are instances of an overseas company stealing a large amount of code and just running it as-is.

As long as you do it outside the jurisdiction of the USA, you'll probably get away with it.

I have seen two instances of this in my professional career (can't talk about either I'm afraid!), but in both cases it had a pretty big impact on the original owner of the code, and in neither case did the company manage to get any compensation.

mytailorisrich2• 4 years ago

Leak of code can be very damaging, not just because it might reveal vulnerabilities but also because it will reveal IP and will allow all sorts of 'IP theft'. Then, losing the code altogether is the death of the company.

umanwizard2• 4 years ago

How do you explain successful companies whose source code is not only on GitHub, but publicly visible? (E.g. Cockroach)

shanebellone2• 4 years ago

Does it actually matter? I might be overlooking something but...

Slack is more than its code. The product is really the aggregate of their engineer's knowledge and internal processes. It's not practical to steal code and build a business or spinoff product around it.

The only legitimate threat seems to be the potential for exploitation. I suppose this might also threaten any backend improvements (economic leverage) they made with proprietary algorithms.

nightpool5• 4 years ago

It's not even their main codebase—if you read closely, it says it was only private code repos stored on Github.com, and that their main repo wasn't.

shanebellone2• 4 years ago

I understand. Just posing a question for discussion.

muglug3• 4 years ago

I work for Slack, opinions my own.

IMO the headline should be updated — much less than 1% of Slack’s private code can be accessed via github.com.

jbotdev• 4 years ago

The article did mention that this breach did not include the “primary codebase”. Where is the rest of the code hosted? I’m guessing some private Git server (e.g. “on-prem” GitHub/GitLab Enterprise)?

nightpool5• 4 years ago

I would be surprised if Slack disclosed publicly exactly which source code provider they used, but something hosted like Github EE sounds like a very reasonable assumption given their response. Maybe someone was given a goal to migrate their main source code repo to a shared Salesforce instance for security or cross team integration reasons, and everything left on Github.com was just the stuff too unimportant to migrate. If so, seems like it payed off for them in this case.

oxfordmale2• 4 years ago

If you don't communicate, you lose the narrative. As the article highlights, it seems Slack has gone out of its way to exclude this incident report from appearing in Google searches. As a result this article will likely appear first.

It is also worth nothing that other major breaches (LastPass) were preceded by source code breaches. The corresponding incident reports also included re-assurances that there was no impact on customer safety. The Slack incident report doesn't specify what type of Github repos were accessed, so it is hard to judge if any sensitive code has been leaked.

From the recent LastPass security incident report:

Based on our investigation to date, we have learned that an unknown threat actor accessed a cloud-based storage environment leveraging information obtained from the incident we previously disclosed in August of 2022 (source code breach)

muglug3• 4 years ago

Slack communicated about the breach here: https://slack.com/blog/news/slack-security-update

There is no `noindex` tag on that page (check the source).

There is a `noindex` tag on an alternate "en-gb" version of that page: https://slack.com/intl/en-gb/blog/news/slack-security-update

This is to be expected — it's presumably SEO equivalent to specifying a canonical URL.

prepend• 4 years ago

The Slack incident report doesn't specify what type of Github repos were accessed, so it is hard to judge if any sensitive code has been leaked.

This is so important. Saying “less than 1% of code” is not very useful as a passwords or config repo being leaked may only be a few bytes. It’s also not customer data. But it’s extremely important.

I think the fact that they are being not forthcoming means they are clueless, or something really bad happened and they are weaseling and hoping nothing bad happens (spoiler: it will).

janosdebugs• 4 years ago

It would be so nice if GitHub deprecated their one key takes all and would let us create per-org personal access tokens. Also, organizations having power over these access tokens.

CubsFan1060• 4 years ago

They do this now. I'll find a link. Link below. Organizations can choose to allow these, require approval before they are valid, and deny the use of traditional tokens.

They are actually per repo.

https://github.blog/2022-10-18-introducing-fine-grained-pers...

jrochkind1• 4 years ago

Note "Public Beta".

Seems like a good step forward; I'm still somewhat shocked it took this long, people have been pointing out the security issues for at least 5 years.

ergonaught• 4 years ago

Offtopic, but, I made the mistake of trying to read this on my mobile phone this morning, where I don't use adblockers per se. Served at least 6 ads, even ignoring the nonsense at the bottom of the page. So annoying and disruptive that I didn't manage to read the story, and will find the details somewhere else.

We really needed a reaction against this nonsense which wasn't just adblockers.

Bummer.

idonotknowwhy• 4 years ago

This is why Firefox is my default browser on Android and I use Ublock Origin.

Lack of real Firefox on iOS is the absolute deal breaker for me switching to iPhone

acdha• 4 years ago

Firefox Focus works as a Safari content blocker: https://www.mozilla.org/en-US/firefox/browsers/mobile/focus/

It's quite effective at defanging the annoying ads without breaking things I care about.

IgorPartola2• 4 years ago

Stolen means Slack no longer has something that the thief has, right?

gkedzierski• 4 years ago

By your definition piracy would not be a theft.

IgorPartola2• 4 years ago

Correct. It isn’t. It is copyright infringement, and there is a lot of case law to support this.

danwee• 4 years ago

We've migrated from Slack to Mattermost some time ago. Best decision ever. Not sure why companies keep using Slack nowadays. It's like those companies many years ago using HipChat instead of Slack. Slack's is the HipChat of today.

tourist2d• 4 years ago

What does Mattermost offer which Slack doesn't?

paulryanrogers• 4 years ago

Open source, can be self hosted, you can secure it yourself and how you want (for better or worse)

michaelmior2• 4 years ago

I think part of the problem is that there are a lot of integrations from Slack that aren't available on Mattermost. You can certainly make the argument that those integrations won't be developed until more users abandon Slack for Mattermost, but for some it's not worth the loss of functionality.

dx0342• 4 years ago

Self-hosted Mattermost and Gitlab can be done reliably at very low cost, had better uptime than their cloud services for us and would've likely prevented leaks like these. At least you can add some layers of security for self-hosted services so that stolen tokens aren't enough to steal code or messages.

splittingTimes• 4 years ago

Do you have write up or guide somewhere on how to specifically set this up, configure and add security?

gkoberger• 4 years ago

Could you expand? I think that's incredibly unfair... Slack changed how almost every business communicates. Mattermost is almost a 1:1 open source copy, and is missing most of the ecosystem (which is Slack's real value).

Sure, for some companies, it's really important to own their data completely. I get that (although I personally trust Salesforce more than most one-off companies when it comes to security). But I don't think it's fair to say that Slack is the HipChat of today at all... it's like iOS vs Android, both are just different flavors of a really good product.

ianpurton• 4 years ago

Slack could open source their code anyway. Slacks competitive advantage is their network of users which is difficult to copy.

vaseem• 4 years ago

Source code is stolen to explore vulnerabilities and develop zero day exploits.

More in this podcast https://darknetdiaries.com/episode/19/

michaelmior2• 4 years ago

Arguably vulnerabilities would be more likely to be found and disclosed if the code was open source.

phpisthebest• 4 years ago

That old classic chestnut of lies by Closed Source companies intent on making CIOs believe Open Source is not secure...

saagarjha2• 4 years ago

That’s not typically how these exploits are developed.

ltbarcly32• 4 years ago

Nope, its a bunch of little communities that are centrally managed and can force their users to a new platform easily. Network effects are minimal, sunk cost of integrations is a greater barrier to moving off platform but that is pretty weak as well.

schnable2• 4 years ago

Agree with the first half, but the integrations is a moat for Slack. First, a comparable integration needs to exist or be created on the new platform. Second, in largeer organizations, you end up with many vested interests (individual teams) that have important integrations and the cost to migrate them takes away from other work they need to do, and creates a chorus of voices against a migration. On top of all that, employees generally like Slack.

umanwizard2• 4 years ago

Oh good, maybe someone will finally fix their many annoying bugs

bjd2385• 4 years ago

This is exactly why secrets should not be stashed in any Git repository. Granted, I'm not sure they're much safer in other managed services dedicated to protecting our secrets, with the news of several breaches as of late :/

j45• 4 years ago

It remains surprising to see such private and valuable codebases hosted in the cloud in an external parties service.

The cloud remains someone else’s shared computer.

marban• 4 years ago

Streisand effect anyone?

Slack's private GitHub code repositories stolen over holidays

Related Stories