Mintlify GitHub read/write token leak

https://mintlify.com/blog/incident-march-13

by 11217mackem • 2 years ago

130 41 2 years ago

mintlify.com

11217mackem• 2 years ago

I'd been interested in Mintlify and their documentation platform for a while.

I saw this pop up based on this Reddit thread and on Twitter as well:

https://www.reddit.com/r/ExperiencedDevs/comments/1bf7eqa/ni...

This seems serious? Is this really serious?

Why would they need to save these tokens in the first place?

candiddevmike2• 2 years ago

If they're a GitHub App, they receive a token to authenticate into your account/org when you grant them access/enable the app.

Everyone should audit their GitHub Apps periodically/avoid using them if at all possible IMO. Most of these integrations are just a convenience for adding webhooks, which you can do yourself without compromising security. Always prefer "outbound" integrations.

akaike• 2 years ago

It’s super serious …

geraldwhen• 2 years ago

SOC 2 compliant. Tells you exactly how useful SOC compliance is.

d4mi3n• 2 years ago

All SOC tells you is that there is a process being (mostly) followed. Always review the reports for your vendors and follow up on any findings or gaps that show up.

It's surprisingly common for places to be SOC compliant, yet their latest report has half a dozen or more gaps/findings.

Rapzid• 2 years ago

Auditors don't get paid to fail their customers ;) I think many people would be surprised just how.. Accommodating the process can be.

candiddevmike2• 2 years ago

Broadly speaking, being SOC 2 "compliant" isn't really a thing. They just had a SOC 2 audit done, you have to read the report to discover how "compliant" they are.

SOC 2 is useful as setting a baseline for how a business and IT org should be ran, assuming it's followed...

ametrau• 2 years ago

An artificial barrier to entry. What a racket.

zx8080• 2 years ago

This opens the door for supply chain attack. Removing mintlify and auditing other integrations ASAP should be considered as #1 priority.

brycelarkin• 2 years ago

I was a Mintlify customer 6 months ago and found their onboarding strange. Their Github app asked for permissions to access all my repos and there was no way to get around that. I just wanted to give them access to my docs repo not my app repo. Ended up having to dig through their settings page and revoking that permission after onboarding.

For customer OAuth tokens, I believe you should NEVER store the access token in the database. Only store the refresh token in the database. When you need api access, get a new access token with the refresh token and client secret.

This prevents incidents like the above as the attacker would also need your client secret, which is ideally not in your database.

e40• 2 years ago

Too generous permissions are something that often makes me not install software. Zoom apps for example. Slack apps another example. Mostly browser extensions.

PierceJoy• 2 years ago

The browser extension permission model is so bad. It’s prevented me from using 99% of extensions I was interested in.

junon3• 2 years ago

This is pretty typical of GitHub, unfortunately. They have terrible scoping and don't give developers the option to even give users that flexibility. It's been like this for over a decade, I don't think GitHub will ever learn.

oefrha3• 2 years ago

Wrong. “GitHub Apps” always (?) had fine-grained repo-scoped permissions, it’s the “OAuth apps” that are stuck with coarse-grained permissions. So developers do have the option, it’s even the preferred option.[1]

In general, GitHub Apps are preferred over OAuth apps.

[1] https://docs.github.com/en/apps/oauth-apps/building-oauth-ap...

Ruepler6• 2 years ago

Exactly. You'd need to implement standard authentication mechanisms, preventing these vectors of attack from existing in the first place.

They are "in the process" and "looking" to do that. As of now if I am not mistaken the main issue still persists and any other new security vulnerability will possibly give full access to tokens again.

Correct me if I am wrong please but I don't see anything in the comment or Blog post saying otherwise.

and are also in the process of completely deprecating the admin tokens for a more secure internal authentication procedure. Not to mention, we're also looking to fully deprecate the need of the GitHub OAuth tokens entirely in the coming weeks.

https://www.reddit.com/r/cscareerquestions/comments/1bh22bq/...

nfriedly2• 2 years ago

Mintlify sponsors one of my open source projects, express-rate-limit, and we use their service to host the docs for it. So I'm probably biased, but I've honestly been pretty happy with it overall - I'm thinking about using it for some of my other projects.

I think the best thing I can say about it is this: Mintlify helps us write better docs. Before Mintlify, we had a humongous readme plus a few one-off articles in the wiki. After switching the Mintlify, we started organizing things better, beginning with a less overwhelming readme.

I found out about this hack when they sent me an email yesterday. I looked through the git history (edit: and github activity), but nothing seems awry.

It sucks, but they're a pretty small business, and I think they're handling it reasonably well.

richbell• 2 years ago

I looked through the git history, but nothing seems awry.

Git history or GitHub event history? You can easily push a malicious commit forged to have the same author and short hash.

nfriedly2• 2 years ago

Good point. I only looked through the git history initially, but I've now looked through both. Same conclusion.

elijahbenizzy• 2 years ago

I really like mintlify and I found this writeup to be straightforward and responsible. Shit happens.

xyst• 2 years ago

So a user (or white hat?) alerts them to security issue on their endpoint. Email comes in at 4:55 PDT on a Friday.

[first red flag] No internal monitoring to check for unauthorized access

Poor engineer(s) tasked with issue on a Friday at end of work day manually rakes logs in their app. After ~1 hr of searching, discovers unauthorized device using leaked credentials

[second red flag] allows any device to hook into their critical infrastructure and access apis that have potential to expose PII and possibly move laterally within the org or customers.

Then poor engineers tasked with painstaking task to rotate all of the tokens throughout the night. Bye bye family time, any planned events (game tickets?). I hate this company already and I have no clue wtf they do.

[third red flag] why the fuck are they storing user tokens in a database. This is apparently a “SOC 2” certified application/company. Shows how much that is worth here.

[edit: fourth red flag] no indication of how long the unauthorized credentials have been used and what customers have been impacted. Very very piss poor logging or purposely omitting due to potential litigation issues

Probably missed a few but this is bad.

The “S” in Mintify clearly stands for security.

cmsparks• 2 years ago

Oof not a fun incident, this is my nightmare as someone who works on this type of stuff.

As an aside, GitHub’s security model for apps/integrations is extremely puzzling to reason about and enables a lot of foot guns. Add the fact that it’s very obtuse to audit integrations (especially within an organization) makes them pretty scary to use sometimes.

nostrowski• 2 years ago

Unfortunate, but impressed by how the Mintlify team is handling it.

darknavi• 2 years ago

Even something as simple as bolding the message about customer repositories being accessed is nice. Not trying to bury the lead.

marcinzm• 2 years ago

Two weeks to notify customers with many first finding out on social media is impressive?

Ruepler6• 2 years ago

They are claming that they resolved the vulnerability that caused the token leak but don't mention it. Doesn't exactly seem transparent to me or like handling it well.

I was contracting for them last year and tried, among other things to build an actual engineering culture that prevents and fixes issues that accumulate to catastrophic incidents like this.

They generally prefer to "ship fast".

I informed them very thoroughly again on January 13th (3+ months after they terminated me for "cultural differences"), because I was worried of this exact nightmare scenario happening very soon.

The reason for this was that they open sourced a package that let's an attacker easily practice and test locally in like a minute.

MDX exposes to Cross site Scripting easily. I assume this is the "fixed vulnerability" they are talking about, just to be transparent.

ethanwillis• 2 years ago

I reported security issues to Mintlify back when they used Discord for support. Gave recreation steps. Never got any feedback that looked to me like they were looking into it.

I reported similar issues to what's described in May 2023.

hanyiwang2• 2 years ago

Hey Ethan! Han from Mintlify here.

It's been awhile I hope you're doing well!

The vulnerability issue that you reported, if I recall correctly, was relating to server-forwarding attacks via our API playground. We've addressed that issue around 8 months ago, and unfortunately, it was not the root of this incident.

Please correct me if I am wrong though!

perpil• 2 years ago

"we cannot confirm that no other such instances occurred". I'd like to understand this more. Do they 1. not have a fingerprint of what the exploit looked like because it was indistinguishable from customer behavior? 2. Not have logging necessary to audit this? Regardless, with the added monitoring to detect suspicious activity, do they also have the logging to see everything a bad actor did if there is a new exploit?

theK• 2 years ago

Did I miss it or does the announcement completely evade explaining HOW the tokens actually leaked?

Ruepler6• 2 years ago

Yea it does. It most likely is related to MDX documents beeing susceptible to XSS attacks. Having worked at mintlify last year I can tell you that this is not surprising at all and I've been warning them extensively of sth like this happening.

hanyiwang2• 2 years ago

This is Han, I'm one of the co-founders from Mintlify.

For the security of our customers from the publicity we were expecting from the announcement, we've decided to leave out the technical details of the breach in the blog post.

The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information.

Our initial patch upon discovering the incident fixed the response of the vulnerable endpoint, but we have since also implemented a sweeping number of security provisions and are also in the process of completely deprecating the admin tokens and GitHub OAuth entirely to prevent an incident like this from ever happening again.

Ruepler6• 2 years ago

"we've decided to leave out the technical details of the breach in the blog post"
"Our dedication to transparency, security, and the trust you place in us remains unwavering."

You are contradicting yourself here.

"The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information."

Why would you leave that out? Seems like it is vital information.

ummonk• 2 years ago

For the security of our customers from the publicity we were expecting from the announcement, we've decided to leave out the technical details of the breach in the blog post.

This doesn't exactly inspire confidence that your service is now secure.

rahulrs• 2 years ago

Out of curiosity, what happened to the customer whose private repo was breached?

Mintlify GitHub read/write token leak

Related Stories