1 bug, $50k in bounties, a Zendesk backdoor

https://gist.github.com/hackermondev/68ec8ed145fcee49d2f5e2b9d2cf2e52

by mmsc • 2 years ago

1637 413 2 years ago

gist.github.com

pavlov2• 2 years ago

The edited title on HN is incomprehensible.

The original is:

”1 bug, $50,000+ in bounties, how Zendesk intentionally left a backdoor in hundreds of Fortune 500 companies”

A better edit might be something like:

“The $50k bug where Zendesk backdoored Fortune 500 companies”

mmsc11• 2 years ago

It was supposed to be

1 bug, 50k:

I don't know why the "1" got dropped.

some_furry• 2 years ago

HN mangles submission titles.

If you submit "Why I care" it'll decide that you meant 'I care".

If you submit "10 More Secrets in Pokemon" it'll decide you meamt "More Secrets in Pokemon".

Conversely, there's an entire cottage industry focused on writing attention-catching headlines, which results in patterns like what HN mangles.

If it's annoying, OP can edit immediately after submitting to overwrite the mangled title with the correct one.

gouggoug3• 2 years ago

That title is also completely misleading because the author did not in fact get paid.

50k corresponds to the money they made with unrelated bug bounties.

I wish they would fix the title so that it properly calls out zendesk refused to pay for a serious bug.

masfuerte• 2 years ago

I understood it to mean that he received $50K from enterprises using Zendesk who were vulnerable to this bug, but it's not entirely clear.

cjbprime3• 2 years ago

"Unrelated" doesn't sound right. Zendesk refused to pay for the vulnerability, so the researcher used it against downstream customers of Zendesk, who did pay the researcher for the impact of that Zendesk vulnerability against their own company.

bsuvc2• 2 years ago

It sounds like the author got stiffed by Zendesk on this bug, $0 due to email spoofing being out of scope.

The $50k was from other bug bounties he was awarded on hackerone.

It's too bad Zendesk basically said "thanks" but then refused to pay anything. That's a good way to get people not to bother with your big bounty program. It is often better to build goodwill than to be a stickler for rules and technicalities.

Side note: I'm not too surprised, as I had one of the worst experiences ever interviewing with Zendesk a few years back. I have never come away from an interview hating a company, except for Zendesk.

paulpauper6• 2 years ago

That is why a black market exists for this stuff.

c0balt• 2 years ago

The black market also exists because the potential payout for serious 0days by official programs is almost always less than what a third-party adversary will pay (if the target(s) for them are worth it).

exceptione2• 2 years ago

If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug, but HackerOne that did not escalate to Zendesk that they should reconsider the exclusion ground in this case.

As an aside, I wonder if those bounties in general reflect the real value of those bugs. The economic damage could be way higher, given that people share logins in support tickets. I would have expected that the price on the black market for these kind of bugs are several figures larger.

richbell5• 2 years ago

If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug, but HackerOne that did not escalate to Zendesk that they should reconsider the exclusion ground in this case.

Correct, the replies seem to have come from H1 triage and H1 mediation staff.

They often miss the mark like this. I opened a H1 account to report that I'd found privileged access tokens for a company's GitHub org. H1 triage refused to notify the company because they didn't think it was a security issue and ignored my messages.

chabons• 2 years ago

The author specifically stated: "Realizing this, I asked for the report to be forwarded to an actual Zendesk staff member for review", before getting another reply for H1. I read this as they escalated it to Zendesk directly, who directed it back to HackerOne.

bigiain5• 2 years ago

If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug

While it's unclear at which stage Zendesk became involved, in the "aftermath" section it's clear they knew of the H1 report, since they responded there. And later on the post says:

"Despite fixing the issue, Zendesk ultimately chose not to award a bounty for my report. Their reasoning? I had broken HackerOne's disclosure guidelines by sharing the vulnerability with affected companies."

The best care scenario as I see it is that Zendesk has a problem they need to fix with their H1 triage process and/or their in and out of scope rules there. And _none_ of that is the researcher's problem.

The worst (and in my opinion most likely) scenario, is that Zendesk did get notified when the researcher asked H1 to escalate their badly triaged denial to Zendesk for review, and Zendesk chose to deny any bounty and tried to hide their vulnerability.

As an aside, I wonder if those bounties in general reflect the real value of those bugs. The economic damage could be way higher, given that people share logins in support tickets.

I think it's way worse than that, since internal teams often share logins/secrets/API keys (and details of architecture and networking that a smart blackhat would _love_ to have access to) in thei supposedly "internal" Slack channels. I think the fact that non Zendesk "affected companies" paid out $50k sets that as the absolute lower bound of "the real value of those bugs. And it's _obvious_ that the researcher didn't contact _every_ vulnerable Slack-using organisation. I wonder how much more he could have made by disclosing this to 10 or 100 times as many Slack using organisations, and delaying/stalling revealing his exploit POC to Zendesk while that money kept rolling in?

I'll be interested to see if HackerOne react to this, to avoid the next researcher going for this "second level" of bug bounty payouts by not bothering with H1 or the vulnerable company, and instead disclosing to companies affected by the vulnerability instead of the companies with the vulnerability? It's kinda well known that H1 buy bounties are relatively small, compared to the effort required to craft a tricky POC. But people disclose there anyway, presumably party out of ethical concerns and partly for the reputation boost. But now we know you can probably get an order of magnitude more money by approaching 3rd party affected companies instead of cheapskate or outright abusive companies with H1 bounties that they choose to downvalue and not pay out on.

mkagenius• 2 years ago

Hackerone staffs are not that good. They usually mark anything from a non famous person as a duplicate (even if it differs in nuances, which eventually lead to much more impact) or straight out of scope.

I think it's just laziness. Plus they hire previous famous reporter as the people triaging the reports, those famous people know other famous people first hand, they usually think "hmm, unknown guy, must have ran a script and submitted this"

I have stopped reporting stuff since last 5 years due to the frustration. And it seems the situation is still the same even after so many years.

renewiltord• 2 years ago

$0 due to email spoofing being out of scope.

Strictly, $0 because he disclosed to customers. But he only disclosed to customers since Zendesk said it was out of scope.

jeroenhd• 2 years ago

HackerOne declared the issue out of scope so I don't see why disclosure would make a difference here. Had this person not notified different companies, they still wouldn't get a dime from HackerOne.

Bad showings all around, for both HackerOne and Zendesk.

gouggoug3• 2 years ago

Side note: I'm not too surprised, as I had one of the worst experiences ever interviewing with Zendesk a few years back. I have never come away from an interview hating a company, except for Zendesk.

Same thing happened to me years ago. Interviewed with them and it was the worst “screening” experience I ever had. After getting a rejection email, I thanked them for their time and said I had feedback about the interview should they want to hear it. They said yes, please.

Sent my feedback, never heard from them again.

swoorup• 2 years ago

Same it was time-wasting interview experience. They seem interested and not interested at the same time. They pinged me for a different role after passing me up for the first role, but didn't get any response later..

layer87• 2 years ago

due to email spoofing being out of scope.

I believe their logic was that only the domain owner can adequately prevent email spoofing by proper SPF/DMARC configuration, and that it’s the customers’ fault if they don’t do that. Which isn’t entirely wrong.

radlad6• 2 years ago

Are Google and Apple not doing proper SPF/DMARC/DKIM? I think they probably are - but this attack worked anyway.

Zendesk wasn't validating the email senders.

nightpool2• 2 years ago

Right, but I would be really shocked if Zendesk's internal email handler was doing any SPF/DKIM/DMARC validation at all. So even if a domain has DMARC set up, Zendesk is probably ignoring it. Which is probably pretty reasonable given how rare DMARC reject/quarantine has been historically

UncleMeat• 2 years ago

In a past life I was involved in a bug bounty program. I don't think the reasoning is as detailed.

When you stand up a bug bounty program you get a ton of "I opened developer tools, edited the js on your page, and now the page does something bad" submissions. "I can spoof some email headers and send an email to myself that looks like it is coming from you" isn't something I've specifically seen due to some weird details about my bounty program but it is something I would absolutely expect for many programs to see.

So you need a mechanism to reject this stuff. But if that mechanism is just "triage says this is dumb" you get problems. People scream at you for having their nonsense bug rejected. People submit dozens of very slightly altered "bugs" to try to say "you rejected the last one for reason X but this one does Y." So you create a general policy: anything involving email spoofing is out of scope.

So then a real bug ends up in front of the triage person. They are tired and busy and look at the report and see "oh this relies on email spoofing, close as out of scope." Sucks.

I think that Zendesk's follow up here is crap. They shouldn't be criticizing the author for writing about this bug. But I do very much understand how things end up with a $0 payout for the initial report.

belter• 2 years ago

Seems a pattern: https://www.reddit.com/r/sales/comments/1eck30a/terrible_zen...

pm90• 2 years ago

I too had the worst interview experience with zendesk. The people I talked to were pretty senior folks too. They just seem to have a very petty and toxic work culture.

barbs• 2 years ago

Mind giving details abut the interview? Must've been pretty bad!

bigiain5• 2 years ago

That's a good way to get people not to bother with your big bounty program.

And possibly to have blackhats to start looking more closely, since they now know both 1) that whitehats are likely to be focusing elsewhere leaving more available un-reviewed attack surface, and 2) that Zendesk appears to be the sort of company who'll ignore and/or hide known vulnerabilities, giving exploits a much longer effective working time.

If "the bad guys" discovered this (or if it had been discovered by a less ethically developed 15 year old who'd boasted about it in some Discord or hacker channel) I wonder just how many companies would have had interlopers in their Slack channels harvesting social engineering intelligence or even passwords/secrets/API keys freely shared in Slack channels? And I wonder how many other widely (or even narrowly) used 3rd party SaaS platforms can be exploited via Zendesk in exactly the same way. Pretty much any service that uses the email domain to "prove" someone works for a particular company and then grants them some level of access based on that would be vulnerable to having ZenDesk leak email confirmations to anybody who knows this bug.

Hell, I suspect it'd work to harvest password reset tokens too. That could give you account takeover for anything not using 2FA (which is, to a first approximation over the whole internet, everything).

Avamander2• 2 years ago

This is a common problem with HackerOne and the likes. It's absolutely awful for anything even a tiny bit more unique or rare.

portaouflop4• 2 years ago

Blame beg bounty hunters for this

ldoughty• 2 years ago

I hate that Zendesk refused to pay out for this bug. The author made a good faith effort to report it. The author also tried to escalate it.

After they decided not to work on it, they later came back and asked him for more information and treat it like a bug...

Author should have gotten a reward. Did everything right if Zendesk claims it's not a in scope bug.

paulpauper6• 2 years ago

That is how it works. Do nothing so that the researcher breaks the rules innadvetedly as an excuse to not pay, and then fix the problem.

m11a3• 2 years ago

Doubtful. It's probably just incompetence, rather than malice.

The incident almost certainly cost Zendesk more in (according to the gist) lost contracts and reputational damage than it would've cost to pay the security researcher a bounty.

paulpauper6• 2 years ago

lol $0 Companies cutting corners on security also skimp on paying bounties. No surprise there. The only possible exception are crypto bounties. Those are much bigger and a greater willingness to pay due to the stakes.

napsterbr• 2 years ago

It's unbelievable that he received no bounty from Zendesk on this one. If it was out of scope, then surely he could have published it anywhere?

yfw• 2 years ago

It's owned by private equity. Slowly cutting costs and bleeding the brand dry

8b16380d• 2 years ago

I mean, this existed long before acquisition. Maybe the response could have been different in the past, but there is nothing to indicate that would be true.

ciabattabread• 2 years ago

A demonstration of the ol’ adage that in order to solve a problem, make it someone else’s problem.

mnau• 2 years ago

Isn't that the the usual workflow of security researchers?

maeil2• 2 years ago

A $1.3 billion revenue company being too tight to pay this after all, even on their 2nd chance, is so short-sighted it's absurd. They're putting out a huge sign saying "When you find a vuln, definitely contact all our clients because we won't be giving you a penny!".

Incredible. This must be some kind of "damaged ego" or ass-covering, as it's clearly not a rational decision.

Edit: Another user here has pointed out the reasoning

It's owned by private equity. Slowly cutting costs and bleeding the brand dry

paulpauper6• 2 years ago

If the bounty is big enough you basically need to retain a lawyer so the whole thing is done right and prevent being scammed.

gavingmiller4• 2 years ago

zendesk is 6k employees, they have general council on staff

mmsc11• 2 years ago

It all makes sense if you consider bug bounties are largely:

1) created for the purpose of either PR/marketing, or a checklist ("auditing"), 2) seen as a cheaper alternative to someone who knows anything about security - "why hire someone that actually knows anything about security when we can just pay a pittance to strangers and believe every word they say?"

The amusing and ironic thing about the second point is that by doing so, you waste time with the constant spam of people begging for bounties and reporting things that are not even bugs let alone security issues, and your attention is therefore taken away from real security benefits which could be realized elsewhere by talented staff members.

richbell5• 2 years ago

you waste time with the constant spam of people begging for bounties

A great blog post on the matter https://www.troyhunt.com/beg-bounties/

NicoJuicy3• 2 years ago

Our company has a bug bounty program:

- handled with priority, but sometimes it takes a couple of weeks for a more definite fix

- handled by the security department within the company ( to forward to relevant PO's and to follow up)

The unfortunate thing about bug bounties is that you will be hammered with crawlers that would sometimes even resemble a DDOS

j0hnyl2• 2 years ago

It's incredibly hard and resource intensive to run a bounty program, so anyone doing it for shortcuts or virtue signaling will quickly realize if they're not mature enough to run one.

ec1096856• 2 years ago

I don’t agree. Bug bounties are taken seriously by at least some companies. Where I have worked, we received very useful reports, some very severe, via HackerOne.

The company even ran special sessions where engineers and hackers were brought together to try to maximize the number of bugs found in a few week period.

It resulted in more secure software at the end and a community of excited researchers trying to make some money and fame on our behalf.

The root cause in this case seems to be that they couldn’t get by HackerOne’s triage process because Zendesk excluded email from being in scope. This seems more like incompetence than malice on both of their parts. Good that the researcher showed how foolish they were.

wil421• 2 years ago

“2) seen as a cheaper alternative to someone who knows anything about security - "why hire someone that actually knows anything about security when we can just pay a pittance to strangers and believe every word they say?"”

It doesn’t make sense, companies with less revenue aren’t the ones doing this. It’s usually the richer tech companies.

kozikow• 2 years ago

A $1.3 billion revenue company being too tight to pay this after all, even on their 2nd chance, is so short-sighted it's absurd.

I'll give an "another side" perspective. My company was much smaller. Out of 10+ "I found a vulnerability" emails I got last year, all were something like mass-produced emails generated based on an automated vulnerability scanning tool.

Investigating all of those for "is it really an issue" is more work than it seems. For many companies looking to improve security, there are higher ROI things to do than investigating all of those emails.

DaiPlusPlus• 2 years ago

We have a policy to never acknowledge unsolicited emails like that unless they follow the simple instructions set-out in our /.well-known/security.txt file (see https://en.wikipedia.org/wiki/Security.txt) - honestly all they have to do is put “I put a banana in my fridge” as the message subject (or use PGP/GPG/SMIME) and it’ll be instantly prioritised.

The logic being that any actual security-researcher with even minimal levels of competency will know to check the security.txt file and can follow basic instructions; while if any of our actual (paying) users find a security issue then they’ll go through our internal ticket site and not public e-mail anyway - so all that’s left are low-effort vuln-scanner reports - and it’s always the same non-issues like clickjacking (but only when using IE9 for some reason, even though it’s 2024 now…) or people who think their browser’s Web Inspector is a “hacking tool” that allows anyone to edit any data in our system…

And FWIW, I’ve never received a genuine security issue report with an admission of kitchen refrigeration of fruit in the 18 months we’ve had a security.txt file - it’s almost as-if qualified competent professionals don’t operate like an embarrassingly pathetic shakedown.

to11mtm3• 2 years ago

https://www.sqlite.org/cves.html provides an interesting perspective. While they thankfully already have a pretty low surface area from overall design/purpose/etc, You can see a decent number of vulns reported that are either 'not their fault' (i.e. wrappers/consumers) or are close enough to the other side of the airtight hatchway (oh, you had access to the database file to modify it in a malicious way, and modified it in a malicious way)[0]

[0] https://sqlite.org/forum/forumpost/53de8864ba114bf6

maeil2• 2 years ago

I understand, this is exactly why I noted "even on their 2nd chance". The initial lack of payout/meaningful response was incompetency by not understanding the severity of the vuln. Fine, happens.

But after the PoC that showed the severity in a way that anyone could understand, they still didn't pay. That's the issue. The whole investigation was done for them.

whstl2• 2 years ago

We also had this problem in my previous company a few years ago, a 20-people company, but somehow we attracted much more attention.

In one specific instance, we had 20 emails in a single month about a specific Wordpress PHP endpoint that had a vulnerability, in a separate market site in another domain. The thing is, it had already been replaced by our Wordpress contractor as part of the default install, but it was returning 200.

But being a static page didn't stop the people running scanners from asking us from money even after they were informed of the above.

The solution? Delete it altogether to return 404.

jejeyyy772• 2 years ago

it never made sense to me why these white-hat hackers don't require payment before disclosing the vulnerability

tptacek5• 2 years ago

Bug bounty people do this all the time. It's almost always a sign that your bug is something silly, like DKIM.

Later

I wrote this comment before rereading the original post and realizing that they had literally submitted a DKIM report (albeit a rare instance of a meaningful one). Just to be clear: in my original comment, I did not mean to suggest this bug was silly; only that in the world of security bug bounties, DKIM reports are universally viewed as silly.

deadliftdouche• 2 years ago

Nice writeup and fuck Zendesk, this could have done so much damage.

zer0x4d3• 2 years ago

We don't know if it hasn't to be honest. State actors and exploit sellers could have known about this bug for years and exploited it before it was found by this white hat

segmondy• 2 years ago

Once I started reading the post, I said to myself, Deja vu.

chc4• 2 years ago

Years ago I had a similar train of thought: Zendesk is used by a ton of companies for their support site, and back then HTTPOnly cookies and javascript site isolation were much less of a thing. I found an XSS bug on Zendesk, which also translates into XSS on any site that used it as `support.fortune500.com` subdomain (which was a lot). You could then use it to exploit the main site, either by leaking user cookies or reading CSRF tokens from page contents because it was a subdomain.

Zendesk gave me a tshirt but not any money for it. C'est la vie.

Thorrez5• 2 years ago

reading CSRF tokens from page contents because it was a subdomain.

Huh? I don't think you can read page contents unless the origin matches exactly (scheme://host:port).

sova• 2 years ago

Zendesk pay the man. He disclosed only after you waved it off as a nonthreat. Pay. The. Man.

Aachen6• 2 years ago

Not even really 'disclosing' but just reporting to affected parties that they've got a problem

That this hurts Zendesk is too bad, it's still the morally correct thing to do and Zendesk probably understands that, too

bsuvc2• 2 years ago

Another example of how weasley Zendesk can be:

They created a fake band called "Zendesk Alternative" just in an attempt to pollute the Google results if you search for an alternative to Zendesk.

http://zendeskalternative.com/

While not illegal, it shows the way they think, a sort of manipulative pettiness.

realusername• 2 years ago

Interesting technique but on my side I see it at the very bottom of the second page of Google so I don't think it's very effective.

ec1096856• 2 years ago

It’s from 2016, so probably lost its mojo.

patcon2• 2 years ago

Aaaaaahhh I am on a rollercoaster of customer experience. I am beyond annoyed at Zendesk for stiffing this kid, but actually kinda charmed by this quirky marketing gimmick.

But also, SECURITY culture concerns beat culture culture. Companies should def consider ditching them for this lapse and their poor form in making it right.

If Zendesk is smart, they should hop on this thread and pay this kid out while everyone is still paying attention in one place, rather than later, when everyone is quietly making business decisions in a thousand little alcoves of the internet.

Otherwise, this is the best thing to happen to the Zendesk Alternatives in a long time

talldayo• 2 years ago

but actually kinda charmed by this quirky marketing gimmick.

Calling yourself 'charmed' by an insecurity-driven marketing shtick that denies rational competition is certainly one reaction.

"The book burning was abhorrent, in principle. But the lights were so calm and the fire was so warm... I was actually kinda charmed!"

givemeethekeys• 2 years ago

Yeah, and free backstage passes to the next Zendesk Alternternative concert! /s

bryanrasmussen3• 2 years ago

but actually kinda charmed by this quirky marketing gimmick.

I'm actually pretty annoyed at the stupidity, it's the kind of thing that even a shitty search engine won't be fooled by and hey when I search for Zendesk alternatives I don't see any brand called Zendesk alternative in first few results.

I mean it's like they're too stupid to do what every other weaselly scumbag does, get some fake reviews up comparing your brand to alternatives with the reviews carefully weighted so your target customer base will be uh, I guess Zendesk is really what we want then.

Or at least buy an ad words for the search - with the words Zendesk - There is No Alternative showing up before all the alternatives.

It's ok Zendesk if you use my clever slogan because you can't think of one on your own - I'm not expecting you to pay me for it.

shepherdjerred• 2 years ago

This has strong Nathan for You vibes

Jerrrrrrry• 2 years ago

Google Sliding - Prince Andrew kinda blew the "subtly" of it with his banger about pizza-human trafficking.

MichaelZuo• 2 years ago

That is pretty bizarre.

Edit: Why don’t they seem to value their credibility?

mtlynch2• 2 years ago

I think that's pretty funny and not particularly malevolent. It's a fake alt rock band called Zendesk. It's obviously tongue-in-cheek and not going to deceive anyone.

Also, anytime I search for "X alternative" the results are all AI-generated garbage anyway, so I'd welcome something quirky and original like this in my results.

IAmGraydon• 2 years ago

It's obviously tongue-in-cheek and not going to deceive anyone.

Yes, that’s it. They paid their marketing team to do this to be funny.

BhavdeepSethi• 2 years ago

The entire keyword buying SEM operates that way too right? At least they call it out as Sponsored results though.

achairapart• 2 years ago

Also noted that the song (or "lyrics" at least) has "Open Source" in its title so they can position for the "Zendesk Open Source Alternative" long tail.

That's evil!

andai• 2 years ago

Black hat SEO.

hackernewds• 2 years ago

they have toured the world, headlining major festivals and sharing the stage with legendary acts like Sweater Head, DynoPlax, and The Banana Nuts. Now, Zendesk Alternative has begun a new chapter in their storied career. They have joined forces with Zendesk® to record an anthemic concept album of epic proportions. On the surface, it's a collection of songs about customer service. Underneath, it's about so much more. Finally, Zendesk Alternative and Zendesk® the customer service software company are together at last.

that is hilarious, but also the most non punk rock thing I've ever read. if Apple did it, every one here would be fawning over how genius of a move it were

appendix-rock2• 2 years ago

No they wouldn’t. You just dislike Apple.

dewey3• 2 years ago

That sounds more like an April Fool's Day joke than something malicious. These were still fun sometimes back in the days (2016).

I wouldn't read too much into this because one unmaintained old website will not going to make or break the SEO game of others.

lathiat3• 2 years ago

I love the old school logos for Twitter, MySpace, Facebook, etc in the footer.

Takes me back to a more innocent time of the Internet.

josu• 2 years ago

I have a similar conspiracy theory for DDG, the rapper. I used to go to DuckDuckGo by typing "ddg" in Google. Now, it's all mentions to DDG the rapper.

tiffanyh• 2 years ago

https://duck.com works.

Ironically, the domain was given to DDG from Google.

mkl• 2 years ago

ddg.gg will take you straight there and is just as quick to type (of course, it's simpler to set it as default).

Elinvynia• 2 years ago

To be fair, his name is Darryl Dwayne Granberry

jeanlucas• 2 years ago

It might make you nervous, but this is our purpose.

I love the self report :^)

otteromkram• 2 years ago

https://zendeskcompetitor.com is still available.

bravura• 2 years ago

Zendesk Competitor. That's actually the name of my AI-generated death metal band.

sangnoir• 2 years ago

While not illegal, it shows the way they think, a sort of manipulative pettiness

Search engine chaff has been in the toolbag of "reputation management" PR firms for a while. Boris Johnson (former UK PM) deployed it many years ago to drown out a viral picture of him in front of a red bus bearing an ad he no longer wanted to associated with, so he was coached to tell an interviewer he has a hobby of painting model buses red.

OptionOfT• 2 years ago

In general it has become acceptable to create fake pages to get traffic to your website. E.g. this like: https://instatus.com/blog/best-open-source-status-page-servi... the 0th entry is taken by the service itself, and they're not open source...

junto• 2 years ago

I help corporates evaluate and buy software. Having an ineffective bug bounty program, especially one that rewards black market activity on a terms & conditions technicality like this, is enough for me to put a black mark on your software services.

I don’t care if you’re the only company in the market, I’ll still blackball you for this in my recommendations.

Zendesk should pay up, apologize and correct their bug bounty program. After doing so, they should kindly ask the finder to add an update to this post, because otherwise it will follow them around like dogshit under their shoe.

exceptione2• 2 years ago

Yes, I think bounties in this class and with this impact should at least be six figures.

If a company loses 120 million a year to security bounties, they will take into account the cost of scrumming/rapid widget delivery.

moritonal• 2 years ago

Would love to see the parts of the market where you've marked off every current option, given each would represent new business opportunities.

xyst2• 2 years ago

Probably any SK company. Bounties are awful and only paid out to SK citizens. Everyone else gets a pat on the back for being a sucker.

yieldcrv7• 2 years ago

HackerOne’s mediator dropped the ball here

They should absolutely inform a client company of a perceived threat, when they agree on the threat

Most of the person’s post and responses here are about Zendesk’s issue, but Zendesk was never informed

for a better PR response, I think now Zendesk could reward this after realizing it wouldnt have been disclosed first, and admonish HackerOne for not informing them and the current policies there

richbell5• 2 years ago

Most of the person’s post and responses here are about Zendesk’s issue, but Zendesk was never informed

It's not clear whether they were informed. The mediator's email says "after consultations with *the team*", which is likely referring to Zendesk's security team.

daghamm2• 2 years ago

This is pretty common on H1, probably due to the amount of crap they receive.

If you are a new user expect your first couple reports to be butchered. It seems to me only reports from well known hackers gets carefully analysed.

M4v3R• 2 years ago

Zendesk was informed. OP specifically said they asked h1 to escalate to the company itself and the second email they present way from someone from Zendesk, who still rejected them, adding that this decision was made “after consulting with the team”.

klabb3• 2 years ago

I swear I’ve seen this vuln years ago. I thought it was already well known that attacker controlled input for email-bridged ticketing systems means attackers can access at least one @company.com email.

I thought this was mainly mitigated by invalidating the assumption that “only authorized employees can control a company email” – it used to be common 5-10 years ago to verify “that you’re an employee” that way, but I just assumed that kind of stopped in favor of whatever SSO/SAML stuff that became popular with enterprise.

Is this the same thing? Or a variation?

mmsc11• 2 years ago

It's a common thing: https://www.google.com/search?q=slack+google+saml+vulnerabil...

RicoElectrico• 2 years ago

Based on experience of my friend I am inclined to believe that Zendesk is full of shit. She's had bad experience with their Polish site which she described as cultish employer.

gavingmiller4• 2 years ago

The piece the author is missing, and why zendesk likely ignored this is impact, and it's something I continually see submissions lacking. As a researcher, if you can't demonstrate impact of your vulnerability, then it looks like just another bug. A public program like zendesk is going to be swamped with reports, and they're using hackerone triagers to augment that volume. The triage system reads through a lot of reports - without clear impact, lots of vulnerabilities look like "just another bug". Notice that Zendesk took notice once mondev was able to escalate to an ATO[1]. That's impact, and that gets noticed!

[1] https://gist.github.com/hackermondev/68ec8ed145fcee49d2f5e2b...

patcon2• 2 years ago

Yes. But respectfully (residual frustration at zendesk might make me curt here) if their security triage team can't see how dangerous it is for an attacker to get access to an arbitrary thread on a their CLIENT's corporate email chains (in this world of email logins and SSO), then they have a big lapse in security culture, no?

Yes, the researcher could have tee'd himself up better, but this says way more about zendesk than it does about the 15-year-old researcher.

23B12• 2 years ago

"If you won't illustrate the impact of our mistake, we aren't obligated to listen to you" is peak CYA

gavingmiller4• 2 years ago

Not even close to the point I was making: If you want to get taken seriously, write to audience.

ec1096856• 2 years ago

The researcher showed how they could hop onto any Zendesk support ticket thread with zero authentication, so that should have been enough given Zendesk was exposing customer data via that attack path.

Clearly Zendesk needs to change things so that the email address that is created for a ticket isn’t guessable.

XCabbage• 2 years ago

Unauthorized read access to private emails you were never legitimately CCed on already is impact. It should not be necessary to come up with a further exploit daisy chained on top of that in order to be taken seriously. (Otherwise why stop at Slack access? Why is that automatically "impact" if email access isn't?)

lysp• 2 years ago

Exactly.

It's possible that some chains could have credentials or other sensitive information in ticket chains.

Aachen6• 2 years ago

Exploit or no, the bug and potential impact are the same. I personally find it a waste of time to sink evenings into an exploit when they're going to fix the bug anyway if I simply tell them about the problem. They also know the system better than I do and can probably find a bigger impact anyway

Of course, this is only a good strategy if you're just wanting to do a good deed and not counting on getting more than a thank you note, but Zendesk or Hackerone (whoever you want to blame here) didn't even accept the bug in the first place. That's the problem here, not the omission of an exploit chain

tptacek5• 2 years ago

I think this is (descriptively) correct, but it's a difficult point to make in a message board argument because of hindsight bias.

gavingmiller4• 2 years ago

It’s a good callout, shouldn’t have editorialized like that.

davedx• 2 years ago

I don't think it is. Getting arbitrary access to corporate support ticket chains seems pretty high impact to me? Isn't that a gigantic data breach (also probably a GDPR breach) already, before you get to the Slack takeover?

dclowd99012• 2 years ago

The dude demonstrated the ability to infiltrate a client’s Slack instance via their vulnerability. If that’s not enough to make the hairs on your neck stand on end as an engineer, go fucking do something else.

thrdbndndn• 2 years ago

He didn't demonstrate this in his initial report to Zendesk.

oarla• 2 years ago

The worse part:"We kindly request you keep this report between you and Zendesk". After being notified of a problem on their side, them ignoring it, now they want to keep things hush hush? That's exactly what the author did in the first place, but they chose to brush it aside. That itself is highly unprofessional. With such an attitude, I'm not surprised that they did not pay out the bounty.

op00to2• 2 years ago

“I will consider not disclosing if you compensate me for my time.”

jjmarr• 2 years ago

You can't ask for money in exchange for not revealing a bug. That's blackmail which is illegal and ethically dubious.

White hat hackers do not require companies to pay them in exchange for not revealing a bug---the reveal of a bug only happens if a company doesn't fix that bug. Companies can be jerks and refuse to pay anything. That doesn't give you the right to blackmail them---you and other security researchers can just refuse to help them in the future.

A refusal to fix the vulnerability is what happened in the original blogpost, so it was fair game for release since the company doesn't care.

Hackers that don't care about ethics or legality won't bother blackmailing companies with vulnerabilities. They'll sell or use the vulnerability to steal more important data, and blackmail companies for millions of dollars in crypto.

daghamm2• 2 years ago

The correct procedure when they fuck up and close the report is to ask the report to be made public. Had he done this, this would have been a non issue.

The reason people don't do this is because they think they have something that can be modified into another bug. Which is exactly what happened here.

kachapopopow2• 2 years ago

And this is how whitehats get turned into blackhats and just choose to use this information to perform social engineering or exploit devices.

kachapopopow2• 2 years ago

I've seen this go from -2 points to 2 points, now back to 1. Interesting how people are so divided on this when people I've brought up this topic to all agree that good people often get taken advantage off especially in cases like this.

yapyap• 2 years ago

Welp, they’re basically begging people to sell 0 days to a 3 letter agency // out of state groups (NSO, etc)

Come on, honor your bug bounty. Especially when it can bite you in the ass hard enough to plummet stock prices if a bug of this caliber is exploited in the companies you serve

zx8080• 2 years ago

”Too big to pay.”

Nothing changes for them even if they ignore one report, especially from an unknown researcher.

It exposes pretty sad state of the industry. Who said enterprise?

mmsc11• 2 years ago

From what I can tell, the vulnerability wasn't even fixed: they just.. changed their spam filter? Whatever that means.

So for this to work still, you need to bypass a spam filter.

They should just force DMARC and SPF like Google has done, and say "your fault if you misconfigure". Also default-off for the CC thing would be a good idea, too, with a warning of what could happen if they turn it on. Alternatively making a non-guessable id for the email.

vitus2• 2 years ago

Hey, now you have to bypass two spam filters, and also email verification from Apple is now marked as likely-spam. Which addresses the very specific Slack infiltration attack, but doesn't address the underlying issue.

layer87• 2 years ago

Requiring their customers to implement SPF and DMARC as a hard technical requirement is probably bad for business. And as mentioned in TFA, they do note issues regarding SPF/DMARC in their policy.

madisp2• 2 years ago

I think in this case it's the customers of their customers, e.g. people sending emails to support@acme-corp.com. In that light requiring all emails coming into support@acme-corp.com to have SPF and DMARC is bad for business indeed, not only for Zendesk but probably also for the fictional ACME corp.

EDIT: they absolutley should not use an autoincrementing int as a "support-chain token" though, that's a workaround they could easily do.

tasn• 2 years ago

This is clever hack and a reminder of how a chain of smaller security issues (guessable ticket IDs, email spoofing, automatically adding emails to tickets, etc.) can lead to larger ones.

Zendesk deserve a lot of flack here, especially after they already realized this is real. However, just to empathize a bit: the amount of spam SPF, DKIM, DMARC "security" reports anyone running a service gets is absolutely insane. So it's very easy to accidentally misclassify what this reporter originally discovered as that by accident.

twoodfin• 2 years ago

Slack seems to be getting off too easy here. The security—as implemented by Fortune 500 customers??—of an org-wide security domain (i.e. what everyone in an org can see) depends on whether any of the supported OAuth providers can be tricked into provisioning an account with @targetorg.com?

This architecture makes 0 sense to me. Even if an org has totally outsourced its identity and auth management to Google (is this possible?), presumably that would include controls over how new @targetorg.com identities are created on the Google end.

No F500 companies are using Apple as an identity provider since they definitely don't sell such services. So why would an F500 company configure Slack to allow Apple OAuth & introduce this vulnerability?

tryauuum4• 2 years ago

I'm not sure (maybe it's the case only with email auth, not oauth). But there's a setting on slack to not automatically allow people with your company email address. So the tools are there to stop the attack

autarch• 2 years ago

I'm pretty sure you can configure your corporate Slack to only allow authentication providers you choose. So if you have a corporate SSO you can just allow that.

kbolino2• 2 years ago

If you're using Google for identity and authentication, you can definitely control who has an active account in your domain. There can be some lag time before disabling or removing someone truly disables all their downstream accesses, but that's largely outside Google's control. The only way to trick your way into getting a corporate domain email address is to socially engineer a domain admin.

Tangentially, this does raise one of my big issues with using OAuth2 for single-sign-on though, which is that it really doesn't address the third-party authorization problem well. Ok, you're bob@example.com, Google has verified that, and example.com is our domain, so we're letting you into our app Foobar. Now what? The scopes you requested were for Google APIs only and have nothing to do with Foobar really. So now we need to implement an authorization system in Foobar that decides what bob@example.com can actually do. This part, and how to do it right, gets glossed over (at best!) by discussions on OAuth2. It also gets glossed over by product and security when they want things "SSO-enabled", which means development time doesn't get budgeted for it. Even just using groups to control coarse access levels requires integration with provider-specific APIs. OAuth2 is great for identity and authentication, but far too little attention has been paid to doing authorization right.

pkaeding• 2 years ago

You can also create a non-email Google account as Bob+external@example.com, as long as you can get email sent to bob@example.com (ie, while you are employed by Example, Inc). Then, you leave your job, but still have a google account associated with an example.com email. Depending on how the app checks the login response, they might mistakenly assume you are part of the example.com org.

m11a3• 2 years ago

This comment makes the most sense to me in this thread.

Further, IMO, sure it's a bug that one can say they control support@company.com. But IMO the real issue is lousy, permissive authorisation that gives access to anything simply by virtue of controlling a @company.com mail. Surely some HR/tech person, when an employee is being onboarded, should be enabling access to some core systems (probably by adding them to a group), and the default state for an account with no groups should be no access.

In any large enough organisation, IME there's a lot of ways to get a @org.com email, and too many people/systems have the ability to create an email than a single centralised IT team.

Kwpolska• 2 years ago

Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.

Do you find it surprising that they use Microsoft Office too? Paying someone else to handle things like this is cheaper than paying developers and hosting a service like this.

npsomaratna• 2 years ago

I’d give the author a break-he’s just 15, after all. I was far less savvy at his age.

zharknado• 2 years ago

Agreed, I smiled a this line. Good reminder that you don’t have to be super experienced to have big insights and impact.

And also that being brilliant doesn’t magically correlate with being knowledgeable.

https://xkcd.com/1053

aftbit2• 2 years ago

Wait... it looks like Zendesk only fixed the issue of Apple account verification emails being added to tickets, not actually the underlying issue?

In addition to this, we also implemented filters to automatically suspend the following classes of emails: User verification emails sent by Apple based on the Reply-To and Message-Id header values Non-transactional emails from from googleworkspace-noreply@google.com Over the coming months, we will continue to look into opportunities to strengthen our Sender Authentication functionality and provide customers with more gradual and advanced security controls over the types of emails that get suspended or rejected.

So is it still possible to hijack anyone's support tickets using the default configuration of Zendesk if you just happen to know their email and ticket ID?

vitus2• 2 years ago

Yeah. Zendesk only put a bandaid in place to prevent this particular attack vector for the Slack infiltration attack, and did nothing for the initially reported issue.

layer87• 2 years ago

Only the customer domain owners can fix the underlying issue, which is a missing SPF/DMARC configuration.

cjbprime3• 2 years ago

That doesn't sound right. Aren't these @zendesk.com addresses?

Aachen6• 2 years ago

They could make the ticket IDs unpredictable so you can't subscribe yourself to any existing ticket by sending it an email

aftbit2• 2 years ago

Zendesk could refuse to allow "ticket collaboration" if customers had a missing or insufficiently secure SPF/DMARC configuration, or at least make customers check a box that says "Tickets may leak their contents to anyone who can send emails".

eatbots• 2 years ago

Reported this exact bug to Zendesk, Apple, and Slack in June 2024, both through HackerOne and by escalating directly to engs or PMs at each company.

I doubt we were the first. That is presumably the reason they failed to pay out.

The real issue is that non-directory SSO options like Sign in with Apple (SIWA) have been incorrectly implemented almost everywhere, including by Slack and other large companies we alerted in June.

Non-directory SSO should not have equal trust vs. directory SSO. If you have a Google account and use Google SSO, Google can attest that you control that account. Same with Okta and Okta SSO.

SIWA, GitHub Auth, etc are not doing this. They rely on a weaker proof, usually just control of email at a single point in time.

SSO providers are not fungible, even if the email address is the same. You need to take this into account when designing your trust model. Most services do not.

cxcorp• 2 years ago

This is very important to keep in mind when implementing OAuth authentication! Not every SSO provider is the same. Even if the SSO provider tells you that the user's email is X, they might not even have confirmed that email address! Don't trust it and confirm the email yourself!

hmottestad6• 2 years ago

And remember to add a random unique id to the reply-to email, otherwise you’ve fallen into the same trap.

harrisonjackson2• 2 years ago

Ah, that makes a lot of sense. This is a foot gun that you can run into even with an auth provider like Auth0 or Clerk let alone rolling your own.

Directory SSO: These are systems like Google Workspace or Okta, which maintain a central directory of users and their access rights.

Non-directory SSO: These are services like "Sign in with Apple" (SIWA) or GitHub authentication, which don't maintain such a directory.

ec1096856• 2 years ago

Not only is Apple non-directory, they are non-discretionary as well, so foisted on services and handled poorly as a result.

bushido• 2 years ago

Out of curiosity, do you know of open source projects or any resources that someone less familiar with SSO can use/read to properly implement SSO?

siddthesquid• 2 years ago

Something like Keycloak?

to11mtm3• 2 years ago

Not the best suggestion but haven't seen others give any yet...

IdentityServer4 [0] is no longer maintained [1] but had SSO support and the source is still on github.

[0] https://identityserver4.readthedocs.io/en/latest/

[1] - They had to go commercial to stay afloat, there wasn't enough contributions from community/etc. That said it's pretty cheap for what it does in the .NET space.

LilBytes• 2 years ago

Keycloak (Java) and Zitadel (Go) are my recommendations.

bitexploder• 2 years ago

Use OIDC. It is based on Oauth. I would fiddle with implementing basic Oauth clients first. Like a Spotify playlist fetcher or something. Just to start getting a feel for the flows and things you would be concerned with.

austinkhale2• 2 years ago

Presumably one of the PMs you’re referring to has posted this article for additional information. Feels like they’re doubling down on their initial position.

https://support.zendesk.com/hc/en-us/articles/8187090244506-...

8n4vidtmkvmk• 2 years ago

Although the researcher did initially submit the vulnerability through our established process, they violated key ethical principles by directly contacting third parties about their report prior to remediation. This was in violation of bug bounty terms of service, which are industry standard and intended to protect the white hat community while also supporting responsible disclosure. This breach of trust resulted in the forfeiture of their reward, as we maintain strict standards for responsible disclosure.

Wow... there was no indication that they even intended on fixing the issue, what was Daniel hackermondev supposed to do? Disclosing this to the affected users probably was the most ethical thing to do. I don't think he posted the vulnerability publicly until after the fix. "Forfeiture of their award" -- they said multiple times that it didn't qualify, they had no intention of ever giving a reward.

thekevan• 2 years ago

So when the researcher said it was a bug, they said, "No, it's fine. No bug bounty, sorry."

THEN the researcher eventually goes public.

Later, Zendesk announces the bug and the fix and says there will be no bug bounty because the researcher went public.

Is that how it went? I mean if so, that's one way to save on bug bounties.

Shank• 2 years ago

In damage control mode, Zendesk can't pay a bounty out here? Come on. This is amateur hour. The reputational damage that comes from "the company that goes on the offensive and doesn't pay out legitimate bounties" impacts the overall results you get from a bug bounty program. "Pissing off the hackers" is not a way to keep people reporting credible bugs to your service.

I don't understand what this tries to accomplish. The problem is bad, botching the triage is bad, and the bounty is relatively cheap. I understand that this feels bad from an egg-on-face perspective, but I would much rather be told by a penetration tester about a bug in a third-party service provider than not be told at all just to respect a program's bug bounty policy.

dclowd99012• 2 years ago

As a former ZD engineer, shame on you Mr Cusick (yes, I know you personally) and shame on my fellow colleagues for not handling this in a more proactive and reasonable way.

Another example of impotent PMs, private equity firms meddling and modern software engineering taking a back seat to business interests. Truly pathetic. Truly truly pathetic.

teddyh5• 2 years ago

That article claims to have “0 comments”, but currently sits at a score of -7 (negative 7) votes of helpful/not helpful. I think they have turned off comments on that article, but aren’t willing to admit it.

EDIT: It’s -11 (negative 11) now. Still “0 comments”.

JonChesterfield• 2 years ago

We, the company that doesn't understand security, can't tell whether this was exploited, therefore we confidently assert that everything is fine. It's self consistent I suppose but I wouldn't personally choose to scream "we are incompetent and do not care" into the internet.

bsbsjsusj3• 2 years ago

Django all auth seems to know this with social provider specific email settings.

marcellus232• 2 years ago

Can you explain a bit more what makes Sign in with Apple different from Google Sign-in? Apple certainly does maintain a list of users with accounts. So what does "non-directory" mean here exactly? Why can Apple not attest that you control that account at sign-in time?

lathiat3• 2 years ago

I think what he means is, if you have an @gmail.com account via Google, that is pretty good proof of control. But if you have any other e-mail (e.g. a custom domain) via Google, it's not.

Similar with Apple, if you were signing in with an @icloud.com, it's pretty good proof, but if you have an Apple ID with a third-party e-mail it's not proof of current control of that e-mail.

That's my guess.

crusty• 2 years ago

Now? nothing. I think this thinking is a relic of Google's status as seemingly the last remaining email provider to automatically create a Gmail account when signing up for a Google account. So using Google SSO meant using your Gmail account, and so control of the email address was nessissary for control of the Google account. If you lose the email account, you lose the Google account. This is not true anymore since you can sign up for a Google account with any email.

Whereas you can (and I believe always could*) create an apple ID with any old email address.

*Maybe this delinked situation only came about when they added the App Store to OS X, and figured they'd make less money if they require existing Mac users to get a new email account in order to buy programs in the manner which would grant them a cut.

Apple has a list of all the email addresses for its sole IDs, but it doesn't control them, and having one deleted doesn't nessisarilly affect the other.

Google and custom domain email have always been delinked from this perspective. You could create a Google account with a custom domain and then point the domain elsewhere or lose control of it, and you'd still retain conto of the account.

Basically, the required example essentially theoretical at this point - maybe it works for employers at companies that also happen to provide SSO services. So if you work at Facebook, Google, Apple, or github and have a me@FGAG.etc.com email dress, and you signed into slack through the SSO that affiliated with your company and the company email, but later don't work there and you've had your work account access revoked, you won't be able to use that SSO to sign into slack. That's what they mean by directory control or whatever.

In contrast, if you sign up to github with your work email account, unless it's a managed device managed by your work, your work doesn't actually control the account. They just vouched for your affiliation at sign up when you verified your email. So if you use a github SSO to sign up for a service that 'verifies' your work email address from github during the process, that won't change when you leave and the company revoked access to the email. Github SSO, in this case, isn't verifying you have an email account @company.com. They are verifying you once did, or at least once had, access to it. This is what they mean by the non-directory whatever.

quacksilver2• 2 years ago

I do web app testing and report a similar issue as a risk rather often to my clients. You can replace Google below with many other identity providers.

Imagine Bob works at Example Inc. and has email address bob@example.com

Bob can get a Google account with primary email address bob@example.com. He can legitimately pass verification.

Bob then gets fired for fraud or sexual harassment or something else gross misconduct-y and leaves his employer on bad terms.

Bob still has access to the Google account bob@example.com. It didn't get revoked when they fired him and locked his accounts on company systems. He can use the account indefinitely to get Google to attest for his identity.

Example Inc. subscribes to several SaaS apps, that offer Google as an identity provider for SSO. The SaaS app validates that he can get a trusted provider to authenticate that he has an @example.com email address and adds him to the list of permitted users. Bob can use these SaaS apps years later and pull data from them despite having left the company on bad terms. This is bad.

I think the only way for Example Inc. to stop this in the case of Google would be to create a workspace account and use the option to prove domain ownership and force accounts that are unmanaged to either become managed or change their address by a certain date. https://support.google.com/a/answer/6178640?hl=en

Other providers may not even offer something like this, and it relies on Example Inc. seeking out the identity providers, which seems unreasonable. How do you stop your corporate users signing up for the hot new InstaTwitch gaming app or Grinderble dating service that you have never heard of and using that to authenticate to your sales CRM full of customer data?

hunter2_• 2 years ago

I wonder why Google would make an SSO assertion along the lines of "yes, this user Bob has email address bob@example.com" in the situation where example.com is not under a Workspace account. Such assertions ought to be made only for Workspace (and Google's own domains such as gmail.com, googlemail.com, etc.) since outside of that it's obsolete proof as you say, i.e. it's merely a username of a Google account which happens to look like an email address, and nothing more.

rkharsan64• 2 years ago

Every time I've left an organization, they have swiftly deleted the company email address/revoked my access to it. I assume every reasonable organization will have processes in place to do this.

I don't see this as a vulnerability: how is Google supposed to know that a person has left the company? You let them know by deleting the account.

laz• 2 years ago

You don't need full blown workspace, which costs money, you can set up "cloud identity free" and claim the domain.

When you're setting it up, you can choose what to do with any existing accounts that are part of your domain: kick them out or merge them in.

kccqzy• 2 years ago

If anyone needs motivation of such unmanaged users, I actively use this feature. I have my own Google Workspace on my own domain. Years ago when I bought a Nest product I found that I couldn't use a Google Workspace account to access Nest. No problem, I create a consumer Google account under my Google Workspace domain. The email looks just like a Workspace account. And it doesn't need any additional Workspace licenses. (I no longer plan to buy any more Nest devices so I'll delete the account once my last Nest product stops working.)

oliwary• 2 years ago

Perhaps the following could be a solution to this issue?

Any OAuth provider should send a flag called "attest_identity_ownership" (false, true) as part of the uaht flow, which is set to true if the account is a workspace account or gmail (or the equivalent for other services), and false if the email is an outside email. Thus, the service handling the login could decide whether to trust the login or proceed otherwise, e.g. by telling the user to use a different OAuth service/internal mechanism where the identity is attested.

wutwutwat• 2 years ago

This is why you store and match on the sso provider’s uuid, not on the email address. Emails are not a unique identifier and never have been. Someone can delete their email account and later someone else can sign up and claim the same email. Anyone matching on email addresses is doing it wrong. I’ve tried to argue this to management at companies I’ve worked at in the past, but most see my concern as paranoid.

ryukoposting• 2 years ago

Isn't the simplest solution here to not support SSO at all?

I get there's a convenience factor, but even more convenient is the password manager built into every modern browser and smartphone. If the client decides to use bad passwords, that's will hurt them whether or not they're using SSO.

pas• 2 years ago

SSO is fine, but verify the email address that the SSO provider has given (unless the provider is authoritative for the email domain)

mjomaa• 2 years ago

You mean automatic account linking with unverified emails?

namaria• 2 years ago

The amount of software that could have been some spreadsheets and an email chain and companies pay enormous amounts of money for and create glaring vulnerabilities in their systems is a big reason why I'll never understand or thrive in a corporate setting.

xyst2• 2 years ago

Zendesk is a piece of shit company for not paying out this researcher.

H1 should restrict ZD from their platform.

ericbarrett• 2 years ago

This was a great writeup, really clear and engagingly written, about an interesting and subtle bug. If the author hadn't mentioned they were 15 I would have assumed it was from a seasoned security professional.

To Daniel/hackermondev: whatever you're doing, keep it up!

kmoser• 2 years ago

Agreed, if I was hiring and the author applied, I would base most of my decision on the quality of this article alone. Just goes to show how engaging communication beats whiteboard interviews, at least for me.

mgaunard• 2 years ago

In my experience, telling people you've found a vulnerability in their system is met with denial or lack of care, and when you demonstrate the exploit for them to take it seriously, they sue you.

You're better off not bothering contacting them.

harrisonjackson2• 2 years ago

Crazy that a company/product that depends so much on email as an interface would have policies to dismiss a bug like this.

It is challenging for Zendesk to enforce or fix DKIM, SPF, and DMARC issues across all client domains so better to just ignore it :grimace:

pheatherlite• 2 years ago

And I thought I was once a clever 15 year old... this was brilliant. Sharp kid.

Though his pondering of 'why do companies use third party support systems instead of rolling their own' gave his age away :)

wglass• 2 years ago

Seems to me this is just as much a problem with Slack and its configuration as with Zendesk.

29athrowaway• 2 years ago

Why would a security researcher risk doing a significant amount of free work for Zendesk in the future?

Why do they even have a bounty program if they do this?

ec1096856• 2 years ago

While obviously Zendesk leaving such a huge hole was the main reason for this exploit (you should obviously fail closed when email signals suggest it’s an unauthenticated address), a contributing factor is that Apple forced themselves to be added as an SSO provider.

So the hacks put in place to deal with Google SSO hadn’t been put in place for Apple’s.

Also, what Fortune 500 company is leaving slack’s email based login feature enabled? Why wouldn’t they all be using a corporate SSO solution tied to their company’s slack directory?

yieldcrv7• 2 years ago

"out of scope," said no attacker ever

I like this kid

failedrequest• 2 years ago

Zendesk handles customer data and says its a great customer support platform.

The irony is that they have the worst support and one of the worst response times to issues reported. You will find many experiences on LinkedIn reporting this.

bronco21016• 2 years ago

You probably didn't speak with a human since Zendesk only allows you to speak to a bot unless you give them several thousand/month.

natch• 2 years ago

Unfortunately the headline makes it look like Zendesk paid a bounty, when they did not. Clearly they have a doofus in charge of security.

lifeisstillgood• 2 years ago

I have been thinking about it for a while, but I no longer think email is worth the trouble. I am flooded with spam, the vulnerabilities are everywhere and locking them down near impossible.

chias• 2 years ago

The dumb thing is that this "out of scope" thing is 100% a Hacker One failure, and exactly the kind of thing I've grown to expect from these triage teams.

"SPF, DKIM, and DMARC issues" is absolutely and positively intended to mean "we don't care if we are missing these headers on our domains", in part because this is 99.9% of drive-by beg bounties (if you are tired of getting "I HAVE FOUND A SERIOUS SECURITY ISSUE IN YOUR WEBSITE" cc'ing security@, legal@, privacy@, and your CEO on a monthly cadence, just set up a DKIM record :P)

Yes, this is technically a bug which is in the space of SPF, DKIM, and/or DMARC. But this is absolutely NOT WHAT THE EXCLUSION IS FOR. Hacker One triage teams should know better, and it's frankly embarrassing that they don't. And it's frankly mortifying that their mediation team also didn't pick up on this.

But it checks out.

This is one of the reasons I will not use Hacker One ever. Bugcrowd is slightly better. Intigriti has (so far) been pretty good. I'm not affiliated with any of them, just have been a customer of all three.

josephh• 2 years ago

A more sensible thing that should've been done is to tokenize the case ID so that you can't just guess it with a numerical range. Also important that you don't leak your key business metrics (# of support cases over time).

atum47• 2 years ago

I once found a neat trick to highjack Facebook accounts; after MSN message died a lot of hotmail accounts were left to die also but a lot of people created their Facebook account using the @hotmail address. I tipped them about that they basically said F you.

brainzap• 2 years ago

when I tested zendesk for potential ticket system I felt that this is half abandoned, there are probably more security issues to find

nodamage7• 2 years ago

In case it's not clear these are the two separate vulnerabilities:

1. Zendesk allows you to add a CC to any existing support ticket by sending a (spoofed) reply from the original requestor's email address to that ticket's Reply-To address and including a CC in the email.

In some circumstances the Reply-To address is based on an auto-incrementing integer so it can be guessed. (Although this may not alway be the case: my email archives show some emails from Zendesk using the integer and other emails using a random alphanumeric string. It seems to vary by company so it might be some sort of configuration setting?)

2. Slack allows third-party domain-wide logins via Sign in with Apple without additional verification that the email belongs to a real person. Here the author of the article pretends to be support@company.com to Slack and Slack lets them into the company.com channel, despite the fact that support@company.com does not actually represent a real user and is only intended to be a receiving email address that forwards into Zendesk. (This sounds more like a configuration problem than anything else though.)

kidbomb2• 2 years ago

IMHO the second problem goes deeper:

Sign In with Apple is allowing you to "create an account"(author's words) on @company.com, which should not be supported in the firat place. Instead, it should rely in a central directory controlled by company.com for authentication

robbiet480• 2 years ago

Those with Apple Business or School Manager can now claim domain names which blocks sign ups under claimed domain names.

nodamage7• 2 years ago

Apparently so do Google and Github according to the other comments in this thread. Seems like a potential design flaw in these SSO implementations.

Rugu16• 2 years ago

Its all because someone’s ego was hurt that their security was breached by 15 year old kid. Sometimes the simplest explanation is the right one.

Eumenes• 2 years ago

Zendesk paid me $500 4 years after I submitted a vulnerability to their bounty program on hackerone. It was a shock. There were default settings indexing internal docs/knowledgebase info to search engines.

wslh• 2 years ago

This is where despite the bug bounty rules someone at Zendesk (a CXO?) should take action and pay.

I would add that HackerOne could also take an action and influence in the positive outcome and not being just a MiM.

tptacek5• 2 years ago

If you want to understand the dynamics of what happened here, a very important detail is that the bounty hunter's report implicated DKIM and SPF, and no bug bounty program in the world takes DKIM reports seriously. DKIM is the archetypical beg bounty. You could find DKIM RCE and HackerOne would still round file your report.

heraldgeezer3• 2 years ago

Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.

Ah, yes, why do laymen always think this?

I mean, I get it, Krupp and mining towns used to be a thing, so it is possible.

But every big company should build a ticketing system? Why not an email solution, OS, network routers too?

lukeh• 2 years ago

As someone else mentioned, he is 15. I would have thought the same thing when I was his age.

heraldgeezer3• 2 years ago

When I was 15 I was jerking off and playing Flyff, not thinking about ticket systems.

matrix12• 2 years ago

I blame @otterly for bringing in the ex yahoo people who locked devs out environments. Causing them to add backdoors for debugging.

heraldgeezer3• 2 years ago

Would this work with Jira, Servicenow, Spiceworks etc...? :x hehe

ZendeskTeam• 2 years ago

Our team at Zendesk has posted some more details about this bug here: https://support.zendesk.com/hc/en-us/articles/8187090244506-...

bredren• 2 years ago

We also want to address the Bug Bounty program associated with this case. Although the researcher did initially submit the vulnerability through our established process, they violated key ethical principles by directly contacting third parties about their report prior to remediation.

What was the planned response for addressing the vulnerability reported through the Bug Bounty program, and how did the plan change after the researcher escalated the issue directly to Zendesk before remediation was completed?

whatthefk• 2 years ago

1. "While this specific issue has been resolved", that was a bug, not an issue.

2. "they violated key ethical principles by directly contacting third parties about their report prior to remediation", what is a violation of ethical principles is to know about a security failure in your application and ignore it, leaving customers at risk, can't wait for some law to pass so people who behave like that face consequences.

3. "We have no evidence that this vulnerability was exploited by a bad actor.", tldr, it don't fixed it until some vendor dropped us, because before that happened, it was cheaper to ignore it.

austinkhale2• 2 years ago

Wild response. It appears you did not learn a single lesson through this process.

rendall2• 2 years ago

...they violated key ethical principles by directly contacting third parties about their report prior to remediation

According to the researcher, they only contacted 3rd parties after Zendesk rejected the disclosure as out of scope, as they are free to do.

If this timeline is incorrect, Zendesk should immediately correct the record. As it stands, accusing the researcher of violating ethical principles looks very bad for Zendesk. Perhaps even libelous.

That it affected Slack was a side-effect of the original bug, and not a new, previously undisclosed bug. Zendesk fixed the original bug, after rejecting the disclosure. Given all that, Zendesk is still ethically bound to honor the bounty, 3rd party disclosures notwithstanding.

rbut• 2 years ago

Extremely poor response. You can't blame him for contacting others affected when you marked it as out of scope. And yet you fail to mention that in your blog post..

mikeortman• 2 years ago

You didn't provide details, you provided a defense, which is weak at best.

UniverseHacker• 2 years ago

This response looks really bad for ZenDesk.

This is blatant dishonesty- the post documented in detail that the reward had already been denied, and the issue ignored multiple times before they contacted 3rd parties. That is not an ethical violation but an ethical necessity- after ZenDesk refused to act, they had an ethical responsibility to inform everyone affected.

This alone is a huge red flag that ZenDesk isn't a trustworthy organization, on top of trying to hide rather than correct security issues unless they get bad press.

If I were ZenDesk, I would pay out the bounty to this kid immediately, and release a detailed public apology explaining how the entire bounty review system has been revamped to take things like this much more seriously in the future.

forvelin• 2 years ago

this is a horrible response. just upvoting so more people can see this mess.

integral_1699• 2 years ago

Terrible response. Your team should do a retrospective on its retrospective.

philipwhiuk• 2 years ago

Other than violating the Bug Reporting program how else would it have been fixed given you failed to recognise the issue?

Are you yourself not at fault for forcing him to violate the terms in order to protect your customers?

rlnvlc• 2 years ago

This response is worse than no response at all.

jazz9k• 2 years ago

I've been making money finding bugs for H1 and have made >100k.

I finally stopped when two large companies have stopped communicating with me over the last year (all bugs have been triaged on the H1 side).

They owe me a total of around 30k. H1 can't do anything about it. It seems there is no actual contract in place to protect researchers.

hmottestad6• 2 years ago

Would love to hear more! I’m always eager to add companies to my blacklist.

tgsovlerkhgsel2• 2 years ago

Same experience where I reported a bug, the company ghosted me, and H1 did not even allow disclosure through their platform.

I generally refuse to go through platforms now (also because I really hate being subject to the psychological pressure of a "social credit system", even though I understand why the platforms do it), so if your company doesn't have an alternative reporting form, or refuses bug bounty payouts when a valid issue was reported directly through them instead of through a platform (hello, Backblaze!), I'm not doing free labor for you and you will likely hear about the bug when either someone else finds it or I include it in a public write-up (if it's a bug affecting multiple companies).

rendall2• 2 years ago

I wonder what would happen if researchers en masse were to boycott a particular platform? Disclose to the companies directly and explain they won't work with X and why. Treat any attempt by the company to kick the disclosure back to platform X as a non-response.

jokethrowaway• 2 years ago

Name and shame the companies

farmeroy• 2 years ago

I've really enjoyed all reading about all the vulnerability blogs posted over the last month! And from so many young researchers, it's super inspiring

joshdavham2• 2 years ago

Slightly off topic, but why does it always seem to be teenagers who are the most talented at finding exploits?

tgsovlerkhgsel2• 2 years ago

Curiosity and near endless time to dedicate to it make for a powerful combination.

joshdavham2• 2 years ago

Beautifully said.

yelnatz• 2 years ago

Kids who are smart and curious have already been exposed to the internet since they were 8/9 years old. So a few years of learning they're already knowledgeable by the time they're in their teens.

No responsibilities and all the time in the world to learn with the right circumstances, they can go far.

Older devs who are experts in this area are already busy making money and working on their employer's security.

vladmk• 2 years ago

SMH and Zendesk responds the wrong way on his post. Guys, just pay the kid instead you're making business owners like me start recommending my friends and clients against using Zendesk...

vednig• 2 years ago

It should be a standard to make bugs public or disclosed to the affected companies atleast a while after it's discovery so at least the companies which are connected to each other can implement their own fix,other wise having closed deals with security researchers behind doors and asking them not to reveal the truth, isn't going to prevent bad actors from using the vulnerability. It is times like these we should reconsider our standard in terms of web security as a whole.

larkinrichards• 2 years ago

Exploit chain--

1. zendesk allows you to add users to a support issue and view the complete issue history by sending a response email to a guessable support email from a person associated with an issue and cc'ing the person to add.

2. Zen desk depends on a spam check for inbound email validity. This check does not appear to catch instances where sender email is spoofed. Zendesk claims this is bdue to DKIM/SPF/DMARC config but I have trouble imagining that 50% of Fortune 500 would get this wrong. There are many automated checks available.

3) Apple issues an Apple ID account to anyone who can receive a verification email Sent to the mailing address (support@company.com)

4) Slack allows you to sign in to a workspace using any Apple ID associated with the workspace domain (e.g. support@company.com)

This researcher reported #2 to hackerone and was declined. Researcher later discovered full exploit with 3 and 4. Did not update hackerone, contacted affected companies directly.

it would have been prudent to update hackerone on the additional finding, but it feels like an easy oversight for a 15 year old after getting rejected on the first round.

Zendesk should take the higher ground and recognize the mistake and correct it. Not get all "ethical mumbo jumbo."

Ylpertnodi2• 2 years ago

it feels like an easy oversight for a 15 year old after getting rejected on the first round.

I'm not 15, but since you ignore(d) me - game over.

valiant-comma2• 2 years ago

Looks similar in concept to this finding from 2017:

https://medium.com/intigriti/how-i-hacked-hundreds-of-compan...

BehindTheMath• 2 years ago

It's linked in the post.

valiant-comma2• 2 years ago

Ahh, I did read the post but missed seeing the link!

gtirloni• 2 years ago

I never understood these technicalities in HackerOne. Oh, you hacked us in a way we said we don't like? Sure, the blackhats will abide.

rthtrjtyjt• 2 years ago

There are countless stories of companies finding ways to not pay bug bounties. At least the black hats get their cut.

kidbomb2• 2 years ago

"Create an Apple account with support@company.com email"

Wait - how is that workflow possible and supported?

In my head, authorization under @company.com would be delegated to a central directory, instead of relying on Apple ID. It is effectively an authentication bypass.

Fokamul• 2 years ago

Big surprise that companies which outsource everything to India and other countries are hacked. No, not really. GL HF

danpalmer• 2 years ago

This matches my experience with Zendesk. We did an integration at my previous company and they were incredibly naive with respect to email setup. We had quite strict requirements around deliverability in particular, and having the Zendesk setup play nicely with our email domain setup that was tuned for deliverability. Sadly Zendesk were unable to engage with the issue beyond "just forward emails from your gmail support inbox" which was a long way behind where we were.

For a company built on email, they're not good at it.

nixpulvis• 2 years ago

My first thought (which is somewhat self-serving because I hate 3rd party sign on) is that this is a great case study for having only one method of sign-in.

dools• 2 years ago

How's that "please keep this between us" request working out for them?

Also I think this is a good reminder that using sequential IDs for anything public facing is almost always a bad idea (or at the very least something you should think very hard about). It would be pretty easy to make ticket ID, for example, a sequential 5 digit number followed by 4 random alphanumeric characters. Everyone could still easily refer to the ticket number, but it wouldn't be guessable.

hedora• 2 years ago

Wait. Did zendesk actually fix the issue, or did they just blacklist a few senders and decide to continue to rely on third party spam filters to save them?

csomar• 2 years ago

You are a sucker if you are using a platform like HackerOne. I reported a bug (Crypto Exchange, Bitmex) 5 years ago. It was not a critical bug, but still. The team kept the bug report open for 5 years; I assume so that it doesn't affect their payout score. They recognized the bug. After 5 years they closed it. Zero communication.

If you value your time and your health, don't use these platforms. If platforms want security, they can hire people by the hour/day and pay them the relevant wages.

system2• 2 years ago

Guess who is going to tell all of his clients about this. Me. I hate corporates pulling this type of crap.

arendtio• 2 years ago

Are you not allowed to discuss a vulnerability you found when it is considered out of scope?

I mean it sounds weird to argue that you won't pay, but do not allow the person to talk about it anyway. Even more so if the person communicated with affected companies instead of some darknet marketplace.

deepsun• 2 years ago

I heard that Zendesk Security team force your _root_ domain to allow their SSL certificates to be issued, per CAA dna record.

I.e. you have domain support.example.com, CNAMEd to Zendesk, so you cannot add any other DNS record to it, but Zendesk should do it on their side. But they refuse, and force you to put CAA to your root domain example.com.

smithstreet• 2 years ago

This vulnerability in ZD appears to be made worse by ZD's Suspended/Spam feature. There is no indication in the spam queue that there is someone CC'd on the email. So an agent may see a ticket from their customer and approve it. Even when you then view the ticket, there is still no indication there is a CC'd third party unless you 'View original' on the email.

cruffle_duffle• 2 years ago

Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.

For some reason this like cracked me up. Sounds exactly like something 15 year old me might say :-).

Let’s build our own ticketing system! How hard could it be?

Upright8905• 2 years ago

I think that the recent third hack of Internet Archive that used the Zendesk ticketing system might have used this bug? Crazy to think about...

1 bug, $50k in bounties, a Zendesk backdoor

Related Stories