Rookie coding mistake prior to Gab hack came from site’s CTO

https://arstechnica.com/gadgets/2021/03/rookie-coding-mistake-prior-to-gab-hack-came-from-sites-cto/

by minimaxir • 5 years ago

280 262 5 years ago

arstechnica.com

forgotmypw17• 5 years ago

I've been pretty sure for a while that Parler and Gab were both honeypots. This makes it seem even more likely.

McGlockenshire• 5 years ago

Never attribute to malice that which can equally be explained by incompetence, or something like that.

adolph• 5 years ago

Never assume xor among different interpretations.

GVIrish2• 5 years ago

Or maybe they just didn't build a strong enough engineering team and quickly ran into the limits of their experience and skills levels.

There are numerous examples of start ups and even mature companies making basic mistakes. This is easily explainable without resorting to conspiracy theories.

btmcnellis• 5 years ago

"solarwinds123" comes to mind.

eternalban5• 5 years ago

maybe they just didn't build a strong enough engineering team and quickly ran into the limits of their experience and skills levels.

"Free Speech platform Gab has announced Fosco Marotto as the company’s new Chief Technical Officer (CTO). According to a blog post from the company, Marotto was formerly a software engineer, production engineer, and developer advocate during a seven-year career at Facebook.

Marotto reportedly brings 23 years of industry experience to the platform along with extensive knowledge in backend infrastructure and insight that will help Gab scale as it becomes increasingly popular."

CTO: https://github.com/gfosco

Parse Server: https://github.com/parse-community/parse-server

Parse Server API doc snippet:

"If your app is compromised, it’s not only you as the developer who suffers, but potentially the users of your app as well. Continue reading for our suggestions for sensible defaults and precautions to take before releasing your app into the wild."

https://docs.parseplatform.org/rest/guide/#security

Per Ars, he removed security code.

JohnTHaller• 5 years ago

Malice or incompetence? Why choose? They've got both!

orblivion• 5 years ago

No surprise to me. At one point when they had an API, I could follow and see posts from "locked" accounts that I couldn't see from the website. I reported it and they said something like "oh yeah we're making a lot of changes right now" and of course never fixed it (until they decommed their API which effectively fixes it).

They're up against the world, and they're spreading themselves very thin, aiming to replace all of the "big tech" use cases, for better or for worse. They hate Silicon Valley but they "move fast and break things" more than anybody, arguably by necessity. I'll cut them some slack if their public platforms have some bugs, but they shouldn't do anything with sensitive user data.

yrgulation2• 5 years ago

The cto should focus on strategic thinking and have one or two devs that lead daily work - and code review for such basic issues (or use a code analyser) to detect sql, xss, xsrf, session management, password based user data encryption, message encryption, and other trivialities. Not that i like gab, but i wonder how many rookie mistakes like this one are then blamed on ”foreign agents” and other media bollox.

gkoberger• 5 years ago

You're right in general, however I think you're over-estimating the size of Gab.

In this case, it's more of a "everyone gets a CxO title because there's fewer than 26 of us" type thing.

claytongulick• 5 years ago

I'm of the general mindset that the CTO should be the most technical.

I think we see a lot of tech company failures when the CTO is just a manager type that doesn't understand the technology and doesn't have the ability to contribute to it.

zelon883• 5 years ago

I'll cut them some slack if their public platforms have some bugs...

Your generosity naively assumes that given enough time they would resolve your concerns. That is never the case with these anti-social social media companies.

It's taboo to talk about, but the correlation between political conservatism and overwhelming amounts of technical debt leading to non-existent operational security is hard to overlook.

carmen_sandiego3• 5 years ago

I’m not sure it’s taboo to talk about as much as it’s just a totally baseless claim.

ugh123• 5 years ago

I think a lot of it also speaks to the pool of quality engineers who are willing to work for a company like Gab, Parler, etc. Companies like that appearing on your resume will no doubt cause friction with would-be employers down the line, for better or worse. I certainly wouldnt work there unless I was absolutely desperate and couldn't look more than a few years down the road.

lawnchair_larry5• 5 years ago

It doesn’t say anything like that. I guess you missed the part about him being a facebook employee. Go ahead, say facebook hires low quality engineers...

user-the-name3• 5 years ago

I'd like to think people would not work for far-right hosts of hatred, racism, sexism and often terrorisms for other reasons than just "it would look bad on my CV". I would hope people would have enough of a moral compass to be able to tell that these companies are, in fact, not morally good.

0x4d464d482• 5 years ago

"move fast and break things"

More like "move fast and break yourself."

NotPavlovsDog5• 5 years ago

This leaves a bad taste as to what the "forensics" could uncover just due to the open development model. It's nice to be the prophet of evident mistakes when the trail is easy to follow, even if you can't exactly master the lingo.

From the article:

   >The change, which in the parlance of software development is known as a “git commit,”

It was a change. Parlance commit. Parlance, per tool used, "git commit" (and then check as to tool standard parlance). My point being, what do we routinely hide thanks to not coding in public? What do engineers routinely hide, when possible?

I would rather, as an engineer, discuss core issues we can fundamentally address: compromising on inadequate workflows (including core architecture and paradigms), commitment to over-delivery, and the ever-dooming deadlines. What victories of the CTO went ignored, as part of "the job"?

It's nice to be smart when you have regular 8 hours of sleep. I've had enough stress to remember just how idiotic many of my decisions were, as a "leader". Most of them went ignored just because we were covered by being invisible, by design. Morally, I can't judge this CTO. If you look at your coding history, can you?

minimaxir• 5 years ago

There's a difference between commiting hacky-but-working code during an Agile sprint and commiting code that allows unsanitized input to a SQL query.

NotPavlovsDog5• 5 years ago

Yes, but then, really, the fundamental popular software paradigm is not just unsanitized but unsanitary. The models for sanitary-by-design are there. It's just math.

   The core leadership behind inadequate decisions, often above the CTO, are frequently of the "don't care about the math, just the numbers" type.

Perhaps the CTO raised concerns. Maybe, not. But if we want an open engineering culture in software, unlike "applied engineering" in other industries, we should actively oppose punishing those that embrace open-to-peer-review models, even when the openness backfires and the history gets removed by the open workflow participants.

We may still have a fragile and unique culture in software, that perhaps contradicts past history such as engineering in construction (look up "corruption construction") or the unique corruption of medicine ("sugar lobby", "food pyramid").

Despite bad decisions and the fumbled cover-up, the attempt to perform in public on their part is valuable to me. We don't have easy access to which of the doctors took money to publish "research" that "calories are the same", pushing for more carbohydrates in diets. This may translate to multiple people, people you might personally know, dying of diabetes.

With open software, we get the names. This should not reward click-bait media witch-hunting.

bosswipe3• 5 years ago

I can 100% judge him. It is the CTO's job to put in place processes and safeguards that reduce the possibility of one of the most common widely known security vulnerabilities. Either he didn't put in the safeguards or he bypassed them, either way it's a fireable offense that put the whole business in danger.

NotPavlovsDog5• 5 years ago

Do you have your commit history available in a public repository? I don't. Honestly, i'm paid for being a professional fuck-up. I just fix things quickly and support my team enough for us to bear the mutual guilt in silence.

cwhiz• 5 years ago

The CTO has only been there since November. No idea what type of situation he or she may have inherited.

However, it looks like the CTO pushed this directly, no PR.

an_opabinia• 5 years ago

commitment to over-delivery, and the ever-dooming deadlines.

Surely the difficulty in recruiting people to work for their shitty website with shitty politics should illuminate for you and everyone also in denial that politics is also engineering.

nicoffeine• 5 years ago

Gab had long provided commits at https://code.gab.com/. Then, on Monday, the site suddenly removed all commits—including the ones that created and then fixed the critical SQL injection vulnerability. In their place, Gab provided source code in the form of a Zip archive file that was protected by the password “JesusChristIsKingTrumpWonTheElection” (minus the quotation marks).

Hopefully this eliminates any doubt on Gab's real mission. It isn't free speech.

eznzt• 5 years ago

I don't understand how someone removing the contents they themselves posted goes against free speech?

Malp• 5 years ago

I believe the parent is commenting on the archive's password and where the company leans ideologically

kevingadd2• 5 years ago

It is always interesting when people complaining about having their speech suppressed turn out to hold (in the former case) extremely orthodox, standard views or (in the latter case) fairly common views, instead of obscure or actively unpopular points of view. (I'd certainly appreciate it if the latter weren't common, but it polls shockingly well in the US right now)

im3w1l2• 5 years ago

That just makes it worse doesn't it? When the suppression has increased in scope to include "standard" views?

danans• 5 years ago

It's also clearly not intended as a password (why would you password protect a public zip archive?), but instead a "yeah you got us, but f** off" kind of message.

It may seem juvenile, but it is the kind of thing people do in an adversarial situation when they have no better hand to play, AKA they "got nothin'".

tzs2• 5 years ago

why would you password protect a public zip archive?

I have no idea why they did it in this case, but I've seen it done to stop overly aggressive antivirus or antispam software from looking inside when the archive contains stuff likely to trigger false positives.

hn_throwaway_99• 5 years ago

On a related note, I've become a gigantic fan of Slonik, https://github.com/gajus/slonik, which simultaneously (a) makes it extremely difficult to expose SQL injection attacks while (b) still letting you write SQL in a very "natural" way using template literals.

It takes advantage of a somewhat-not-well-known feature of Javascript template literals in that you can apply functions to them, e.g. sql`SELECT foo FROM bar WHERE id = ${userEnteredValue}`. No need to manually escape userEnteredValue, the sql template literal function does it for you.

WrtCdEvrydy• 5 years ago

honestly, use prepared statements...

You get the same benefit of "template literals" but the data and query are separated at the DB level. The DB knows what is data and what is query therefore any attempt at escaping from SQL will be quashed there.

byhemechi• 5 years ago

That's what the library does. It just uses tagged template literals[1] so that it "feels" like you're writing a template string, while really it's a prepared statement.

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

wwww4all3• 5 years ago

Look up gell-mann amnesia effect. The article doesn’t know much about OWASP practices.

SQL injection bugs are not rookie mistakes, it’s prevalent in many current and future applications. Look into Vtech sql injection hack, a large company with lots of resources had similar bug.

Look at previous hacks, solar winds hack, Sony hack, were all preventable common hacks.

outworlder• 5 years ago

a large company with lots of resources had similar bug.

Large companies are even more likely to run into this sort of issue.

It's still a rookie mistake. Any given well established company will contain a large number of 'rookies'. It's up to the company and everyone involved to make sure these are caught before going into production.

jahabrewer• 5 years ago

A mistake can be both "rookie" and widely prevalent. That's exactly how I'd describe SQL injection.

wwww4all3• 5 years ago

Go ahead and try to implement a fix to sql injection bug in any of your systems. I can guarantee there’s a sql injection issue somewhere.

The reason it’s so prevalent is because it’s not a rookie issue and very difficult to fix properly, without impacting significant changes.

That’s why OWASP has it as huge part of security analysis and resolution.

xtracto• 5 years ago

You would be amazed at the number of "developers" that have no idea what an SQL injection is when I ask them as part of my Full Stack developer interview.

rsynnott6• 5 years ago

SQL injection bugs are not rookie mistakes

SQL injection bugs of this fairly trivial type are. This is literally what web tutorials were pleading with PHP developers not to do 20 years ago, and they weren't new then.

jusssi• 5 years ago

It's a failure of tooling. The library, or the compiler, should stop you from interpolating stuff into SQL strings. We've already seen things like this keep happening over and over until it's made impossible.

The whole situation is similar to having a construction scaffolding without safety railings, and calling someone falling off it a rookie mistake.

GongOfFour2• 5 years ago

All of this query is safely accomplishable via composable query building in AREL going back to Rails 4.2 (though it's much nice in 5+). Silly stuff.

yrgulation2• 5 years ago

Ok so if in 2021 a “cto” or developer, let alone former facebook developer, doesn't use bound and escaped parameters as a minimum then that developer needs to take a break and catch up on things.

mmastrac• 5 years ago

Is their source tree under AGPL3? From a mirrored copy of what purports to be a clone from before it was pulled down.

https://git.rip/gab/gab-social/-/blob/develop/LICENSE

This will make things interesting for them going forward.

kstrauser6• 5 years ago

Yes. It was forked from https://github.com/tootsuite/mastodon .

stickfigure• 5 years ago

Developers: Sanitize user input

No, don't sanitize user input. Don't trust it, which is a big difference. Use bound parameters and this problem goes away.

acdha3• 5 years ago

A better answer is both not trusting it and validating it: parameters prevent data from being interpreted as code but validation is also important because allowing unexpected inputs could lead to other vulnerabilities (e.g. XSS in an error message or a partially-generated page) or complications when making changes in the future if you learn that common clients have come to rely on your sanitization logic cleaning up their requests.

bcrosby95• 5 years ago

Please no. Trying to validate your input for any format that currently exists or may exist in the future is really wrong. You need to handle this at the edges - for SQL this is using prepared statements, for XSS its about using a framework that makes the correct choice (default escape HTML entities).

Anything else is madness. Of course, if you're dealing with legacy systems you might have to do something tragic like not allow people to put quotes in passwords.

nuclear_eclipse• 5 years ago

Sanitize, normalize, and distrust. All three cover different use cases.

tomc19852• 5 years ago

“Sadly Rails documentation doesn't warn you about this pitfall, [...]" said Dmitry Borodaenko, a former production engineer at Facebook who brought the commit to my attention wrote in an email.

This is completely and utterly untrue.

https://guides.rubyonrails.org/security.html#sql-injection shows examples that are exactly like the code in question in that commit

Bound parameters were a new thing like 15 years ago.

kstrauser6• 5 years ago

Maybe they were new 15 years ago in Rails, but I was using them in Perl in the mid-90s.

There is no excuse for writing an SQL injection in 2021. Zero. None. And if you're in the position to write code that'll be deployed to production, you darn well better have it reviewed by peers before it's merged.

If the CTO did this and worked around the developers, he's a freaking idiot. If the engineers saw this and signed off because he's the CTO, they're freaking idiots. I wouldn't ordinarily be this harsh about it, but come on. SQL injections in 2021? That's astoundingly bad.

rsynnott6• 5 years ago

Maybe they were new 15 years ago in Rails

That's when Rails came out :)

I'm fairly sure ActiveRecord has always supported them.

dragonwriter2• 5 years ago

Maybe they were new 15 years ago in Rails,

Well, Rails was < 2 years old then, so everything was new in Rails.

ratsmack4• 5 years ago

I would think a Ruby code fuzzer would have caught something like that... if they have used one.

btilly• 5 years ago

Bound parameters were a new thing like 15 years ago.

When they were new depends on what language and database library you use.

Perl's DBI had them 25 years ago.

AdamN• 5 years ago

Dear lord I'm getting old ... :-)

tomc19852• 5 years ago

Probably; I vaguely remember them being discussed at my very first programming job around mid 2000's, which was writing Perl

banana_giraffe• 5 years ago

And Visual Basic itself had them 23 years ago as well.

Though, the inflection point, at least in my circles, was in 1998, when rain.forest.puppy talked about "piggy backing" SQL commands.

Even there they mentioned using parameters with stored procedures to protect yourself. It's just clear no one in IIS land did at the time, so there was a wide open exploit playground for everyone that saw that release of Phrack, once they got done playing with directory traversal exploits.

FanaHOVA• 5 years ago

It's like Twitter calling Rails LEGO because they weren't able to scale it :)

GongOfFour2• 5 years ago

The other funny thing is this bit:

Specifically, line 23 strips the code of “reject” and “filter,” which are API functions that implement a programming idiom that protects against SQL injection attacks.

That's not true at all. If you are going to do technical analysis that calls out mistakes, the publication should have multiple technical people review the article.

lilyball• 5 years ago

I had the same reaction, but upon reflection, the reject/filter actions themselves aren’t what’s safe, it’s the usage of functions that accept unsanitized input and safely wrap the SQL that’s going on here.

theptip• 5 years ago

Unfamiliar with Ruby/Rails, why's that untrue? Is it because `reject` / `filter` are just array mapping functions rather than ActiveRecord methods? (Best explanation I could come up with on a quick skim.)

xref• 5 years ago

Come on man, don’t just comment “the article is wrong” then walk away. Put in some effort.

avgDev• 5 years ago

Yikes. I'm working on an internal web app which can only be accessed on our network and even I am using parametrized queries to prevent injection.

What a joke, and then blaming the documentation...just wow.

amatecha• 5 years ago

It looks like the Gab devs followed the documentation precisely then! Just... apparently they copied the wrong part?! heheh ;)

Lukas_Skywalker• 5 years ago

„ Specifically, line 23 strips the code of “reject” and “filter,” which are API functions that implement a programming idiom that protects against SQL injection attacks.“

This is not correct. The mistake was to use ‚find_by_sql‘ without parametrizing the query. The mentioned reject and filter methods are merely skipping some of the data the query returns.

Delk• 5 years ago

The previous code that used those functions probably did prevent a SQL injection as a side effect, as using them avoided making a direct SQL query at all.

But you're of course correct that it's not the replacement of an ORM call with SQL that's the problem.

abvdasker2• 5 years ago

This is bad code and it's a little surprising to me that a former Facebook engineer wrote it (and subsequently became the CTO).

It's been a while since I wrote any Rails but the offenses that jump out just from a cursory inspection:

- large raw SQL query which could almost certainly be accomplished in a more idiomatic way with AREL or ActiveRecord

- no user input sanitizing

- using a regular string literal for a large text block instead of a Ruby here document

- leaving a mess of commented-out code at the bottom of the method

Apparently Gab isn't exactly hiring the best and brightest.

AmericanChopper2• 5 years ago

At a glance my first though was "there's probably SQLi here", but it's not obvious to me that attacker controlled parameters could actually be used to exploit this. The ID and pagination parameters all seem to be ints, and it's not obvious to me that an attacker could turn them into arbitrary strings. Is custom_filters.phrase the attacker controlled string?

kemayo4• 5 years ago

There's nothing in that code that says that any of those parameters have to be ints. They're certainly expecting ints, but there's no casts in that code that would enforce that. Presumably somewhere upstream they're just passing directly from the POST/querystring into these API functions.

Remember: Ruby is a dynamically typed language.

shadowgovt2• 5 years ago

It's always worth remembering that there's a difference between employment and a university degree. Universities don't grant degrees until someone has proved they have whatever competency the degree indicates. Employment means you got paid by the company, but what you did to get paid could be lots of things... And it's pretty easy to lie on a résumé. Universities have a reputation to lose if they grant a degree to someone who goes on to mess up badly; corporations have no such ties to a former employee, and Mr. Marotto making a major error when he's no longer employed by Facebook doesn't reflect poorly on Facebook.

"Former Facebook engineer" means Facebook paid him... No more and no less.

bashwizard• 5 years ago

Not to be that guy but some of the worst security vulnerabilities I've seen has been from senior devs with college degrees. Having a degree =/= you know your shit or being "immune" to absurdly stupid mistakes like this.

You'd be surprised how often I've found leaked API tokens, passwords in plaintext etc on github for instance.

mrunkel• 5 years ago

Universities don't grant degrees until someone has proved they have whatever competency the degree indicates.

Universities grant degrees based on the successful completion of course requirements. I don't know if it's fair to conflate that with what you said. As long as you can get C or better in every course required, you have a degree. Does that necessarily mean that you are competent?

It's an even further stretch if you think that at University cares all that much if one of their Computer Science graduates makes a mistake.

kodah• 5 years ago

Software Engineer and former SRE-SE here; also a two year drop out.

Universities don't grant degrees until someone has proved they have whatever competency the degree indicates

That is not a rule. I have interviewed countless folks with CS and CE degrees that can't get past basic scalability questions in abstract systems design. I've started plenty of hour long interviews only to learn five minutes in that the candidate has a four year degree and cannot correctly identify a use case for binary search nor implement it in plain code.

Universities have a reputation to lose if they grant a degree to someone who goes on to mess up badly

They absolutely do not lose reputation. We don't publish on a website where the degrees came from that allowed Cambridge Analytica to exploit the American people. Everyone will easily write off the offending programmers lineage as a fluke and move on with their lives. Nobody really cares where you came from in this world, just what you're up to now.

Mr. Marotto making a major error when he's no longer employed by Facebook doesn't reflect poorly on Facebook.

I agree that this would be a pretty trash take and a poor precedent to establish long term.

"Former Facebook engineer" means Facebook paid him... No more and no less.

Simultaneously, this is not true. Working for Facebook carries some prestige, as we can see from all the folks who proudly sport "x-Facebook", "x-Google", "x-Netflix" show us. Obviously some people make use of this market and obviously someone is foolish enough to believe the sheen. The value in that perception rests with the viewer; if they hired him because of his work at Facebook then it was significant, if they would've hired him regardless then it matters very little.

tester342• 5 years ago

Universities don't grant degrees until someone has proved they have whatever competency the degree indicates.

haha

you'd be really shocked if you knew about the great lengths people go to get their degree and learning stuff ain't one of them :-)

smoldesu• 5 years ago

I'm not terribly surprised on their hiring practices. I got a Gab account a few years ago for kicks, and I was banned within a week. I never got any justification for my ban, and after scouring my locked account for any offending posts I can only conclude that their moderators are horribly corrupt.

I love decentralized software, but don't let Gab fool you. They're just out to grab power right now.

np_tedious• 5 years ago

Idk when this guy worked there, but today a Facebook product dev likely doesn't get a chance to make a mistake like this. Which is good. But it also means people who've only worked there can have some really surprising gaps

robinduckett• 5 years ago

Why is it surprising to you that Facebook could hire someone inept?

abvdasker2• 5 years ago

I mean I did say it's only a little surprising.

ufmace2• 5 years ago

Have you ever worked on any really large platforms or databases? IME it's pretty common that the idiomatic Arel/ActiveRecord code you might normally write can become hopelessly slow when the query gets complex and the datasets get large. I'm not trying to pick on ActiveRecord here, pretty much all ORMs I've ever used will do this sometimes. The only way to get the speed up to something acceptable may be for someone who knows SQL and the DB Schema well to write a custom SQL query that makes the most of the database indexes and optimizations.

When a large product in Production is having performance issues, it's very possible that you'll write something like that in a time crunch to get the site back up. The code might not be the cleanest possible because you aren't sure how well it will work yet. It might be checked into Git because any remotely sane deployment procedure requires it. And maybe somebody years later will pore through the Git logs and come back and drop snark on you without knowing anything about what was involved in writing and releasing that code and they've never had to deal with something like that.

bsder2• 5 years ago

People like to slag on other people's programming skills, but are you sure that all code you wrote doesn't have some absolutely boneheaded mistake in it?

I like to think I'm pretty good, but I would never claim that and certainly wouldn't put money on it. I'm sure I've got boneheaded mistakes scattered through my code. Sometimes you're having a bad day. Sometimes the reviewers are a little pressed for time. It doesn't take much for really stupid code to slide through as long as it appears to work.

Glass houses and all that ...

TameAntelope• 5 years ago

I've completely given up judging technical decisions as representations of the person who made them's skill or ability.

I still judge the technical decision on how it's impacting the system today (specifically around whether or not it's sustainable), but trying to ascribe intent or brand something as "dumb" is almost always done without the relevant information that went into the decision being made.

bsder2• 5 years ago

This is the problem being in a field (programming) that doesn't actually value experience.

I got this knocked into me by a very senior engineer when I complained about some really silly circuit on an older version of a microprocessor. His comment: "Okay, let's design that adder better. Oh, by the way, you only have two layers of metal in that technology. Let's walk through the options." My response: "Uh, no. We don't need to. There are exactly 3 options and only one that works." His response: "And now you know why the engineers prior to you made that decision. Don't assume your predecessors were untouchable geniuses, but don't assume they were moronic idiots, either. Just like you, they had a job to do, did it, and, most importantly, shipped it."

danaliv• 5 years ago

I am 100% sure that none of the code I’ve written in the past 20 years interpolates user input into SQL. There is absolutely no excuse for this, and there hasn’t been for a very long time. Prepared statements have been a thing for longer than half the users of this website have been alive.

Remember Bobby Tables? That was 2007.

vntok• 5 years ago

This seems like hubris, although you might be correct. Is there a repository of your code that we could download and run checks on?

0x4d464d482• 5 years ago

I've had some bone headed and careless mistakes make it into production, especially when I was a junior dev, but it was always something that messed up performance or functionality, e.g. a front end widget not responding or an unhandled API parameter.

When it came to user security I _never_ fucked around. I have about 6 years of experience now and consider a trivial and catastrophic mistake like this unconscienably reckless. Never mind 20 years and experience working at a FAANG company.

gfodor• 5 years ago

Honestly you have to go pretty hard against the grain to do SQL attacks.

However, I believe in blameless postmortems, even for CTOs. I'd feel awful to see this in the press.

ulisesrmzroche• 5 years ago

The irony tastes like candy. More proof that we totally live in a simulation.

015a2• 5 years ago

Fuck this story, and shame on ArsTechnica for publishing it.

We shouldn't call out and shame individuals for making a mistake like this. Any company which even does something like this internally is toxic; the blame lies on the company and the process, not the individual. The flimsy excuse "oh, theys a bawd bawd company filled with icky people" doesn't hold water; we're talking about basic decency here.

You want to write a story about the commit? Hold the company responsible? Encourage processes and procedures which would have caught this? That's fine. Blur out the names. Ars isn't a tabloid, but they're acting like one; there are real, serious consequences for shit like this. You didn't set out to create the perfect ammunition for someone within a company arguing against the open sourcing of some project, but here it is.

I'm physically disgusted by this reporting.

ratsmack4• 5 years ago

Remember that ArsTechnica does reporting for money. The more sensational, the more clicks.

I think National Inquirer and The Sun uses the same tact.

uniqueid• 5 years ago

If I were a journalist who wanted to rake in money it wouldn't occur to me to include dull screenshots of diffs and commit messages. For that matter, it wouldn't occur to me to work for a niche outfit like Ars Technica either. My dentist probably earns more than a tech columnist.

smt883• 5 years ago

What about when people went after the security head of Equifax following their massive breach that ruined a few lives? Was that unfair?

spydum• 5 years ago

Incredibly unfair, the character assassinations are totally uncalled for. The infosec community has a real problem with tearing each other apart. So much so, that journalists now also think it's okay. * Just a reminder, they were tearing into the CISO for having a liberal arts degree (as if college degrees actually prepared you for anything..).

Totally agree here: blame the company and the leadership for not investing in the right processes and tools. People will always make mistakes.

themacguffinman4• 5 years ago

I think it's unfair. A single person doesn't bear the whole responsibility of a breach like Equifax's. The breach is just the final symptom of deeper problems with company leadership & policy allowed to fester by ineffective public oversight and bad incentives. The witch hunt will change nothing and absent policy, it will probably happen again because the specific people were never the problem.

slickrick216• 5 years ago

Deeply unfair bordering on sexist and elitist. People mentioning her degree was one of the most nasty things I’ve seen.

dj_mc_merlin3• 5 years ago

I was of the same opinion as you before reading the article.. but really? Genuinely just putting user input in SQL queries? I mean, everybody starts somewhere.. but how the hell do you get to be CTO of anything when nothing goes off in your brain when you're interpolating raw user input into SQL queries? WHAT? How? Everyone slips up, concatenates strings when they shouldn't, whatever.. but this genuinely looks like he didn't know that SQL injections are a thing. That is unacceptable for a technical leader.

a012• 5 years ago

In my book, any C levels shouldn't have direct access to production, and espectially when you become a CTO you stop coding.

blabitty• 5 years ago

Maybe it wasn't a mistake

millzlane• 5 years ago

I barely know much about programming. I have finished LPTHW years ago but I'm not a coder by any stretch. Even I know to google how to sanitize user inputs before putting something into production.

pillefitz• 5 years ago

Managing people for the most part, helping out because staff got sick, writing parts of the code between meetings, being interrupted by your child.

user-the-name3• 5 years ago

Suffice to say, if you are remotely competent, you don't go work for far-right assholes.

So you get to be the CTO but actually being an awful enough person to put up with the company.

markdown2• 5 years ago

On the contrary, more of this please. Personal accountability is sorely lacking these days, from the heads of Fortune 500 companies getting away with golden parachutes after ruining countless lives in 2008, to shit like in the Ars story.

Sure, hold the company responsible, but don't just let the person get away with it so they can run off and do the same at another company. Crooks and arseholes go from company to company causing damage everywhere they go because firing companies won't tell anyone while they let a person go. This seems to be a result of just that culture of silence, because how else can you explain someone becoming CTO of an org like this without know basic stuff about their own area of expertise?

When minions make mistake, let the company take the heat. But when the person screwing up is in leadership and therefore directly responsible, it ought to fall on them.

markdown2• 5 years ago

Here's another example from today of a PM who moves on after screwing US taxpayers: https://news.ycombinator.com/item?id=26336252

kstrauser6• 5 years ago

You didn't set out to create the perfect ammunition for someone within a company arguing against the open sourcing of some project, but here it is.

One important note: Gab is currently running a fork of the Mastodon software, licensed under the AGPL-3.0. It's not their right to decide whether they want to publish their source code.

sokoloff3• 5 years ago

It is however their option whether they want to squash, rebase, and/or rename committers when they share the source code as they are obligated to do under AGPL.

shp0ngle• 5 years ago

They do provide the source code. Just not as a git repo.

sokoloff3• 5 years ago

At our company, we would not blur/obfuscate the actors' names for internal communications about technical errors.

The key is that humans make errors, errors of commission, errors of omission, errors of misplaced/almost brilliance, and errors of outright apparent stupidity. IMO, you should dive deeply to understand exactly what happened (which includes who did what and why), but you don't use that as ammunition against the individual humans.

Pretending that nameless, faceless committers did this thing isn't as useful as "even our CTO can make a stupid error [and not get fired for it]; here's how we seek to eliminate this entire category of error reaching production: <1> <2> <3>"

themacguffinman4• 5 years ago

Internal communications are pretty different from a news article. A company can enforce a culture & policy for internally discussing mistakes. Publishing details in a popular outlet like Ars Technica opens up individual employees and the company in general to unproductive and unfair polemic, especially if the company in question is already unpopular for other reasons. It's no surprise that few companies open up their internal communication to public scrutiny.

lmm• 5 years ago

We shouldn't call out and shame individuals for making a mistake like this.

For a random line worker, sure. For the CTO, the position is absolutely relevant. You can't have a conversation about what processes or procedures might have caught this without taking that into account; in every small company I've worked for the CTO was either tacitly or explicitly able to bypass any amount of code review etc..

The fact that it's the CTO is informative, educational, and important, and should be reported. Their personal details are irrelevant but hiding them would also be meaningless.

jdxcode5• 5 years ago

Yes, and I'd argue this was beyond a simple "mistake", it's borderline negligence for anyone but a very junior engineer: https://archive.vn/oxbck

EDIT: that link isn't working, but here is a screenshot: https://imgur.com/4lyElZI

wwww4all3• 5 years ago

It’s easy to be armchair sql injection expert and point fingers.

I can guarantee that any system that you’ve worked on has numerous OWASP security bugs. You’ve probably looked at the bugs countless times and never noticed it.

Every software engineer of all levels has overlooked obvious sql injection bugs in their code base. Most likely you’ve added to the bug list.

Software bugs are simple part of any development effort. All major companies, Microsoft, Google, Facebook, etc. has very simple bugs like this in their systems.

That’s why they pay out bug bounties, it’s cheaper for them to add the bug and have some random security researcher find the bugs for them.

jollofricepeas• 5 years ago

Yeah most management books (See: Radical Candor) say that executive’s should own up to their mistakes publicly and receive public criticism so that it makes it easier for subordinates to give criticism and even receive it themselves privately.

On another note, how do we know that the new CTO of Gab didn’t just do this on purpose so that there would be a breach?

oji0hub2• 5 years ago

Even a small company has a huge amount of things going on at the same time. Any one thing in isolation is simple, but put together they amount to a huge amount of work where you can't put the time needed into become an expert on everything.

That applies to anyone any employee, including the CTO, especially so. They're just people.

Publishing an entire article to shame them is just disgusting. Like, why? The reporter if anyone deserves being shamed for apparently making a career of flinging shit at people.

qzw• 5 years ago

... the blame lies on the company and the process, not the individual.

What about when the individual is responsible for establishing the processes? And when you blame "the company," are you not ultimately referring to those in charge, i.e., the C-level executives? After all, don't these same top executives claim a large share of the reward by arguing that their positions carry more responsibility and risk? So it seems entirely fair that they should expect to be held personally responsible when there are failures under their watch.

justapassenger3• 5 years ago

CTO of a hot company, knowingly exposing user data (only 1 year CS students can claim that they don’t know that you have to sanitize any input into hand crafted SQL queries) totally deserves a public scrutiny.

0xy5• 5 years ago

You're using "knowingly" wrong, unless you can prove intent. You're also totally incorrect, considering how many companies have been hit with SQL injection vulns.

just_use_eval• 5 years ago

I mean ArsTechnica is owned by Condé Nast, which in turn owns Reddit. Reddit is very anti-conservative, and has been this way since at least 2014. It's not far-fetched to think that maybe Condé Nast, the parent company is lead by people with similar or compatible values, which in turn affects their other platforms as well (such as ArsTechnica). Another thing to consider is that Gab is sort of a competitor to Reddit, so why wouldn't they fight it.

Reddit investors: "Reddit raised $50 million in a funding round led by Sam Altman and including investors Marc Andreessen, Peter Thiel, Ron Conway, Snoop Dogg, and Jared Leto."[1]

- Sam Altman is an open democrat and has donated 250k USD in support of the democratic party.[2]

- Peter Thiel is pro-LGBT and pro-Trump.[3]

- Snoop Dogg is known for liking (and singing about) weed.

I don't know about you, but this explains Reddit's front-page and their overall content policy quite well. There is not as much information about ArsTechnica as there is about Reddit, but if we do some research I think the realities of ArsTechnica's potential biases will reveal themselves as well.

[1] https://en.wikipedia.org/wiki/Reddit

[2] https://en.wikipedia.org/wiki/Sam_Altman

[3] https://www.nbcnews.com/feature/nbc-out/openly-gay-libertari...

ramraj07• 5 years ago

Take a step back and consider for a moment that these coders replaced their gitlab repo with a zip file with the password JesusChristIsKingTrumpWonTheElection; sounds more like over grown kids who never got an adult lesson run this site so a bit of name calling at the CTO level is warranted.

rodgerd• 5 years ago

We shouldn't call out and shame individuals for making a mistake like this.

The first response of the GAB exec team to a bug which has exposed all the private information to being leaked was an expletive and slur-laden attack.

I am quite comfortable with the CTO being treated the same way he is happy to treat others.

emodendroket2• 5 years ago

He's literally the CTO. The buck stops with him. If the procedure's a mess, guess whose fault it is?

cyansmoker• 5 years ago

Hold the company responsible?

Technically, the CTO is an officer of the company. In a very real sense, he is (a piece of) the company. Furthermore, how would one "hold the company responsible?" -- review their security culture? Where do you think that culture comes from?

shalmanese• 5 years ago

Yeah, we should not blame software engineer Fosco Marotto for introducing a rookie error into the site, we should blame the person in charge of setting up standards and processes so such errors never make it into production, Fosco Marotto!

ggggtez2• 5 years ago

We shouldn't call out and shame individuals for making a mistake like this.

Yes we should. This individual is the company. He's the CTO. "Chief" Technology Officer. The buck has to stop somewhere. How can you possibly suggest with a straight face that he should not be held professionally responsible for the code that:

A) He personally wrote B) He is responsible for professionally as the CTO

ReactiveJelly• 5 years ago

oh, theys a bawd bawd company

wow is that what I sound like

ggggtez2• 5 years ago

You also seemed to miss this part of the article, which is very telling beyond just shaming someone

Besides the commit raising questions about Gab’s process for developing secure code, the social media site is also facing criticism for removing the commits from its website. Critics say the move violates terms of the Affero General Public License, which governs Gab’s reuse of Mastodon, an open source software package for hosting social networking platforms.

Gab is likely breaking license agreements by performing this sort of cover up. So yes, this is newsworthy.

Karunamon• 5 years ago

Correct me if I’m wrong, but you don’t need a link to the code to comply with the GPL. Providing it on request is still kosher.

015a2• 5 years ago

Gab is likely not breaking license agreements, as long as they provide it upon request. Which was never tested.

That's the problem with reporting like this; it jumps to conclusions to justify outrage, while clearly being written by someone who is overworked, underpaid, underexperienced, and only has a cursory understanding of the systems at play.

Of course, most people are underexperienced in domains like this, so they rely on traditionally high-quality reporting sites like Ars to provide accurate, precise, and fair reporting. The wording Ars used is accurate; your interpretation of their wording isn't. That's the responsibility journalists have; what they say matters, a lot, because people interpret what they say in different ways.

Ars said that they're "being accused" of breaking license agreements. You said they're "likely" breaking license agreements. That's a world of difference; if for no other reason than because one is correct, but useless, and the other is (probably) wrong.

chovybizzass• 5 years ago

I got fired from people.ai for having a bug slip into production. I had put it behind a feature flag so only our employees could see. Still got fired.

29athrowaway• 5 years ago

The CTO of an org should be technically competent. If you are not technically competent, you have no business being a CTO.

If a civil engineer makes an egregious mistake and a bridge collapses, it's not toxic to say an egregious mistake was made.

And, in an organization, if you are unable to recognize and call out mistakes (in a respectful, constructive manner that is), you're a failed organization.

Rookie mistake is certainly not a constructive way to report on this problem, but it is true.

atsushin• 5 years ago

Normally I'd be inclined to agree with you on this, your bottom-line software engineers shouldn't be outed and publicly shamed. In this case, however, the CTO, a person at the top of the org chart, must be cognizant of these sorts of vulnerabilities especially at a company with a growing userbase and a target on its back (mind you I don't support Gab or anything they stand for, it's just the reality of things). More importantly, you have the CEO actually, really trying to scapegoat the incident on 'mentally ill <t-slur> demon hackers' when its their own incompetence in the top ranks to blame. It's an exception to the rule.

eric4smith• 5 years ago

You know what's really a shame? People overusing the word "toxic".

You used it here a couple times in here so far.

I've been on the wrong end of a SQL injection once and I was called out publicly multiple times for it to my endless shame from my customers who bought the software and were hacked.

Was that toxic for them to call me out?

Heck no. In the 15 years since that time, I made triple-sure to never code something that could lead to a SQL injection again.

climb_stealth• 5 years ago

To be honest that situation you have been in sounds horrible. I would not wish that on anyone.

There's this tweet floating on the web which this made me think of. Something to think about I guess.

If you have suffered in life and want other people to suffer as you did because you "turned out fine", you did not in fact turn out fine.

doggydogs94• 5 years ago

I agree with you regarding the tabloid journalism aspect of the ArsTechnica story.

jpeg_hero• 5 years ago

ultimate engineer entitlement: we are entitled to make $500k per year and not be accountable for our mistakes.

gfosco2• 5 years ago

Hi HN! Great feedback here, as always.

I can talk about the commit, and I can talk about the article.

We were getting crushed with scaling from 3M to 30M visits a month, and every day was a 16 hour day. The site was falling over on a daily basis, and the performance of the home feed was atrocious. The APM traces were full of loops. The muting and blocking features were incurring tons of additional queries. As I remember it, one night I decided to write a SQL query to replace it. I believe I thought the values were already being sanitized, but I have no problem agreeing that I should've looked into that further and made sure. I'm not a rails dev, and I'm generally negative towards rails and ActiveRecord. I wrote SQL for many years in the past, and as the folks who went through my stackoverflow saw, I am not unaware of the importance of sanitizing user input. I've written much code that properly sanitized user input, in various languages.

It's really easy to imagine that Gab is a large company, considering the breadth of their platform and products, but you'd be shocked and impressed. It's still really early days, like this is only Chapter 2 for us.. we're growing and will keep getting better.

This article is interesting to me for other reasons though: 1. there's been no confirmation this commit was related to anything that has happened. and 2. the person who reached out to Ars with the story, a link to the commit and a quote, is a former co-worker with a personal grudge. I can only imagine the glee with which he wrote that email.

For the last 2 days we were being extorted for nearly half a million dollars. I received a death threat today, first one ever. I think we're doing something important. Go ahead with your code critiques.

Have a great day,

Fosco

marcus_holmes• 5 years ago

I've been on the other end of a journalist with a grudge, and it's not something I'd wish on anyone.

It's interesting that the technical language in the article is so clearly mangled. Either Ars Technica has no editors that know how to edit (unlikely), or that know the basics of git (again, it seems unlikely). It reads like a journalist taking notes from a dev ("uh-huh, a 'git', right. Oh, no a 'git commit'. really?").

heyitsjlac3• 5 years ago

Diddums the people using your site send thousands of death threats on it every day. Try to toughen up, buttercup.

lioeters• 5 years ago

former co-worker with a personal grudge

That explains a lot, and shame on ArsTechnica for enabling that person's petty revenge.

Anyone, including a CTO, can make mistakes and introduce accidental vulnerabilities. It's not productive nor healthy as a culture to blame someone for being imperfect.

The best thing to do, as a company, is to learn from such a mistake, and improve processes to catch and fix it earlier, before it gets released.

It's disappointing to see Ars participate in a low-ball personal attack, calling out a "rookie coding mistake". They're encouraging a toxic mindset.

ufmace2• 5 years ago

Thanks for the honest description of what really happened there. I've also had the experience of needing to rewrite a query that used a bunch of safe ORM statements but was painfully slow into a big, messy raw SQL query that was probably less safe in order to restore performance. It's easier than many would think to miss whether or not some check was performed somewhere along the line when it's in the middle of a complex process.

Meanwhile, Ars still has a few decent articles/departments, but it seems like 50% of their content is tabloid-grade cheap and mindless bashing of anyone who disagrees with the elite mainstream opinion. Probably like 80% of the mainstream media right now is even worse.

I wonder how Ars would fare if somebody was to tally up all of the boneheadedly stupid mistakes they've made in journalism, grammar, spelling, site administration, etc, and publish articles trashing the specific employee who did it each time.

msmaryjane2• 5 years ago

Hi Fosco! Thanks for this breakdown. It’s helpful to know that you’re still incapable of accepting accountability. Can we assume from your cagey “there’s been no confirmation,” defensive sign off, and immediate retreat from transparency as confirmation that you did indeed shit the bed? Maybe you got a little rusty after 5+ years of Facebook paying you to post garbage on Workplace.

And don’t give too much credit to Dmitry - he didn’t think of you differently than any other fascist white supremacist. You earned this coverage through the sheer force of your own incompetence.

Best of luck to you all heading into Chapter 3! Can you give us any exclusive hints as to what’s coming? I think it’s even odds between a wave of mass shootings and being overwhelmed by child porn.

m463• 5 years ago

Step 1: assign blame.

ugh.

postalrat• 5 years ago

Maybe programmers should stop using the word "engineer" to describe themselves if they don't want to hold responsibility.

fock• 5 years ago

well, even if it's rightwing idiots it seems you can go relatively far. We just deleted all our labs data because a colleague didn't get that you had to unmount and destroy the existing (benchmark) zpool to create a new one when finally setting up some new drives. Didn't ask for help or anything but went straight to "fix" it with `nvme format` on the raw devices. Hope the backup works at this point (that's the other colleague who accused me of paranoia when I mentioned 2FA with yubikeys for admins and then silently removed his smart outlet from the network once I ran a powercycle on it...).

__abc• 5 years ago

Oooooof. Rails (which I think they may be using) makes sanitization fairly easy.

naebother2• 5 years ago

Besides the commit raising questions about Gab’s process for developing secure code, the social media site is also facing criticism for removing the commits from its website. Critics say the move violates terms of the Affero General Public License, which governs Gab’s reuse of Mastodon, an open source software package for hosting social networking platforms.

Lmao, that's not how AGPL works. Although, it would be pretty funny if a license explicitly required you to post all your Ls for public ridicule.

dragonwriter2• 5 years ago

Lmao, that's not how AGPL works.

The GPL family, including the AGPL, defines source code that must be provided as “the preferred form for making changes”; if the form the distributor uses is a VCS repository with full history, publishing either static snapshots or a repository with an expurgated history could arguably be a violation of both the letter and the spirit of the licenses. The language of the licenses makes the requirement dependent on actual development practices, not fixed, and that's not an accident.

naebother2• 5 years ago

I'm guessing you're referring to the first sentence in section 1:

The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work.

I don't think that's referring to the method by which I make changes or manage those changes. It's simply trying to make it clear that source-code is something that should be in a form you can actually change, and not something obfuscated. e.g. minified, transpiled, object-code, etc. The second sentence gives the first more context.

The spirit of all GNU licenses is that the end-user should have the freedom and ability to understand or modify software that's *distributed* to them for themselves. It has nothing to do with having a full history of the source-code available. What does full history even mean? I'd like every auto-save to every LGPL licensed file, please.

trhway• 5 years ago

Looking at that mess of a code (even without the cherry top of injection) I kind of wonder why ruby is hip and php isn't.

heyitsjlac3• 5 years ago

The dude who got a tummy ache from this was even funnier than the story itself. This is a great site.

Rookie coding mistake prior to Gab hack came from site’s CTO

Related Stories