“Please don't waste maintainers' time on your KPI grabbing patches”

https://lkml.org/lkml/2021/6/18/153

by belter • 5 years ago

517 253 5 years ago

lkml.org

coldacid• 5 years ago

Looks like LKML purged all mail of June 18?

_joel• 5 years ago

The thread is on the left, up to Jun 22.

jabedude• 5 years ago

What does "KPI" mean in this context? Never heard the term before

wrren• 5 years ago

Key Performance Indicator. The implication is that some Huawei engineers are rewarded for contributing to the kernel.

hprotagonist• 5 years ago

Key Performance Indicator: ie, someone at huawei’s job performance review depends on “made N open source contributions to major projects”, and Goodhart’s Law strikes again.

travisgriggs2• 5 years ago

Came to the comments to learn what KPI was. And leaned about Goodharts law: https://en.m.wikipedia.org/wiki/Goodhart%27s_law

adreamingsoul• 5 years ago

key performance indicator? kernel programming interface?

I too ponder the meaning.

GonzaloQuero• 5 years ago

"Key Performance Indicator". I believe the implication here is that some people from Huawei are sending useless patches just to pad their performance values and look better inside the company.

topranks• 5 years ago

Indeed... although I'd point the blame more squarely at management deciding that "number of patches" is the right way to measure someone's performance.

Not that I have any idea if that's the case here, but I've worked in places like that. Idiotic approach.

tut-urut-utut2• 5 years ago

I see that these "cleanup" patches are not bringing much value, bust I also don't see why fixing spelling mistakes or log messages is considered as harmful, KPI boosting or not.

The maintainer even said if someone else sent those patches it would be OK, but not if Huawei employees do it. If they distrust Huawei so much, why not just ban them from committing, the same way they did recently with university "security researchers".

exdsq• 5 years ago

I guess if you have hundreds of trivial patches it'd take ages to review and merge them, and due to kpis Huawei are being deliberately inefficient

marcinzm• 5 years ago

Is this an issue with the workflow that the kernel uses in terms of time spent per patch? In most workflows this seems like it'd take under a minute to review, approve and merge each one although the CI/CD might add some computing cost.

ma2rten• 5 years ago

These small patches create some overhead for maintainers. He actually said in the follow up message that grouping them together is fine.

Please at least merge all those small fixes into a larger patchset, and with a good cover letter to explain the reason (and auto-tool to do the change if possible) for all the involved maintainers, so that all of us are on the same page.

hundchenkatze• 5 years ago

I understand the overhead, but it seems like bundling lots of unrelated cleanup fixes into one large commit would make it easier to sneak in something nefarious.

ThaDood• 5 years ago

Right? I am not really sure I see the problem. It seems to me like Huawei is just submitting clean-up patches and the maintainer has higher expectations of what patches they should be submitting. Granted I am not a developer not a maintainer of any project so that might be why it is not obvious.

nneonneo3• 5 years ago

If there’s a pattern of them submitting a ton of these kind of patches, that could be viewed as gaming a metric. Huawei might, for example, be assessing kernel developer performance partially on # of upstreamed commits, and I guess that’s what is being suggested here.

moftz• 5 years ago

Doing as many patches as you can may make the approvers fatigued and more likely to approve something that is bad or malicious.

ridaj• 5 years ago

It's not about security. It's about wasting reviewers' time. I too would be annoyed if someone was submitting, say, whitespace-only changes, or similar cleanup, non-functional changes - I still have to review them and my time would be better spent looking at actually meaningful changes that make the product better. It's OK if someone is just learning the ropes with an easy change, but this seems to be more like people trying to game a metric by submitting lots of small changes, which wastes the reviewers' time for no actual user benefit.

bombcar• 5 years ago

I think the proper way to do a “cleanup” patchset is to communicate with the maintainer beforehand and ask them what they’d prefer

41209• 5 years ago

I think there was some medium post by a guru where he claimed the best way to get experience was to submit as many spelling error pull request as possible. That way he can throw in his resume that he's contributed to a dozen or so open source projects. Given how broken the hiring process is, it wouldn't surprise me if this works

treesknees• 5 years ago

There is a certain balance to this however. Where I work we generally have a rule of no formatting changes mixed in with real code changes to make reviewing easier. It's a huge pain to see a ton of white space fixes mixed in with logic changes. As long as they're separate commits in the same pull request it's usually OK. We also generally don't have formatting-only pull requests because it just adds overhead and indirection to git-blame, and doesn't add value to the end product.

viraptor2• 5 years ago

It's likely the context of available time and resources that matters here. Huawei has the ability to contribute a lot, but instead sends an experienced, highly paid person around to hammer in a nail that was sticking out slightly and calls you over (when you could be doing something more productive) to high five a job well done. This would be appropriate for an beginner apprentice.

Yeah, it might've been slightly annoying to someone, but it's a visible waste of time overall.

da39a3ee3• 5 years ago

When people make trivial typo PRs on my open source projects I say thanks very much, close their PR, and make the change in a commit under my name. This way, if their interest is in improving the project then it worked. And if their interest is in contributions then it works also because it doesn't pollute their contribution history with trivial changes that aren't real contributions.

theli0nheart2• 5 years ago

That’s incredibly disrespectful. With that approach, I would be shocked if anyone would want to contribute to your project again after such behavior.

duxup3• 5 years ago

I think the concern here is that at huawei there's some kernel patch metric, or incentive, or something like that. And following Goodhart's law... "When a measure becomes a target, it ceases to be a good measure."

So now there's a target that costs maintainers time, possibly lots of time, and costs huawei nothing.

There's really no control here to dial it back other than to ask 'please don't do this'.

phkahler• 5 years ago

> I see that these "cleanup" patches are not bringing much value, bust I also don't see why fixing spelling mistakes or log messages is considered as harmful, KPI boosting or not.

My best guess: It creates a small amount of work for a maintainer to review and merge. This is worthwhile if a new contributor is learning their way around, and getting new contributors is very important to an OSS project. To have a large company submitting a bunch of these is a distraction for maintainers who have better things to do than support someone trying to inflate their metrics.

tut-urut-utut2• 5 years ago

It's true that it creates an overhead for maintainers. But, on the other hand, maybe Linux process needs to be improved to relief the maintainers from having to review every single patch.

Linux as a project is big enough, and if maintainers are the bottleneck, maybe it's time to have sub-maintainers to whom such tasks could be delegated.

jacquesm• 5 years ago

The easy solution would seem to be for the person accepting the patch to check a box that says 'this patch materially improved the codebase'. That would put an instant stop to gaming the KPIs.

sys_64738• 5 years ago

In general fixing spelling or logging errors are P4 or P5 at best so they should only be fixed if something more important is touching those files. That how I've always viewed it.

CivBase• 5 years ago

Looks to me like the problem is not the content of the patches, but the way they are submitted as many small patches all from the company. I suspect the maintainers wouldn't be as concerned if all of the contributes were independent (no "@huawei.com" address) or if they bundled the patches together to reduce overhead.

powerapple3• 5 years ago

I would agree with you. Those commits would be the easiest one to merge. How difficult is it to approve a typo correction? Saying a typo correction is not productive is actually the opposite: it takes very little time to review and approve. I don't know why people are mad that these commits are not from a student.

mustacheemperor• 5 years ago

The dialog between the author of this message the responses on the mailing list add some clarity. There's nothing inherently wrong with providing cleanup fixes to this kind of project, but submitting those as individual merge requests instead of one grouped patch is just serving to increment the developer's "number of patches submitted" KPI at the expense of the time spent by maintainers reviewing every individual patch.

hashhar2• 5 years ago

Without any context it looks like someone overreacted. It's essentially saying anybody else than Huawei sending cleanup patches is welcome but Huawei is not. Then they try to backtrack that statement by putting out a long list of things which aren't comparable in complexity to the original topic.

Without more context I bet this thread is going to go off-topic.

MichaelRazum• 5 years ago

Sounds like he is saying. Huawei is doing a lot of trivial cleanups to be able to say like they are a major contributor. Basically the argument is:

- trivial patches are ok, if you are just doing it for practice. Like a student.

- trivial patches are not ok, if you are doing a lot of them and only for the purpose to get a higher contribution ranking (like commits per company)

szszrk• 5 years ago

For me as well. This discussion plus recent 'kernel should use github or similar" articles recently makes me think about sponsors.

Doesn't Kernel use sponsors approach, like Debian used to for packages? You can push anything, but you find yourself a competent maintainer who filters your work. Like puts your trivial commits in bulk ;)

hashhar2• 5 years ago

I don't think Qu says that Huawei isn't a major contributor.

See:

> My contributions to the kernel in the past have mainly been on optimizing the performance of the ARM64 SMMU driver, > including the iova optimization, strict mode optimization, and the lazy mode optimization. Also working on the > development of some ARM SoC drivers.
You indeed have done solid contribution to the kernel in the past, thus better could have been done.

Also:

Even without checking the git log, I can easily think of some big contributions from your employer, like EROFS and F2FS. Thus I don't have any doubt about that.

buran77• 5 years ago

The remark close to the end, that this is just further damages Huawei's already damaged reputation, especially after beginning with "I did the same at some point", hints at a personal issue rather than professional objection.

There's no professional reason to take such an unrelated jab so it feels more like this individual just found an opportunity to settle a score, or has a different problem with Huawei that's harder to support publicly.

pphysch• 5 years ago

Yep. This is incredibly petty and unprofessional from the reviewer. Keep the ridiculous geopolitical slapfights out of the Linux kernel.

dangerface2• 5 years ago

I think the missing context is that Huawei is one of the largest technology companies in the world, every thing they do is built on linux, improving it would improve their business but their contribution to the project is a spell checker. Ok but clearly not their finest work.

NSA has implemented backdoors in software through bug fixes that appear to be just benign typo or spelling fixes but actually allow an exploit in combination with some other exploit etc. To any one accepting the patch it just looks like a typo fix maybe a little odd but nothing dangerous.

Huawei have the people with the skills to do really impressive work but they spend that time and resources fixing typos, this is a little odd.

ptrkmhms• 5 years ago

[Citation needed]

NSA has implemented backdoors in software through bug fixes that appear to be just benign typo or spelling fixes but actually allow an exploit in combination with some other exploit etc.

ineedasername• 5 years ago

The context is lack of consideration for people's time.

To use an analogy, imagine you wrote a proposal at work and asked for feedback from your boss. Instead of a single substantive response, they send back a few dozen individual emails, each one a comment on word choice, or font size, or a suggestion for a paragraph break. Even if each one of them might have some minor merit, it was done in the most time wasting way possible, maybe because their boss measures their work output by how many emails they send.

Huawei isn't the kernel boss, but they are essentially submitting feedback. Qu is saying that, for a company the size of Huawei, one that is massively reliant on Linux and has developers' time dedicated to it, he expects them to make their own submissions in a less time wasting fashion. He also suggests the proper way to do it.

matheusmoreira3• 5 years ago

https://youtu.be/fMeH7wqOwXA

If you're hired to do this stuff, you're on your own, you better know what you're doing.

Huawei pays professional developers to contribute to the kernel. Of course they should be held to a higher standard. They should be working on something substantial, not fixing minor problems. Surely there are far more important things to work on. Failing to prioritize important issues coupled with incentives for kernel contribution means they are putting in minimum effort for maximum personal gain at the expense of maintainers.

burnished2• 5 years ago

can you quote the part that supports

>It's essentially saying anybody else than Huawei sending cleanup patches is welcome but Huawei is not.

? I don't think you can because the article does not say that.

quyleanh• 5 years ago

"you're really just damaging the already broken reputation."

Actually, with this email and the viral on HN, I think this quotation is true for Qu Wenruo. He/she has just damaged the already broken reputation.

Not only Huawei, but also other companies will take a look at this accident to get the learned lessons.

layoutIfNeeded• 5 years ago

Oh no! Won't somebody please think of the companies?!

wwarner• 5 years ago

The committer defends his patch, and Qu responds very constructively with this list of more important work to tackle https://lkml.org/lkml/2021/6/21/342

chapium• 5 years ago

Qu sounds incredibly awesome.

specialist2• 5 years ago

Can we measure "impact" of changes?

PKI measuring just commit count is pretty silly. Like SLOC. Cite folklore.org story of -10,000 lines added.

Whereas Qu's suggestions are useful, important. Am totally ignorant about Linux, kernel, etc. But maybe there's already a leaderboard where the hivemind helps prioritize work.

iicc• 5 years ago

A measure of the ratio of patch effort to review effort (as subjectively judged by the reviewer, perhaps to the nearest order of magnitude) might be informative.

squiggleblaz• 5 years ago

That was a shockingly productive and constructive resolution.

Are we doing Hacker News wrong? Should we have a system to encourage models of good and productive interchanges, which leave the participants happy and the product better? I think hostility is easier but I guess it's also addictive, even for us.

I don't want to be shocked by seeing this. I want to see more of this.

Nexxxeh• 5 years ago

Thanks for posting this. I had initially wondered how, if not through this process, you were supposed to fix minor stuff like comments and error messages.

I'm not saying cleanup is not important, in fact we have routinely cleanups of typos/grammar for btrfs. (And I guess mostly caused by myself?)
Please at least merge all those small fixes into a larger patchset, and with a good cover letter to explain the reason (and auto-tool to do the change if possible) for all the involved maintainers, so that all of us are on the same page.

aritmo• 5 years ago

There is much hostility against Huawei. In the email, the message title was really offensive and even assumed something about KPI, as if the Huawei engineer was inventing patches in order to count as more contributions.

Even the editorialized subject of the post on HN is polemic.

What is this problem with Huawei? It is the US government that is doing the economic war against China and because of that, decided to cripple Huawei.

queuebert• 5 years ago

One KPI in which Linux dominates: drama.

viraptor2• 5 years ago

This happened quite often in openstack too. Since any contributor with a merged change a few months before the annual conference got a free entry ticket, you could see people doing trivial formatting fixes at a certain time of the year.

Similar to the issue with Digital Ocean and Hacktoberfest https://blog.domenic.me/hacktoberfest/

People are always going to try to game metrics.

jayofdoom• 5 years ago

from an Ironic core.

There were several months where I think we had contributors running a spell checker against patches as a review. They would miss obvious code syntax issues but would request lots of spelling changes (including British->American spelling).

You get what you measure. There's always going to be someone trying to do the absolute minimum to increment the number they get judged on.

roenxi2• 5 years ago

Is there anything going on here that suggests this is not a routine maintenance issue?

The Kernel maintainers have a lot of people who want to submit code to the kernel, and have to maintain standards. I'd imagine there are a lot of emails like this one.

Good on Huawei if they are incentivising open source contributions. Shame on them if they are wasting OSS maintainer time. Presumably well done Qu for promoting some order in a chaotic world.

nevinera3• 5 years ago

It's not nefarious, it's a pattern of a particular company trying to game a metric, to boost their reputation.

Pretend you were a dev, and your effectiveness was mostly described within your 500-engineer department in terms of "how many pull-requests" you merged each month. In the abstract, that seems like a reasonable metric to use - it does seem to correlate with actual output, and encourages smaller slicing (which is generally positive). But two years down the line, you find this one guy, consistently in the top 10 rankings, and you feel like he's just not accomplishing that much, so you start looking into his actual workstream. And you find that he does do normal work, but he produces 10 times as many 1-5 line PRs as the average, each of them adjusting some bit of copy or changing an error message. For the better! And his QA isn't bothered, because his work is very easy to verify.

But he's not actually one of the most effective devs in the company, and it's really irritating that his process-games (a) seem to work (the VP of engineering definitely knows him as a 'high output engineer') and (b) impose some unnecessary load on other bits of the company's process (like the QAs and burning CI credits).

That's the emotion in play here. "Stop wasting our time, we can see what you're doing."

roenxi2• 5 years ago

This looks boring. This is a scene that plays out in every company weekly. The stakes are trivial, there is no evidence of ill intent anywhere. Nobody has done anything particularly badly behaved. Nothing here is a shining example of good behaviour. There isn't a feel-good aspect.

I suppose the question is, who are the 100+ people who upvoted this, and why?

bsder• 5 years ago

The point is that if you were serious about fixing the spelling stuff, you'd put it into a single patch and not break it apart.

The fact that all these fixes are in separate patches means that its being used to game a Key Performance Indicator.

zeusk3• 5 years ago

Hi Leizhen, and guys in the mail list,

Recently I find one patch removing a debug OOM error message from btrfs selftest.

It's nothing special, some small cleanup work from some kernel newbie.

But the mail address makes me cautious, "@huawei.com".

The last time we got some similar patches from the same company, doing something harmless "cleanup". But those "fixes" are also useless.

This makes me wonder, what is really going on here.

After some quick search, more and more oom error message "cleanup" patches just show up, even some misspell fixes.

It's OK for first-time/student developers to submit such patches, and I really hope such patches would make them become a long term contributor.

In fact, I started my kernel contribution exactly by doing such "cleanups".

But what you guys are doing is really KPI grabbing, I have already see several maintainers arguing with you on such "cleanups", and you're always defending yourself to try to get those patches merged.

You're sending the patch representing your company, by doing this you're really just damaging the already broken reputation.

Please stop this KPI grabbing behavior, and do real contribution to fix the damaged reputation.

^^ Original message

Not sure where he said it'd be okay if anyone else sent it - but I can see why a string of "cleanup" from a company can be seen as more as metric manipulation than any actual worthwhile contribution.

dang• 5 years ago

Can you please not copy the entire OP like that? It's confusing, obscures your specific point, and leads other commenters to reply to random things in the OP, which is fine of course, but that's what the thread is for in the first place.

We detached this comment from https://news.ycombinator.com/item?id=27630257.

zeusk3• 5 years ago

Well the reply was to a comment that didn't seem to have read the link so I felt it was apt but alright.

sdesol4• 5 years ago

Since there are talks about padding numbers for KPI, I did a quick analysis of the linux kernel for single line changes that belong to commits with only one file change and this is what I got for the last 90 days:

https://public-001.gitsense.com/insights/github/repos?q=comm...

Scroll down to the bottom for a breakdown of the file types.

To review the file changes (click on the files) and commits, look at the following:

https://public-001.gitsense.com/insights/github/repos?p=comm...

Disclaimer: I'm the creator of GitSense and there is a bug where the window size won't show 90, but it is for 90 days.

zealsham• 5 years ago

The tone of the email reminds me a lot of Linus Torvald’s brutal code review comments in the google group back then.

Can it be said that the “no nonsense” behavior often exhibited by maintainers of Linux kernel could be one of the reason the project grew to become what it is today

criddell• 5 years ago

Sure.

But that could be good or bad depending on whether you think Linux has lived up to its potential or not.

ahartmetz• 5 years ago

Google group? Do you mean LKML? As the name says, it's a mailing list.

duxup3• 5 years ago

Open Source is such a surprisingly high trust ecosystem. I really hope that isn't something that has to be changed in the future.

I can 100% understand the concern here.

Possibly there is some sort of internal corporate metric or just personal bragging rights get tied to these kinds of patches. It costs the company nothing to incentivize this and the maintainers get all the work associated with it.

Like all metrics "When a measure becomes a target, it ceases to be a good measure." (Goodhart's law). There's really no reason for huawei to change other than for the maintainers to ask them to not do that thing.

I recall a talk from a maintainer once gave at some company (I forget where). He detailed what kind of patches get accepted and what don't. They were pretty straight forward standards, some more obvious than others (you know ... say what the patch does accurately).

It wasn't stated but he clearly expected some level of professionalism or just efficiency from the folks giving him work to look through their code on behalf of their company.

I don't think that's too much to expect.

nacs• 5 years ago

There's an external metric too -- there are public lists of "Biggest company contributions to Linux" where the likes of Intel/Google/Red Hat/NVIDIA appear:

https://news.itsfoss.com/huawei-kernel-contribution/

It's good for PR.

lmilcin6• 5 years ago

I have worked in team with some kernel developers at Samsung, years ago, so let me put some perspective for those who do not understand the dynamics.

In some companies, the amount of patents or Linux kernel patches you get accepted is direct measure of your success.

As you know, whatever you measure becomes a target -- these guys feel very pressed to get ANY kernel commits accepted, no matter how small, irrelevant or nonsensical. Because in the very end almost nobody sees the amount of value that a company creates for Linux users. What everybody is looking at is that "Company X is nth on the list of biggest kernel contributors".

So "KPI grabbing" is meant to mean the practice of not caring for the value or quality of your contributions but rather for the count of the commits or LoC you can get accepted.

This is damaging to Linux kernel developer community because it just creates work for maintainers without creating much or any value at all.

These maintainers are absolutely critical resource for the project and their number and availability translates directly to how much stuff can actually be done. So no wonder people are irked by it.

newsclues3• 5 years ago

Maintainers need their own metrics that are effective, and publish them and work to have partners use them.

duxup3• 5 years ago

I like this idea, but the gamification / rules / meta around it sounds like it would be even more work. And like all metrics it would quickly become a target and... Goodhart's law.

This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.

lmilcin6• 5 years ago

The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue.

But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.

tgsovlerkhgsel• 5 years ago

Doesn't work. "When a measure becomes a target, it ceases to be a good measure."

You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.

radiator• 5 years ago

Maybe a panel of maintainers could publish a quarterly review with attribution/thanks to the most important contributors. No metrics to game, only pure human expert opinions.

exikyut• 5 years ago

[Generic hand-waving about how LWN kinda does this, in a high-signal manner unfriendly to statistical summation]

everybodyknows• 5 years ago

Maintainers aren’t interested in taking on extra chores. However, important contributors already do receive attribution, in the format of news articles, here:

https://lwn.net/Archives/

jldugger• 5 years ago

IMO part of the problem is that Greg KH publishes this annual 'state of the kernel' that is just a list of metrics to game. Which feels like it was started to shame non-RH companies to contribute more often. So, kinda maybe beware what you wish for?

Example: https://www.youtube.com/watch?v=SIQr2-Dh0es

jart3• 5 years ago

Could someone ELI5 to me what KPI even stands for? Does that mean like they're like the shock troopers of open source?

neilalexander• 5 years ago

Key Performance Indicator.

martincmartin• 5 years ago

https://en.wikipedia.org/wiki/Performance_indicator

Engineer's performance is measured, in part, by number of kernel commits. So, the number of kernel commits is a key performance indicator for the developer.

NikolaNovak• 5 years ago

KPI = Key Performance Indicator = "thing we decided is important for us to measure and increase"

As such exactly what it is, will depend on company/industry/market/team/project/etc. It may be revenue or units sold or widgets made or turnover or customer satisfaction or days without accident etc. In principle, it communicates to teams what is important to business and helps everybody focus and sync.... with usual real-world caveats and implementation risks.

I imagine here, for a large-company linux team, a "KPI" was "how many commits you made" - they measure it, they track it, and reward/recognize/punish team members based on it. So people start working toward increasing the KPI in any feasible way - if you only want me to increase the KPI of commits done, OK, fine, I'll do some commits.

alxlaz2• 5 years ago

KPI = Key Performance Indicator. It's basically a list of things that are considered during your evaluation. The more items off that check list you cross (if they're one-time items, like "ensure an uptime of X"), or the more of each you do (if they're things like "upstream patches" or "patents granted"), the better your score is.

It's a shit system that does exactly the opposite of what it's supposed to do IMHO, but that kind of stuff doesn't belong in an ELI5 explanation :-D.

NAR8789• 5 years ago

TIL what ELI5 means

sudhirj• 5 years ago

Oh god no. It means their managers evaluate them on that metric. In this case, the company has set targets on how many Linux kernel patches employees should land.

faeyanpiraat• 5 years ago

It's like the days when outsourcing software development to india was new, and they charged for number of lines of code written.

protomyth• 5 years ago

IBM apparently went that way too https://www.youtube.com/watch?v=kHI7RTKhlz0

wyager• 5 years ago

This is why ousting Linus puts the long-term health of the kernel at risk. It’s very difficult and expensive, at a personal and professional level, to tell colleagues to “fuck off” if they are submitting low quality garbage for reasons related to their salary. Linux was Linus’s baby, he had nearly absolute control, and he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons.

Now, if the people approving patches are just normal employees who can get fired if a kernel sponsor complains enough, it’s going to be much harder for them to successfully deflect all the bullshit without ever caving in.

otikik• 5 years ago

I think the email was completely effective in denouncing a behavior. Telling people to "fuck off" isn't required (nor desired, in my opinion).

Linux health happened *despite* Linus' manners, not because of them.

zxzax2• 5 years ago

If anyone gets ousted from the process, they probably deserve it, for wasting more maintainer time. Nobody is immune from that, not you, not me, not the maintainers, not the BDFL. Expecting the BDFL to be able to have full control and to review every patch is also nonsense that causes the exact same type of issues: https://lwn.net/Articles/393694/

geofft• 5 years ago

Good theory, but it's factually wrong. "Maintainers" in this context doesn't refer to Linus personally, it refers to individual maintainers of parts of the kernel. You can check the kernel MAINTAINERS file, and note that such a file has been there for years.

There has been a longstanding problem of people submitting low-quality patches to maintainers - not to Linus personally - and that problem predated Linus stepping back for one release.

Linus was not ousted and continues to have nearly absolute control; he has, for decades, chosen to exercise that control by delegating ownership of parts of the code to other people. Those people receive and review patches and incorporate them into their repos, which Linus pulls in bulk. (This is the origin of the term "pull request," which predates GitHub's use of it.) He quickly reviews those patches to make sure there's nothing weird, but he generally trusts that his "lieutenants" are making good decisions.

That means that the time being wasted here is the time of individual maintainers, who are reviewing the patch, making a full judgment on it, and pulling it into their tree. Linus sees all such cleanups once per cycle and looks at them fairly quickly.

Almost all of those maintainers have been "normal employees" of various companies for many, many years. Again, take a look at the MAINTAINERS file.

wolverine8764• 5 years ago

he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons.

Linus disagrees with you, so strongly that he invested a great amount of his time, reputation, and political capital in changing his behavior and that of his organization.

https://news.ycombinator.com/item?id=18000698

I think I'll trust Linus on that. Linux seems pretty successful.

moonshinefe• 5 years ago

Thanks for your perspective I didn't actually realize that dynamic was at play (at an organizational level, anyway, I'm well aware individuals game # of contributions).

These organizations pushing trivial/garbage patches and wasting maintainers' time absolutely have to be called out by the community as unacceptable, so I'm glad to see the post. This is a massive waste of time, time which open source contributors/maintainers often are already short on.

cjohansson• 5 years ago

Sounds like the typical issue of business intelligence using quantity over quality to rate work performance

croes2• 5 years ago

"When a measure becomes a target, it ceases to be a good measure."

thih9• 5 years ago

Corollary: "Businesses that base their growth on A/B testing only, turn into gambling or pornography".

squabble• 5 years ago

I sometimes wonder if something like this has happened with money.

markus_zhang5• 5 years ago

Discussion in Chinese online community zhihu if anyone is interested:

https://www.zhihu.com/question/466111598/answer/1951896502

epage• 5 years ago

To save English speakers a couple of clicks to translate: https://translate.google.com/translate?hl=en&sl=zh-CN&u=http...

belter8• 5 years ago

Keep in mind this community is probably censored:

https://qz.com/1063073/in-china-you-now-have-to-provide-your...

From the online translation, the mention by one the commentators that the Kernel Maintainer is also Chinese...give me a bit of a chill...

To make it clear ...Not that I would not trust the maintainer, instead, my concern is that depending where he is based a window could be open to unpleasant pressure...

MangoCoffee• 5 years ago

进git看了一下，笑死了。这位华为大佬曾经在一天里对同一个文件提交了6个fix，每一个fix修改注释里面的一个拼写错误，还有一个是调整include 顺序。被reject了以后还发邮件去defend。打个比方就是你假装帮导师做数据，一个数据没做但是一天发给他6个版本，每个版本改论文里的一个错别字，同时要求导师把你名字加到作者里。导师说求你别给我发了，然后你跑去办公室和他争论这个错字很重要，换了谁都要发飙啊。

https://www.zhihu.com/question/466111598/answer/1953367097

this comment is funny.

markus_zhang5• 5 years ago

Yeah some comments are funny and insightful, but again I have no idea whether what they said was true or not.

mook• 5 years ago

One of the answers raises an interesting possibility:

https://www.zhihu.com/question/466111598/answer/1953785089

Basically, they claim it's possible that it's not about KPI at all; but rather, Huawei has added some internal processes to try to raise code quality. But those processes, rather than focusing on things like algorithms, ends up mostly catching unimportant things like minor security issues in tests and code style issues in third-party code. So they suspect that these trivial patches are somebody internal to Huawei trying to satisfy their own internal processes by fixing upstream.

That would mean it's still about KPIs, just not those of the patch submitter.

lifeisstillgood• 5 years ago

There is an old story of two surgeons - one with a near zero patient death rate and one with a much higher patient death rate. But the surgeon with the higher death rate is tackling harder more complex surgery on sicker patients - while the other avoids the complex stuff.

There is nothing wrong with either.

liuw• 5 years ago

Qu Wenruo is not a Linux kernel maintainer. The submission title is incorrect and over dramatic.

moron4hire• 5 years ago

Well, nice to see this kind of bullshit isn't just happening to small Open Source projects. Unfortunately, it seems like these kinds of commits are the vast majority of the commits I get on my projects.

paganel• 5 years ago

already broken reputation.

What's that "broken reputation"? Apart from the media circus about 5G. Anything that Huawei has done wrong related to the Linux kernel or the open source community generally speaking?

HarryHirsch• 5 years ago

Metric brushing, clearly, not the nationalistic panic over 5G and spying concerns.

The maintainer says there has been a torrent of trivial fixes from Huawei, and this is just more of the same. Someone wants to get ahead at work on the back of Linux maintainers, just like in the recent example with the University of Minnesota.

hn8788• 5 years ago

You can look through their wikipedia page and see that a lot of early success for the company was based on stolen information from companies like Cisco.

arvigeus• 5 years ago

https://securityboulevard.com/2020/12/was-this-huaweis-faile...

jvanderbot• 5 years ago

The sense I got from the post was: "It's fine for noobs to cut their teeth on small issues, but as one of the largest tech companies, I'd expect more from your commits. It's obvious you're not even trying".

matheusmoreira3• 5 years ago

I agree with this. It's reasonable to expect more from well-paid professionals.

pmcollins• 5 years ago

The point of the email is that the submitter is indeed trying, trying to improve their stats within their organization, at the expense of maintainers’ time, rather than working on the kernel in good faith.

camjohnson26• 5 years ago

Also from the thread posted earlier:

I'm not saying cleanup is not important, in fact we have routinely cleanups of typos/grammar for btrfs. (And I guess mostly caused by myself?)
Please at least merge all those small fixes into a larger patchset, and with a good cover letter to explain the reason (and auto-tool to do the change if possible) for all the involved maintainers, so that all of us are on the same page.

vesche• 5 years ago

In case anyone forgot about this: https://grsecurity.net/huawei_hksp_introduces_trivially_expl...

anzlq• 5 years ago

Happens in CPython as well, I like the term "KPI grabbing". If you complain, the central committee will denounce you as non-inclusive.

Nice that there is still some sanity on LKML.

tomrod2• 5 years ago

This is going to become a more common theme. Geopolitics spill into everyday life in all sorts of unexpected ways.

turminal• 5 years ago

This issue doesn't have anything to do with geopolitics.

tomrod2• 5 years ago

I believe that it does on a second order condition. Huawei is under tighter scrutiny because of current events in geopolitics. Were it Google or MIT, I believe it reasonable to expect that the maintainers wouldn't be as prescriptive.

nneonneo3• 5 years ago

The fellow who is being named in this particular PR has been _quite_ busy lately, submitting mostly typo fixes and whitespace fixes: https://lore.kernel.org/lkml/?q=f%3Athunder.leizhen

Sometimes their patches aren't even valid - they tried to fix "borken" to "broken" and the maintainer was not happy: https://lore.kernel.org/lkml/YK3wOkX6I78j73zD@gmail.com/ (this does come down to not being familiar with this particular bit of slang - but they push back and argue a bit which doesn't help)

Sometimes the maintainers are not happy just getting a patch that fixes one line of whitespace: https://lore.kernel.org/lkml/20210608105943.2376328c@oasis.l...

sdesol4• 5 years ago

Here is this person's commit history for the last year for additional context:

https://public-001.gitsense.com/insights/github/repos?q=auth...

They do appear to contribute regularly enough (relatively speaking) and based on some of the busfactor metrics, they are the sole maintainer or main maintainers for about 50 files.

And if you look at their one line change commits, they do seem to be valid:

https://public-001.gitsense.com/insights/github/repos?p=comm...

Disclaimer: I'm the creator of the tool for the links above

delroth• 5 years ago

And if you look at their one line change commits, they do seem to be valid

By construction, yes. You're only looking at their commits that were merged into Linus's tree.

TimWolla2• 5 years ago

they tried to fix "borken" to "broken" and the maintainer was not happy […] this does come down to not being familiar with this particular bit of slang - but they push back and argue a bit which doesn't help

I did not know "borken" either, but I am aware of "borked" and "broken". Based on that email thread someone else already attempted to fix this in the past.

Maybe it's an indication that the so-called "joke" is not actually funny and it should be adjusted to either "borked" or "broken" to not cause others to send the same fix?

Tainnor5• 5 years ago

Agreed. While the kernel developers can obviously do whatever they want in terms of in-house jokes, the claim that "this is just normal usage, Google it" is a bit ridiculous. They could have just said, "we know it's wrong, we just thought it's funny".

("Borken" is a city in Germany, that's my only association with it )

red_trumpet• 5 years ago

It's OK for first-time/student developers to submit such patches

zeusk3• 5 years ago

That's not the same as "anyone but huawei" as the parent comment noted.

Huawei is not being singled out here because huawei or china but because of a specific behavior that's being called out here.

airhead969• 5 years ago

Second-stringers do this. They can accumulate all of the certifications, bogus "patches," and patents they want, but they'll still have zero street-cred. What's more, companies who promote and advance based on these factors are only fooling and embarrassing themselves with weak staff.

The other problem is a fair number of Eastern managers don't know or care about the business or management details, they all of want the prestige of a title with the least effort. A lot of no-name businesses in China are more BS than corporate ones elsewhere. After-all, why is there such a huge market for white business actors?

joelkevinjones• 5 years ago

LLVM has quite a number of people who have "review after commit" privilege. These are people who are trusted to know the difference between changes needing review and those that don't. As a result, people can fix typos, fix formatting, etc. without burdening reviewers. Those sorts of changes don't go through the usual review process, but on occasion may be reverted, by either the committer or someone else who has commit privilege.

a-and• 5 years ago

It's unfortunate since ideally any contribution is a good contribution, but the Linux kernel has been so highly dependent on maintainers, that them being highly protective of their time is totally warranted.

As someone who works at a big corp with regular OSS contributions, this doesn't surprise me at all. Recently a "Look at how amazing we are" e-mail was sent out to my team touting the number of PRs submitted to a widely-used OSS project.

If your management incentivizes it, you'll try to game the system.

h_anna_h• 5 years ago

Remember when they tried to push a backdoor into the kernel last year? I am surprised with how they kept allowing them to submit patches. They acted much more boldly with the issue with the university of Minnesota.

I am not a grsec fan but here https://grsecurity.net/huawei_hksp_introduces_trivially_expl...

Tainnor5• 5 years ago

Similar thing happened with the Hacktoberfest incident last year: https://blog.domenic.me/hacktoberfest/

In short, DigitalOcean unwisely incentivised a bunch of people to make PRs on open source repositories unaffiliated with DO so they could get a T-Shirt. It went so far that some guy actually made a very popular YT video about how you can make a PR for a small typo fix (I guess I still fail to understand why you'd need a YT video to explain that). Maintainers were none too pleased.

vzaliva• 5 years ago

It looks like fundamental problem is that contributors are rated based on number of commits vs. qulaity of their contirbutions. They should fix that instead of discouraging contributiors.

jxramos• 5 years ago

Can someone explain this line to me please

    what you guys are doing is really KPI grabbing.

gdulli• 5 years ago

There's a movement away from evaluating people or teams qualitatively, which is difficult and prone to dumb biases, and towards quantitative measures, which are easy and prone to dumber biases.

Presumably here (I don't know if it's been proven) people in that org are being measured by how many commits they make, ignoring the depth or quality of the commits. Because it's someone's idea of a Key Performance Indicator (KPI.) So people are "grabbing" easy commits.

Rapzid• 5 years ago

Got hit a while back with a GitHub account that was "bombing" OSS projects with Docker image "best practice" fixes related to Deb "recommends" or some other such stuff. Most of their history for the past year was exclusively opening these PRs with canned messages.

Some of the projects were very large(Apache Arrow, a large Google project IIRC), and they were opening commits up against largely test images.. And a LOT of them. On one of the larger projects somebody rightfully asked "What's the value add here?" who was promptly ignored by what I can only assume is a project manager who start humoring the submitter by trying to help them get this entirely useless work merged. It turned into an epic fail-whale causing all the projects tests to fail and that initial dev spent "too much time" trying to untangle it all before putting their foot down and reverted the changes.

Other project maintainers got wise to these fishy PRs and after inspecting the submitters history just started telling them to sod off.

powerapple3• 5 years ago

Now I understand the story: Qu was salty because his debug message was deleted by the person from Huawei. Qu was mad. Such a trivial event, and still made to HN because of a few keywords.

dustinevan• 5 years ago

This same sort of thing is killing stackoverflow haha.

sigjuice• 5 years ago

Aren't these sorts of trivial fixes best sent to the kernel janitors mailing list?

https://lore.kernel.org/kernel-janitors/

“Please don't waste maintainers' time on your KPI grabbing patches”

Related Stories