Dstat project ended due to RedHat replacing it with its own dstat tool

https://github.com/dagwieers/dstat/issues/170

by tanelpoder • 7 years ago

168 152 7 years ago

detaro• 7 years ago

redhat announcing their "new dstat": https://www.redhat.com/en/blog/implementing-dstat-performanc...

astine• 7 years ago

The claim is that the original is no longer being maintained. Is that true? It changes a lot about what I would think of this move.

nightfly2• 7 years ago

The git repo linked here has 6 commits on Jan 16, 2019. And Nov 23, 2016 the last new commits before then.

jkaplowitz• 7 years ago

Even if it isn't being maintained, I wish they had changed the name if they were forking or reimplementing without permission. There's a reason trademark law exists and doesn't require registration if you can prove the situation.

But yeah, it's a lot harder to fight Red Hat (soon IBM!) over that if the project is inactively maintained and there's no registration.

Still, whether legal or not, the name squat is not a good look for Red Hat.

EwanToo• 7 years ago

The issue below has more details, it does look more complex than this post suggests?

https://github.com/dagwieers/dstat/issues/156

codewiz• 7 years ago

From the git changelog, it doesn't appear to be completely unmaintained, but not very vital either: https://github.com/dagwieers/dstat/commits/master

Red Hat's announcement (posted in February 2019), also cites lack of Python 3 support as another reason for replacing the original dstat, but Python 3 compatibility was added last January.

jasonkester• 7 years ago

No, the original was finished.

Feature complete, bug free software doesn’t need to have patches released every week. That is the ultimate goal of a small utility like this.

It’s a shame that so many developers mistake this state for “unmaintained”or “abandoned”.

thisgoodlife• 7 years ago

I have been using dstat for years. It's a great tool. But why kill the project just because some company has a project with the same name? There is more than one distro, you know

type0• 7 years ago

It is the Big Gorilla, the silverback if you like. For other distros that would be even more confusing when some of their tools will be ported there.

pas2• 7 years ago

The project was already dead. But now the maintainer[s] found time to complain about how evil RH simply opted to provide a py3 compatible replacement and provide it as dstat.

dagwieers23• 7 years ago

That's not exactly what happened, but great to see you are making improvements in your fiction-writing skills :-) Keep it up !

marktangotango• 7 years ago

Performance monitoring tools for enterprises can be very lucrative. Surely they’ll move to an “open core”, closed source commercial license for the useful bits, if they haven’t already.

loeg6• 7 years ago

That isn't Redhat's model. They open the code (it's mostly 3rd party OSS anyway), including packages (hence, CentOS) and sell access to the updates system and support contracts.

lathiat• 7 years ago

There's a couple of areas that's not actually true. For example their "insights" package which has rule based detection of problems to fix etc is open source but the actual ruleset is not.

mindcrime2• 7 years ago

That isn't Redhat's model.

Well... it hasn't been so far. But this isn't the old Red Hat anymore. This is IBM Hat. And for all the talk about keeping Red Hat mostly independent or whatever, I think we all know that's bullshit. Acquiring companies always give that little dog and pony show, and it never holds up in the long-term. There's no way that IBM culture won't wind up infecting Red Hat. Now whether that's a good thing or a bad thing is a question I'll leave to the philosophers. But I'd be very leery of assuming anything regarding what RH will or won't do going forward.

icedchai2• 7 years ago

I literally use dstat all the time! This is very sad.

op00to4• 7 years ago

Why? What has changed? Do you use the plugin functionality of dstat, or do you just use the dstat command? If it's the latter, then you can do everything you could with dead-dstat and more with pcp-dstat.

icedchai2• 7 years ago

It saddens me when a useful, good open source project closes. I realize I can still use it, and I will... That's not the point.

throw7• 7 years ago

I only know enough to get me in trouble, but pcp is a general "framework" for performance monitoring... they have pcp versions/rewrites of all types of traditional monitoring utilities like vmstat, iostat, top.

Note: they don't overwrite/replace those utilities on the command line. You can still use the traditional utilities same as it ever was. The pcp versions are called like so: "pcp dstat" or "pcp atopsar".

snorrah• 7 years ago

Curious, i know dagwieers from being involved in the Ansible project (RedHat owned). Seems friction is occurring, given the language in his post.

dagwieers23• 7 years ago

What language ? It is being quite factual really.

sdiepend• 7 years ago

Time to go to Puppet Dag? :)

tedivm4• 7 years ago

Having read through this post and the previous comments[1] I have to say that RedHat did the right thing in the completely worst possible way.

Stealing the project name was a mistake. The authors are right to be upset by the fact that people used to get their project when they did "yum install dstat" but now are getting a completely different (and not completely compatible) project. I think a lot of the friction could have gone away if they had just named it "pcp-dstat" or something similar, but instead they essentially took this person's namespace without discussion, and really for no good reason that I can tell.

The other issue is that forking a project without even talking with the people running it is a very touchy subject. Ultimately it does look like the project was dead- even to the point where people were asking to contribute but not getting responded to, and the fact that python2 is EOL and python3 support wasn't added until after this drama started shows that the project wasn't really "active". So while I do think they should have reached out, I totally understand why that slipped their mind.

So at this point I think RedHat should apologize and fix the package name.

[1] https://github.com/dagwieers/dstat/issues/156

jsnell2• 7 years ago

You've misunderstood. The "dstat" package was removed from Fedora, rather than have its contents replaced. The package containing the new program is called "pcp-dstat", so they already did everything you think they should have done.

tmyklebu• 7 years ago

What makes you think he misunderstood? What happens on fc30 when you 'yum install dstat' as he suggested?

This webpage listing the contents of the fc30 pcp-system-tools package: https://fedora.pkgs.org/30/fedora-updates-x86_64/pcp-system-...

lists a file called '/usr/bin/dstat'. The pcp-system-tools package obsoletes a package called 'dstat'. If you go to https://pkgs.org/download/dstat, and look under "Fedora 30," you get pcp-system-tools.

[Spoiler: 'yum install dstat' installs pcp-system-tools, which has a symlink /usr/bin/dstat -> /usr/libexec/pcp/bin/pcp-dstat.]

tedivm4• 7 years ago

To quote a comment from a dstat collaborator from the above linked thread-

That said I don't fault you for make a new tool that does all the new fancy things you mentioned above, I do think it's a little sketchy to make a new tool called dstat without at least asking here first. I know that when I hit Fedora 29 and did a yum install dstat and I get a bunch of pcp stuff I was pretty confused.

The way they have things aliased right now, if you type in `yum install dstat` you will get this new program (including all of its baggage).

dagwieers23• 7 years ago

It didn't just slip their mind, they did not really care: https://bugzilla.redhat.com/show_bug.cgi?id=1614277#c7

The decision to do this was made before June 2018, 18 months after the last activity. One of the reasons cited was lack of activity, but that is no thanks to them, I guess.

That is why I am convinced the goal was to replace it from the onset, there was no interest in helping out the project. In fact "no activity" was the right excuse to make their action seem legitimate. Attempting to contact the project could have jeopardized that plan.

They could have just removed "dstat" from the distribution, and added a note that users can now use pcp-dstat. But now they made it impossible for users to add the original dstat. Let alone the support nightmare of having a different tool with the same name. There's no winning this one.

dralley3• 7 years ago

I totally agree that they should have left a note on the issue tracker - even if they thought it would go unread. However I think you're being a little unfair on this one point.

there was no interest in helping out the project.

They saw that people were filing issues and making pull requests and that those people were getting total radio silence. I can understand how someone might think "well, people are already trying to 'help the project' and the maintainer isn't even acknowledging it, so it's a waste of time".

And while again I totally understand and agree that they should have tried to contact you, you've known about the removal for like 9 months and you apparently never reached out to them either? Like, a couple words in reply to a bugzilla or an IRC message does not amount to picking a fight with a company.

zaphirplane• 7 years ago

Are you the person that used to maintain a bunch of rpm(s) ? Thanks a lot for doing that, it helped me a couple of times

jeltz• 7 years ago

While Redhat did not handle this situation well you are also being very unfair to them. Why would they waste their time trying to contribute to a dead project? If I see that patches do not get merged and that bug reports go unanswered I will not bother doing anything because I know my time will be wasted, and I do not fault RedHat for doing the same.

What they should have done is tried to reach out to you before forking.

tedivm4• 7 years ago

Apparently reusing the name without talking to the author was a conscious decision on their behalf-

https://bugzilla.redhat.com/show_bug.cgi?id=1614277#c7

appstateguy2• 7 years ago

meh. ultimately this is OSS. the maintainer was negligent. if you're not a good OSS maintainer, you don't deserve to maintain control. it's not reasonable to be indefinitely patient when the original author shows no initiative to maintain their project

ungamed• 7 years ago

Are all of those people commenting Red Hat employees ?

devnullbyte2• 7 years ago

I disagree, they were right to use the same namespace and its questionable how you conflate that to stealing. If something is abandoned, how can you steal it? The project was dead, PR's and issues were ignored and the lack of Python 3 support was a huge issue (which the developer pathetically tried to shut down discussion over by releasing a python 3 port just hours before closing the project).

Changing the name space would have been problematic, as so many people use automation tools now to install package dependencies, rather than break their work because someone has zero intention of maintaining a project, they did the right thing and used the same namespace.

Had this been a case of RH disagreed with the developers direction or any factor around an active project, it would have been a shitty move, but considering the developer ignored any offers to help and let the project to fall into a stale un-maintained state, I don't see any grounds for him to throw his rattle out of the pram and blame RH here.

dagwieers23• 7 years ago

Nobody stepped up to maintain the project, not even Red Hat. Some people have offered help, and I have send out a few GitHub invitations to people that offered, but no takers.

I assume people offering to help only wanted to see their own contributions merged.

I shouldn't be surprised people here take positions while not having been involved in the project or know the whole history.

In my opinion, stopping the project was the only sensible thing to do at this point. Red Hat replacing the tool with one of their own (which I wholeheartedly disagree with for various reasons) created a dead-end. It's the final blow to a dying project.

Wrt. Python 3 support, I simply did it because it was so easy to do, not because I thought this would be a game-changer. But hey, please do read into whatever fits your narrative :-)

proctor• 7 years ago

this strongly reminds me of the hn post from five days back in which the main thread talked at some length about a very similar scenario

"He missed a big one: you have no way to stop Linux distributions from hacking up your software, and you'll suffer the consequences of whatever they do."

https://news.ycombinator.com/item?id=19935648

chubot• 7 years ago

Yeah I thought of the same comment. That is unfortunate.

Maybe Debian is more respectful since it's open source not backed by a company with enterprise customers?

bonzini5• 7 years ago

Not really, remember the ssh key fiasco which was also due to a Debian-only patch.

(Disclaimer: I work for Red Hat)

raverbashing• 7 years ago

"respectful" as in let's wait years for a maintainer to come up with a solution and when we try to propose our own the maintainer trows a fit and turns it down?

mattdm6• 7 years ago

Eh. Although we don't always do things perfectly, Fedora has a pretty strong policy of working with upstreams first.

By contrast in my personal experience as a developer of something that got packaged in Debian, I discovered later that that package had a bunch of patches added, a man page written (!) and so on, none of which was bad, but also none of which was fed back to me at all.

(In case not obvious: I work for Red Hat on Fedora.)

peterwwillis• 7 years ago

Honest question: aside from "they provide Enterprise support", does anyone have a reason they would actually use redhat or centos for new servers? Why not use a Debian-derived one, which seems to both be more common, and have less constant creepy corporate influences?

cpitman• 7 years ago

Disclaimer, I work for Red Hat. That being said...

The big things for our customers isn't just support, it is stability. When you build your systems on RHEL 8 (just released), you will get 10+ years of support and patches for that system. Those changes are all guarenteed to be binary compatible, so the risk of patching ever impacting systems is low. If there is a critical security vulnerability (say Heartbleed in openssl), Red Hat will backport the fix to every supported version of RHEL. That means you get the security fix without (1) having to upgrade the entire openssl dependency and risk breaking something else and (2) without pulling in newer changes that might have their own vulnerabilities.

Going along with the stability, vendors can QA their products against RHEL as a known target.

Last reason for now, if you are in an industry with any kind of compliance requirements, like FIPS, Red Hat jumps through the hoops to get the certification. That makes RHEL the only option in many environments.

user59944613• 7 years ago

I'd say both stability and update.

Debian never upgrades anything (unless security issue). All packages are frozen at the time of the initial major release. Have fun using curl/htop/nginx/libreoffice from 5-10 years ago, plenty of obscure bugs and missing command line flags.

RHEL actually maintain their system packages.

jteppinette• 7 years ago

which seems to be more common

I have exclusively used the Fedora line in enterprise environments (CentOS, OEL, RHEL).

The point about long term support is definitely #1.

protomyth• 7 years ago

We had a vendor that we bought software from require, as part of the software requirements, both Red Hat and Oracle. This is not unique as many small specialized software shops do the same. I still have some government software that requires Windows 2008 and no other version is supported.

lunchables• 7 years ago

Extremely long life span (10 years), support if you want it, great community, great package manager, and in rhel 8 really great new features like appstream.

throw20162• 7 years ago

This is undoubtedly shady behavior. dstat is a widely regarded and mature tool. Looking at its repo for 'activity' without context - maybe nothing needs to be changed and its working as designed - is completely disingenious.

Why should repo activity without context be used as an indicator of anything in discussion instead of focusing on what Redhat has done?

This is openly hijacking an open source project by a billion dollar company because it can and makes a mockery of not only open source colloboration culture but basic professional behavior. Has Redhat reached out to the author, made any requests, tried to work out some way forward, offered to pay for the brand name? Cmon this is simply indefensible.

dralley3• 7 years ago

Looking at its repo for 'activity' without context - maybe nothing needs to be changed and its working as designed - is completely disingenious.
Why should repo activity without context be used as an indicator of anything in discussion instead of focusing on what Redhat has done?

"Activity" doesn't just refer to commit traffic. People were filing issues and making pull requests and not getting any responses for more than a year. And furthermore it was a Python 2 tool and the Python 2 EOL date is fast approaching.

I agree that the packagers should have at least left a note on GitHub (even if they thought it would go unread) - but clearly it was not a "maybe it's just mature and nothing needed to be changed" situation.

throw20162• 7 years ago

This is muddying the waters. Surely if you have gone through the trouble of looking at the repo and finding these pull requests you would also know the context and should then share it to give others here a context. Without that you are just here making 'vague' charges given dstat is a well known, highly regareded and mature tool. The Python3 port was already done by the author so the python2 eol seems a complete non sequitor. Again a rush to condemn without good reason.

Are you saying anyone running a Python2 project can expect to have Redhat reimplement it in Python3 secretly and have people defend it on HN? This kind of behavior is simply indefensible.

It's obvious there is a culture clash of open source developers sponsored or working for corporates conflating heavy activity on Github that they are paid to perform as their day jobs as the only open source model. But open source was traditionally about people having other jobs and using their spare time to develop open source without profit motives because they believed in the movement. They certainly did not face corporates and paid developers analysing their frequency of contributions to dismiss their efforts and justify an unethical takeover.

devnullbyte2• 7 years ago

"dstat is a widely regarded and mature tool. Looking at its repo for 'activity' without context - maybe nothing needs to be changed and its working as designed - is completely disingenious."

Its based on a entire language which is about to go EOL (Python 2)

certik• 7 years ago

I submitted this PR long time ago: https://github.com/dagwieers/dstat/pull/50 that fixed an important installation issue, and it took the author 5 months to look at it and simply close it without providing any hints how to fix the problem in some other way. So I was discouraged to contribute further to the project.

rurban2• 7 years ago

You added a security problem. The fix is obvious for everybody working in security, but it's questionable if it should be discussed there.

lstamour4• 7 years ago

I can see how the original CVE patch removed using the current working directory for loading plugins, but it seems like less of a security issue to simply look up one directory relative to the current binary install, assuming the binary is stored in ./bin after being built and plugins would be stored in ./shared ... Now, it might be inconsiderate to look up a folder when one could be preconfigured based on a common root or home folder, but it doesn’t necessarily seem as bad to trust the parent folder of a binary as trusting the current working directory’s contents?

rurban2• 7 years ago

Reminds me on moreutils parallel (C), hijacking GNU parallel (perl), and changing the command line options. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050

JdeBP4• 7 years ago

This sort of thing happens regularly. For another example of many, witness the effects on users of the change of the "mailx" command in Debian.

* https://unix.stackexchange.com/q/469780/5132

* https://unix.stackexchange.com/q/489477/5132

gigatexal3• 7 years ago

Why not come up with a new name for the project instead of rage quitting?

anthony_doan2• 7 years ago

Why not come up with a new name for the project instead of rage quitting?

Why should he change the project name when he originally have it first?

Also why did you have to frame it as rage quitting?

gigatexal3• 7 years ago

It’s the way of the world that larger fish get to do as they please.

A simple name change and he can continue to do his work. It’s the idea of turning the other cheek. That or trademark the name I guess.

pas2• 7 years ago

It was already dead, so pragmatically there was nothing to do, this is just a swan dance on RH's bugzilla.

Yes, RH should and could have filed and issue saying that they started working on a fork/port-to-PCP, and they intend to provide the dstat executable. But usually at that point, the work is already started, and there's almost nothing to be done. How long RH should wait for that ticket? What if the maintainer says, wow, cool, I'll make a py3 version in 3 months. But RH needs it in 2? What if the maintainer disappears again?

It's not great, but it was abandoned. :(

dagwieers23• 7 years ago

It wasn't abandoned, it didn't see any maintenance for 18 months.

I don't get paid for doing this work. Red Hat does. They get paid for offering it to customers that apparently demands that it ships with RHEL8. They have been shipping it with RHEL since RHEL3.

They haven't contributed to it ever in that timespan.

They could have paid me for maintaining it, they could have offered to maintain it for me. (Because it is a lot of work to accept and verify pull requests). I am a freelancer, I have to work for a living I don't have a lot of free time.

Instead, they ripped out the plugin backend, added the PCP backend and now you can't use it as a drop-in tool that can't run PCP. Your synology NAS, WRT router or JeOS platform. Because Red Hat does not care about anything other than RHEL. They are business-oriented. And that's fine.

Is the Open Source ecosystem better off ? I sincerely doubt it. It's being replaced by paid-for engineers with business interests. In the long term it is killing the community.

clhodapp• 7 years ago

Devil's advocate: Isn't this kind of behavior how we ended up with what became the POSIX command-line utilities? A bunch of people re-implementing and extending each other's stuff without changing the name?

I know that the word has changed a bit since then and that it's a bit different to appropriate a soul-less corporation's command names vs an individual volunteer.. It just happened to strike me as odd at the moment I came across this thread that we're so up-in-arms about this when we're all happily using utilities that were produced by the very same process every day.

zonidjan2• 7 years ago

I don't know that massively popular utilities were ever "reimplemented" with the same name. Ported, or forked, or extended, but not reimplemented. It's not clear to me if pcp-dstat is a fork or a total rewrite.

Also, most of those utilities were named things which solely described what they did. It's a bit different to make a new "cut" vs a new "clhodapp".

JdeBP4• 7 years ago

Then there is a lot of history that you do not know, starting with BSD and progressing through the late 1980s and early 1990s when there was a torrent of "clones" of things.

The "Vixie cron" that you might even still be using is a "PD" clone of the Unix cron program, reimplemented from scratch with the same name. The van Smoorenburg "init" that you might have heard of was a clone of (an at the time sadly already out of date) Unix init program. All of the stuff in GNU coreutils and other similar GNU toolsets was a deliberate reimplementation from scratch.

And then there are the Thompson, Mashey, Bourne, Almquist, Bourne Again, Korn, Z, and Watanabe shells. And BusyBox and ToyBox. And /usr/lib/sendmail . And the "mailx" utility. And "awk". And "vncviewer". And "getty". And so on.

* https://unix.stackexchange.com/a/508142/5132

* http://jdebp.uk./FGA/inittab-getty-is-history.html

* https://news.ycombinator.com/item?id=19988382

* https://news.ycombinator.com/item?id=17005677

* https://unix.stackexchange.com/a/316279/5132

throwaway2048• 7 years ago

almost every BSD tool is a rewrite of the original unix tools, and every GNU one is a separate rewrite too.

mattdm6• 7 years ago

So, from a Fedora Project point of view, I'm not really sure how we could have done this better. This change was properly announced and publicized as https://fedoraproject.org/wiki/Changes/MergeDstatAndPerforma... and approved by FESCo (the Fedora Engineering Steering Committee) in August 2018 (https://pagure.io/fesco/issue/1956).

The change page says "The original dstat utility has reached end of life - it does not support python3 and there are no plans to update it." This may not have been right after all, but I don't think it's in bad faith. Perhaps FESCo should have dug a little deeper, but there aren't really any particular red flags indicating that that's warranted.

I'm open to suggestions, though -- we do want to continually improve our processes, and we do want to work well with upstreams. And, y'know, be better to long-time Fedora ecosystem friends.

dagwieers23• 7 years ago

Well, I had to find out this was planned and acted on after the decision was made and the alternative was written without any regards of the original project.

And that is fine if it would stay pcp-dstat as-is.

The most upsetting to me is that Dstat is no longer a python tool you can drop on a JeOS, Synology NAS or a WRT router to get it to work, you now have to install PCP and all its dependencies to make it work, which goes against the original design goals. (i.e. we used to support Python 1.5 for a long time to accommodate RHEL2.1 during its life-cycle)

Also, by taking the Dstat code, removing the plugin mechanism and replacing it with a PCP backend, writing a python plugin for Dstat is no longer possible. This is promoted as being a feature as "your plugin is now a config file" which is a bit disingenuous as you have to write a PCP backend which is a lot harder.

And it is not even a drop-in replacement, it only implements the built-in counters, not the full set of plugins (i.e. --top plugins are missing). So the argument that it needs to work as-is for existing customers does not hold true either.

By taking that name (with the Red Hat clout) there is no chance of anyone taking over maintenance without having to deal with 2 products using the same name, which I guess is forcing your wish on other distributions too.

Wrt. the change was properly announced. I bet the checklist was properly checked. Or to quote Douglas Adams:

“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”

mattdm6• 7 years ago

Well, we do try to do better than the Cottington planning department. Like all changes, it was posted to devel-announce (https://lists.fedoraproject.org/archives/list/devel-announce...) to get more visibility than just a devel-list post (or a random wiki page).

In retrospect, it would have been nice for the devs working on this to contact you directly and explicitly about the proposed change. I'm sorry that didn't happen.

But here we are now -- at this point, what outcome would you like?

rain1• 7 years ago

dagwieers, sorry this happened man that really fucking sucks. They should have done this differently and reached out to you for communication and stuff. Pretty disappointed with them for this. Hope you keep on and find other projects that excite you to work on.

Dstat project ended due to RedHat replacing it with its own dstat tool

Related Stories