Ask HN: How should open source support critical dependencies?

https://news.ycombinator.com/item?id=23715134

36 20 6 years ago

news.ycombinator.com

In something slightly similar to the infamous leftpad issue, many NPM builds started failing yesterday because 'npm audit' now reports a security vulnerability for lodash. lodash is currently in use in over 7 million Github repos, and it's nearly impossible to use NPM without some transitive dependency that depends on lodash.

This security vulnerability was reported in October (https://hackerone.com/reports/712065) and there have been 2 PRs open for 2 months that fix this issue, but lodash hasn't had any releases for a year. There is essentially one (unpaid) person who has power to release lodash, a library that a huge majority of reasonably-sized javascript projects now depend on.

How should the open source community address essentially abandoned projects that have become critical in the ecosystem?

benologist• 6 years ago

I saw a great suggestion the other day -

The npm ecosystem needs something like the distinction between Ubuntu's "main" and "universe" repositories, so that you have a smaller subset of known-good packages with harmonized transitive dependencies, stronger centralized curation, a lot more direct scrutiny, tighter control over versioning, and some party that is responsible for addressing vulnerabilities and ensuring appropriate maintainership.

https://news.ycombinator.com/item?id=23680734

ddevault2• 6 years ago

The approach embodied by package managers like npm - but also Python, Go, and others besides - is fundamentally broken. The software maintainers should not be the sole point of failure in publishing their package.

The approach used by distributions is much, much better. Having a separate team of people responsible for packaging up software is a much better approach. This would allow the security patches to get in, help enforce a sane baseline of packaging standards, prevent the introduction of malware, and have someone accountable whose interests align more with downstream than upstream.

Kinrany2• 6 years ago

Using a package manager like npm does not prevent having a separate team of people responsible for validating and re-publishing packages under their own namespace. A new organization could step in and make their own whitelist at any time.

ddevault2• 6 years ago

It is a cultural problem.

pacifika• 6 years ago

Fork it? The better solution is to use less dependencies. Or in this case write JavaScript the browser can execute today, not to standards that might not be implemented tomorrow.

Or work with a paid version of npm if you’re app is business critical.

renewiltord• 6 years ago

If you're using yarn, apply the patch in your GitHub fork, bump the version, use a resolutions block to point to your repo (I think ref based doesn't work well, might be old info that it doesn't, though).

IMHO coordination based methods will work poorly here.

hannob• 6 years ago

The ability to fork and gain control if necessary is one of the main reasons one wants open source in the first place. So fork it.

Other than that: I find the microdependency / mass dependency trend in NPM very irritating. As a rule of thumb I think you should be able to know at least roughly all your dependencies + transitive dependencies and take some care that they're healthy projects. If you can't name them all - you probably have too many of them.

Kinrany2• 6 years ago

It should be enough to know your direct dependencies.

The problems that can be caught by knowing transitive dependencies but can't be caught by knowing direct dependencies, are rare.

Perhaps open projects could make a pledge of some sort that guarantees important properties as long as all dependencies also commit to the same pledge.

soneil• 6 years ago

I think the problem with forking is you can't take the 'lodash' noun on NPM. So you make a fork of lodash, fix the outstanding issues - but getting every single dependency to actually accept it is a whole 'nother mission. And as long as the original project 'owns' that noun, this is true for everyone.

Does npm have a facility to say a package provides another? Would npm resolve that? Or at least accept that if I have package lojack-mine and it "provides:lojack" then this dependency is resolved?

jka• 6 years ago

Yup, you got it; the migration/reclaiming of package names is pretty key here.

One tricky aspect of the 'metapackage' approach (where, for example, hypothetical 'lodash-legacy' or 'lodash-ng' packages could both fulfil a 'lodash' dependency slot) is that someone needs to be responsible for determining which packages can fill which slots.

In cases where the package alternatives are close forks and API-compatible, that's easier to agree on - but in a large ecosystem, the solution needs to scale in terms of human and machine review time, while also providing a good level of trust to developers when they are selecting or migrating dependencies.

There are likely to be some tricks to reduce the assessment workload required in each case -- running test suites from the 'original' package and applications which rely on it against each candidate, for example -- but ultimately either the package maintainers, the community, the NPM organization themselves -- or some mix of all of the above -- have to determine the packages that could fulfil each slot.

Good packaging is hard, and it's maintenance work rather than green-field development. We (collectively) really should incentivize and fund it more, and ideally learn from and credit existing solutions.

I'm sure there are people who enjoy reviewing (and automating review of) packages, knowing that they're improving code quality and performance for huge numbers of developers. And similarly, there are plenty of developers who provide useful feedback on package quality by reporting bugs, running benchmarks and writing qualitative comparisons.

To guess at where we may be heading in the near-term: a pure Microsoft-subsidiary ecosystem is developing in this area -- GitHub and NPM provide all the data and infrastructure to host, scan and gather user feedback for the processes mentioned above.

I think they'll collectively do a good job, although any approach does take time to build and deploy. I imagine it'll also raise longer-term questions around the governance of open source software development and what role Microsoft would like to take in that.

On a more abstract level the problem space relates to network effects and influence goals. Microsoft believes that the open source developer community is either here to stay, or is here at least for a sufficient duration of time for them to be important -- and it would like to provide influence and guide rails to that community.

hn_throwaway_99• 6 years ago

If you can't name them all - you probably have too many of them.

Well, I want a unicorn, too. That is simply not feasible for anyone using any common javascript/node frameworks - many people's builds started failing with literally thousands of `npm audit` failures, even when their direct dependencies were relatively small.

Github/Microsoft just bought NPM, I really hope they are able to add some features that make it easier to specify different resolution algorithms when pulling dependencies so it's easy for any individual to apply a patch up-and-down the dependency chain, even if the main project hasn't been updated.

MattGaiser• 6 years ago

Really there just need to be more people involved in open source. The web is being built on the unpaid contributions of a few and when they don’t have time or mess up due to being rushed, piles of things collapse.

I suspect if a human resource audit of open source were done, there would be surprisingly few people involved in the maintenance of these things.

codegeek• 6 years ago

I think companies with deep pockets need to step up a bit. I mean if anyone at FAANG etc are using packages like lodash, they should ideally team up and create some type of a funded group/co-op that funds these libraries and their maintenance and removes dependency from one person. You cannot just "charity" the way out of these situations.

mahmoudimus2• 6 years ago

Completely agree. I hate that companies do not give back to this -- but it is changing and that's a positive.

Stripe & Segment have their open source grants, WhatsApp Co-Founder (Jan Koum) donated $1MM USD to FreeBSD, DuckDuckGo donates to companies since 2011 (https://duckduckgo.com/donations) etc.

At the company I have founded, I am putting my money where my mouth is -- we have a budget to allocate up to $25,000 per year to open source products that we find value in. As we hit certain revenue milestones, I have every intention of donating an even larger % to funding open source initiatives as part of our "giving back" corporate social responsibility.

It is a shame most companies do not do the same. I cannot change them, but I can control change within what I do. So, thank you to all the makers and builders out there, I will make it my personal mission to figure out a way to give back to you!

gingerlime2• 6 years ago

I want to do it at my company as well (albeit at a much smaller scale), but how? I asked previously without much luck https://news.ycombinator.com/item?id=23439670

(Not to mention try to get it as a taxable donation... which complicates things further)

bjourne• 6 years ago

Good question! One Python example is docopt. It is a very well-liked command line parsing library but haven't seen any maintainer activity in two years. https://github.com/docopt/docopt But people don't realize that so they keep filing bug reports and submitting pull requests.

There is a docopt-ng package here: https://github.com/bazaar-projects/docopt-ng But users haven't found it yet. And sadly, there is no guarantee that the new maintainer won't also lose interest and abandon the project.

Definitely not an ideal situation. Yes, you can fork but how do you get visibility? How can the developer announce to the community that he or she wants to maintain the software?

evolve2k• 6 years ago

Establish an initiate like ‘Ruby Together’

Ruby Together is a grassroots initiative committed to supporting the critical Ruby infrastructure you rely on.
Your financial backing allows Ruby Together to pay expert, professional developers to work on improving critical infrastructure projects.
The Ruby Together board of directors, elected by members, selects the most impactful projects to fund.
Ruby Together publishes monthly updates showing progress and our proven results.

It brings together companies and individuals concerned about stability and health of the ecosystem, provides a aggregated place to do high impact sponsorship of the ecosystem and the elected board then sets about making a detailed workplan and working out the best distribution of funds.

One dev on lodash? Discuss its importance; offer the dev funds to keep things up-to-date and if not them, have the resources to work out another way.

Check out their early reports to get a better idea of the initial refactoring work they did within the ecosystem.

https://rubytogether.org/

zzo38computer• 6 years ago

I mostly try to reduce dependencies, both direct and indirect, regardless of the programming language in use.

Ask HN: How should open source support critical dependencies?

Related Stories