We should all be using dependency cooldowns

https://blog.yossarian.net/2025/11/21/We-should-all-be-using-dependency-cooldowns

by todsacerdoti • 8 months ago

489 272 8 months ago

blog.yossarian.net

elevation• 8 months ago

You could do a lot of this with CI if you scheduled a job to fetch the most recent packages once a month and record a manifest with the current versions, then, if no security issues are reported before the end of the cooldown period, run integration tests against the new manifest. If no tests fail, automatically merge this update into the project.

For projects with hundreds or thousands of active dependencies, the feed of security issues would be a real fire hose. You’d want to use an LLM to filter the security lists for relevance before bringing them to the attention of a developer.

It would be more efficient to centralize this capability as a service so that 5000 companies aren’t all paying for an LLM to analyze the same security reports. Perhaps it would be enough for someone to run a service like cooldown.pypi.org that served only the most vetted packages to everyone.

AndrewDucker• 8 months ago

Why would you use an LLM rather than just sort by CVE score?

ok123456• 8 months ago

CVE scores are junk. Most CVEs are noise.

jayd163• 8 months ago

Doesn't this mean you're leaving yourself open to known vulnerabilities during that "cool down" time?

jcalvinowens17• 8 months ago

Yep. Not only vulnerabilities, but just bugs in general, which usually matter more than than vulnerabilities IMHO.

bityard• 8 months ago

Do you believe new releases don't introduce new bugs?

woodruffw9• 8 months ago

To be clear, there's no reason why you can't update dependencies in advance of a cooldown period. The cooldown is an enforced policy that you can choose to override as needed.

(This also doesn't apply to vulnerabilities per se, since known vulnerabilities typically aren't evaluated against cooldowns by tools like Dependabot.)

swatcoder3• 8 months ago

No.

A sane "cooldown" is just for automated version updates relying on semantic versioning rules, which is a pretty questionable practice in the first place, but is indeed made a lot more safe this way.

You can still manually update your dependency versions when you learn that your code is exposed to some vulnerability that's purportedly been fixed. It's no different than manually updating your dependency version when you learn that there's some implementation bug or performance cliff that was fixed.

You might even still use an automated system to identify these kinds of "critical" updates and bring them to your attention, so that you can review them and can appropriately assume accountability for the choice to incorporate them early, bypassing the cooldown, if you believe that's the right thing to do.

Putting in that effort, having the expertise to do so, and assuming that accountability is kind of your "job" as a developer or maintainer. You can't just automate and delegate everything if you want people to be able to trust what you share with them.

jayd163• 8 months ago

If you could understand the quality of updates you're pulling in, that solves the issue entirely. The point is that you can't.

There's no reason to pretend we live in a world where everyone is manually combing through the source of every dependency update.

zahlman2• 8 months ago

A sane "cooldown" is just for automated version updates relying on semantic versioning rules, which is a pretty questionable practice in the first place, but is indeed made a lot more safe this way.

I'm reminded of how new Setuptools versions are able to cause problems for so many people, basically because install tools default to setting up isolated build environments using the latest version of whatever compatible build backend is specified (which in turn defaults to "any version of Setuptools"). qv. my LWN article https://lwn.net/Articles/1020576/ .

astrobe_• 8 months ago

TFA shows that most vulnerabilities have a "window of opportunity" smaller than one day. Are you anxious going on week-end because Friday evening a zero-day or a major bug could be made public?

jayd163• 8 months ago

Well then you agree that the answer is yes. At the end of the article a 14 day window is mentioned but not dismissed and does not mention the downsides.

Havoc• 8 months ago

we should all

Except if everyone does it chance of malicious things being spotted in source also drops by virtue of less eyeballs

Still helps though in cases where maintainer spot it etc

smaudet3• 8 months ago

also drops by virtue of less eyeballs

I don't think the people automatically updating and getting hit with the supply chain attack are also scanning the code, I don't think this will impact them much.

If instead, updates are explicitly put on cooldowns, with the option of manually updating sooner, then there would be more eyeballs, not fewer, as people are more likely to investigate patch notes, etc., possibly even test in isolation...

woodruffw9• 8 months ago

(Author of the post.)

The underlying premise here is that supply chain security vendors are honest in their claims about proactively scanning (and effectively detecting + reporting) malicious and compromised packages. In other words, it's not about eyeballs (I don't think people who automatically apply Dependabot bumps are categorically reading the code anyways), but about rigorous scanning and reporting.

mkoubaa3• 8 months ago

Maybe it's more about mileage (attacks that actually happen) than eyeballs.

tjpnz• 8 months ago

You might read the source if something breaks but in a successful supply chain attack that's unlikely to happen. You push to production, go home for the evening and maybe get pinged about it by some automation in a few weeks.

pico303• 8 months ago

I agree. This "cooldown" approach seems antithetical to some basic tenants of security in the open source world, namely that more eyeballs makes for better code. If we all stop looking at or using the thing, are these security professionals really going to find the supply-chain problems for us in the thing, for free?

Instead of a period where you don't use the new version, shouldn't we instead be promoting a best practice of not just blindly using a package or library in production? This "cooldown" should be a period of use in dev or QA environments while we take the time to investigate the libraries we use and their dependencies. I know this can be difficult in many languages and package managers, given the plethora of libraries and dependencies (I'm looking at you in particular JavaScript). But "it's hard" shouldn't really be a good excuse for our best efforts to maintain secure and stable applications.

jcalvinowens17• 8 months ago

I hate this. Delaying real bugfixes to achieve some nebulous poorly defined security benefit is just bad engineering.

wrs2• 8 months ago

Do you upgrade all your dependencies every day? If not, then there’s no real difference in upgrading as if it were 7 days ago.

jerlam2• 8 months ago

Your CI/CD might be setup to upgrade all your dependencies on every build.

ktpsns• 8 months ago

Unattended upgrades for server installations are very common. For instance, for Ubuntu/Debian this updates by default daily (source: https://documentation.ubuntu.com/server/how-to/software/auto...). No cooldown implemented, AFAIK.

Of course we talk about OS security upgrades here, not library dependencies. But the attack vector is similar.

jcalvinowens17• 8 months ago

I upgrade all dependencies every time I deploy anything. If you don't, a zero day is going to bite you in the ass: that's the world we now live in.

If upgrading like that scares you, your automated testing isn't good enough.

On average, the most bug free Linux experience is to run the latest version of everything. I wasted much more time backporting bugfixes before I started doing that, than I have spent on new bugs since.

midasz• 8 months ago

Renovate (dependabot equiv I think) creates PRs, I usually walk through them every morning or when there's a bit of downtime. Playing with the idea to automerge patches and maybe even minor updates but up until now it's not that hard to keep up.

swatcoder3• 8 months ago

The point is to apply a cooldown to your "dumb" and unaccountable automation, not to your own professional judgment as an engineer.

If there's a bugfix or security patch that applies to how your application uses the dependency, then you review the changes, manually update your version if you feel comfortable with those changes, and accept responsibility for the intervention if it turns out you made a mistake and rushed in some malicious code.

Meanwhile, most of the time, most changes pushed to dependencies are not even in the execution path of any given application that integration with them, and so don't need to be rushed in. And most others are "fixes" for issues that were apparently not presenting an eminent test failure or support crisis for your users and don't warrant being rushed in.

There's not really a downside here, for any software that's actually being actively maintained by a responsible engineer.

jcalvinowens17• 8 months ago

You're not thinking about the system dependencies.

Meanwhile, most of the time, most changes pushed to dependencies are not even in the execution path of any given application that integration with them

Sorry, this is really ignorant. You don't appreciate how much churn their is in things like the kernel and glibc, even in stable branches.

33a• 8 months ago

A lot of security problems can be solved by moving slower.

solarist2• 8 months ago

All of security problems can be solved by not moving at all.

kykat2• 8 months ago

What about food security

theoldgreybeard• 8 months ago

jokes on them I already have 10 year dependency cooldowns on the app I work on at work!

cheschire• 8 months ago

Retirement Driven Development

nitwit005• 8 months ago

This did make me think of our app running on Java 8.

Although, I suppose we've probably updated the patch version.

marcosdumay• 8 months ago

There's always the one that requires Java 8, and the one that requires Java >= 11.

layer82• 8 months ago

People in this thread are worried that they are significantly vulnerable if they don't update right away. However, this is mostly not an issue in practice. A lot of software doesn't have continuous deployment, but instead has customer-side deployment of new releases, which follow a slower rhythm of several weeks or months, barring emergencies. They are fine. Most vulnerabilities that aren't supply-chain attacks are only exploitable under special circumstances anyway. The thing to do is to monitor your dependencies and their published vulnerabilities, and for critical vulnerabilities to assess whether your product is affect by it. Only then do you need to update that specific dependency right away.

silvestrov• 8 months ago

I think the main question is: do your app get unknown input (i.e. controlled by other people).

Browsers get a lot of unknown input, so they have to update often.

A Weather app is likely to only get input from one specific site (controlled by the app developers), so it should be relatively safe.

duped• 8 months ago

A million times this. You update a dependency when there are bug fixes or features that you need (and this includes patching vulnerabilities!). Those situations are rare. Otherwise you're just introducing risk into your system - and not that you're going to be caught in some dragnet supply chain attack, but that some dependency broke something you relied on by accident.

Dependencies are good. Churn is bad.

latentsea2• 8 months ago

I'd rather have excellent quality test suites, and pipelines I can have a high degree of confidence in to catch these issues and mitigate the risk, than dependencies that basically never get updated and then become very difficult to do so, which just introduce different classes of risk.

jbkgujklgui3• 8 months ago

Vulnerabilities are not rare, the two most popular programming languages, javascript (node) and python, have 1000+ CVEs in their official docker images. I.e. in practice useless and shouldn't be used by anyone for anything.

embedding-shape2• 8 months ago

for critical vulnerabilities to assess whether your product is affect by it. Only then do you need to update that specific dependency right away.

This is indeed what's missing from the ecosystem at large. People seem to be under the impression that if a new release of software/library/OS/application is released, you need to move to it today. They don't seem to actually look through the changes, only doing that if anything breaks, and then proceed to upgrade because "why not" or "it'll only get harder in the future", neither which feel like solid choices considering the trade-offs.

While we've seen to already have known that it introduces massive churn and unneeded work, it seems like we're waking up to the realization that it is a security tradeoff as well, to stay at the edge of version numbers. Sadly, not enough tooling seems to take this into account (yet?).

hypeatei2• 8 months ago

Sadly, not enough tooling seems to take this into account

Most tooling (e.g. Dependabot) allows you to set an interval between version checks. What more could be done on that front exactly? Devs can already choose to check less frequently.

tracnar• 8 months ago

You could use this funky tool from oss-rebuild which proxies registries so they return the state they were at a past date: https://github.com/google/oss-rebuild/tree/main/cmd/timewarp

pas• 8 months ago

"it'll only get harder in the future"

that's generally true, no?

of course waiting a few days/weeks should be the minimum unless there's a CVE (or equivalent) that's applies

jerf2• 8 months ago

I fought off the local imposition of Dependabot by executive fiat about a year ago by pointing out that it maximizes vulnerabilities to supply chain attacks if blindly followed or used as a metric excessively stupidly. Maximizing vulnerabilities was not the goal, after all. You do not want to harass teams with the fact that DeeplyNestedDepen just went from 1.1.54-rc2 to 1.1.54-rc3 because the worst case is that they upgrade just to shut the bot up.

I think I wouldn't object to "Dependabot on a 2-week delay" as something that at least flags. However working in Go more than anything else it was often the case even so that dependency alerts were just an annoyance if they aren't tied to a security issue or something. Dynamic languages and static languages do not have the same risk profiles at all. The idea that some people have that all dependencies are super vital to update all the time and the casual expectation of a constant stream of vital security updates is not a general characteristic of programming, it is a specific characteristic not just of certain languages but arguably the community attached to those languages.

(What we really need is capabilities, even at a very gross level, so we can all notice that the supposed vector math library suddenly at version 1.43.2 wants to add network access, disk reading, command execution, and cryptography to the set of things it wants to do, which would raise all sorts of eyebrows immediately, even perhaps in an automated fashion. But that's a separate discussion.)

dap• 8 months ago

At my last job, we only updated dependencies when there was a compelling reason. It was awful.

What would happen from time to time was that an important reason did come up, but the team was now many releases behind. Whoever was unlucky enough to sign up for the project that needed the updated dependency now had to do all those updates of the dependency, including figuring out how they affected a bunch of software that they weren't otherwise going to work on. (e.g., for one code path, I need a bugfix that was shipped three years ago, but pulling that into my component affects many other code paths.) They now had to go figure out what would break, figure out how to test it, etc. Besides being awful for them, it creates bad incentives (don't sign up for those projects; put in hacks to avoid having to do the update), and it's also just plain bad for the business because it means almost any project, however simple it seems, might wind up running into this pit.

I now think of it this way: either you're on the dependency's release train or you jump off. If you're on the train, you may as well stay pretty up to date. It doesn't need to be every release the minute it comes out, but nor should it be "I'll skip months of work and several major releases until something important comes out". So if you decline to update to a particular release, you've got to ask: am I jumping off forever, or am I just deferring work? If you think you're just deferring the decision until you know if there's a release worth updating to, you're really rolling the dice.

(edit: The above experience was in Node.js. Every change in a dynamically typed language introduces a lot of risk. I'm now on a team that uses Rust, where knowing that the program compiles and passes all tests gives us a lot of confidence in the update. So although there's a lot of noise with regular dependency updates, it's not actually that much work.)

stefan_• 8 months ago

Thats because the security industry has been captured by useless middle manager types who can see that "one dependency has a critical vulnerability", but could never in their life scrounge together the clue to analyze the impact of that vulnerability correctly. All they know is the checklist fails, and the checklist can not fail.

(Literally at one place we built a SPA frontend that was embedded in the device firmware as a static bundle, served to the client and would then talk to a small API server. And because these NodeJS types liked to have libraries reused for server and frontend, we would get endless "vulnerability reports" - but all of this stuff only ever ran in the clients browser!)

patrick451• 8 months ago

Jumping straight to the new release because it fixed one security bug has always struck me as a round about way of trying to achieve security through obscurity, especially when the releases include tons of other changes. Yes, this release fixed CVE-123, but how many new ones were added?

adamanteye• 8 months ago

I once almost managed to get back on the release train. I was tasked with adding new features to a piece of software originally developed years ago for internal use at a university. It was running PHP 5 on Debian Jessie. The first hurdle was that no Docker image existed for the production environment (PHP 5). After patching all the code and getting it to run on PHP 8, another issue surfaced: the MSSQL server it needed to communicate with only supported TLS 1.0, which had been removed from the OpenSSL version included in Debian 12—the base image for my PHP 8 setup.

In the end, I decided to implement a lightweight PHP 5 relay to translate SQL requests so the MSSQL server could still be accessed. The university stuffs were quite satisfied with my work. But I really felt guilty for the next guy who will touch this setup. So I still didn’t quite make it back onto the release train… as that PHP 5 relay counts.

jerf2• 8 months ago

Also, if you are updating "right away" it is presumably because of some specific vulnerability (or set of them). But if you're in an "update right now" mode you have the most eyes on the source code in question at that point in time, and it's probably a relatively small patch for the targeted problem. Such a patch is the absolute worst time for an attacker to try to sneak anything in to a release, the exact and complete opposite of the conditions they are looking for.

Nobody is proposing a system that utterly and completely locks you out of all updates if they haven't aged enough. There is always going to be an override switch.

justsomehnguy• 8 months ago

People in this thread are worried that they are significantly vulnerable if they don't update right away

Most of them assume what if they are working on some public accessible website then 99% of the people and orgs in the world are running nothing but some public accessible website.

bumblehean2• 8 months ago

The thing to do is to monitor your dependencies and their published vulnerabilities, and for critical vulnerabilities to assess whether your product is affect by it. Only then do you need to update that specific dependency right away.

The practical problem with this is that many large organizations have a security/infosec team that mandates a "zero CVE" posture for all software.

Where I work, if our infosec team's scanner detect a critical vulnerability in any software we use, we have 7 days to update it. If we miss that window we're "out of compliance" which triggers a whole process that no one wants to deal with.

The path of least resistance is to update everything as soon as updates are available. Consequences be damned.

BrenBarn2• 8 months ago

The practical problem with this is that many large organizations have a security/infosec team that mandates a "zero CVE" posture for all software.

The solution is to fire those teams.

tetha• 8 months ago

I really dislike that approach. We're by now evaluating high-severity CVEs ASAP in a group to figure out if we are affected, and if mitigations apply. Then there is the choice of crash-patching and/or mitigating in parallel, updating fast, or just prioritizing that update more.

We had like 1 or 2 crash-patches in the past - Log4Shell was one of them, and blocking an API no matter what in a component was another one.

In a lot of other cases, you could easily wait a week or two for directly customer facing things.

hiAndrewQuinn• 8 months ago

You view this as a burden, but (at least if you operate in the EU) I’d argue you’re actually looking at a competitive advantage that hasn't cashed out yet.

Come 2027-12, the Cyber Resilience Act enters full enforcement. The CRA mandates a "duty of care" for the product's lifecycle, meaning if a company blindly updates a dependency to clear a dashboard and ships a supply-chain compromise, they are on the hook for fines up to €15M or 2.5% of global turnover.

At that point, yes, there is a sense in which the blind update strategy you described becomes a legal liability. But don't miss the forest for the trees, here. Most software companies are doing zero vetting whatsoever. They're staring at the comet tail of an oncoming mass extinction event. The fact that you are already thinking in terms of "assess impact" vs. "blindly patch" already puts your workplace significantly ahead of the market.

ozim2• 8 months ago

Fun part is people are worried about 0 days but in reality most problems come from 300 or 600 days old not patched vulns.

nrhrjrjrjtntbt2• 8 months ago

I agree. Pick yout poison. my poison is waiting before upgrades, assess zero days case by case.

weinzierl2• 8 months ago

"The thing to do is to monitor your dependencies and their published vulnerabilities, and for critical vulnerabilities to assess whether your product is affect by it."

Yes

"Only then do you need to update that specific dependency right away."

Big no. If you do that it is guaranteed one day you miss a vulnerability that hurts you.

To frame it differently: What you propose sounds good in theory but in practice the effort to evaluate vulnerabilities against your product will be higher than the effort to update plus taking appropriate measures against supply chain attacks.

slaymaker1907• 8 months ago

And it needs to be said that you generally cannot tell if a vulnerability is critical for a given application except by evaluating the vulnerability in the context of said application. One that I've seen is some critical DoS vulnerability due to a poorly crafted regex. That sort of vulnerability is only relevant if you are passing untrusted input to that regex.

cosmic_cheese• 8 months ago

I think there’s a much stronger argument for policies that both limit the number and complexity of dependencies. Don’t add it unless it’s highly focused (no “everything libraries” that pull in entire universes of their own) and carries a high level of value. A project’s entire dependency tree should be small and clean.

Libraries themselves should perhaps also take a page from the book of Linux distributions and offer LTS (long term support) releases that are feature frozen and include only security patches, which are much easier to reason about and periodically audit.

pengaru• 8 months ago

I'd be willing to pay $100 to upvote your comment 100x.

Y_Y• 8 months ago

How do you think dang puts bread on the table?

tcfhgj• 8 months ago

Won't using highly focused dependencies increase the amount of dependencies?

Limiting the number of dependencies, but then rewriting them in your own code, will also increase the maintenance burden and compile times

skydhash2• 8 months ago

A lot of projects are using dependencies, but are only using a small part of. Or are using them in a single place for a single usecase. Like bringing in formik (npm), but you only have one single form. Or moment, because you want to format a single date.

buu7005• 8 months ago

I think AI nudges the economics more in this direction as well. Adding a non-core dependency has historically bought short-term velocity in exchange for different long-term maintenance costs. With AI, there are now many more cases where a first-party implementation becomes cheaper/easier/faster in both the short term and the long term.

Of course it's up to developers to weigh the tradeoffs and make reasonable choices, but now we have a lot more optionality. Reaching for a dependency no longer needs to be the default choice of a developer on a tight timeline/budget.

xmodem5• 8 months ago

Let's have AI generate the same vulnerable code across hundreds of projects, most of which will remain vulnerable forever, instead of having those projects all depend on a central copy of that code that can be fixed and distributed once the issue gets discovered. Great plan!

xmodem5• 8 months ago

I've seen this argument made frequently. It's clearly a popular sentiment, but I can't help feel that it's one of those things that sounds nice in theory if you don't think about it too hard. (Also, cards on the table, I personally really like being able to pull in a tried-and-tested implementation of code to solve a common problem that's also used by in some cases literally millions of other projects. I dislike having to re-solve the same problem I have already solved elsewhere.)

Can you cite an example of a moderately-widely-used open source project or library that is pulling in code as a dependency that you feel it should have replicated itself?

What are some examples of "everything libraries" that you view as problematic?

skydhash2• 8 months ago

Anything that pulled in chalk. You need a very good reason to emit escape sequences. The whole npm (and rust, python,..) ecosystem assumes that if it’s a tty, then it’s a full blown xterm-256color terminal. And then you need to pipe to cat or less to have sensible output.

So if you’re adding chalk, that generally means you don’t know jack about terminals.

ted_dunning2• 8 months ago

The problem isn't the implementation of what I want to do. It's all of the implementations of things I never cared about doing. And the implementation of what I want to do that is soooo much more complex than it needs to be that I could easily have implemented it myself in less time.

The problem is also less about the implementation I want, it's about the 10,000 dependencies of things I don't really want. All of those are attack surface much larger than some simple function.

igregoryca2• 8 months ago

Most of your supply chain attack surface is social engineering attack surface. Doesn't really matter if I use Lodash, or 20 different single-function libraries, if I end up trusting the exact same people to not backdoor my server.

Of course, small libraries get a bad rap because they're often maintained by tons of different people, especially in less centralized ecosystems like npm. That's usually a fair assessment. But a single author will sometimes maintain 5, 10, or 20 different popular libraries, and adding another library of theirs won't really increase your social attack surface.

So you're right about "pull[ing] in universes [of package maintainers]". I just don't think complexity or number of packages are the metrics we should be optimizing. They are correlates, though.

(And more complex code can certainly contain more vulnerabilities, but that can be dealt with in the traditional ways. Complexity begets simplicity, yadda yadda; complexity that only begets complexity should obviously be eliminated)

mkoubaa3• 8 months ago

The lower level the dependency is, the more unjustifiable it is for it to have its own dependencies. This ought to be a point of competition between libraries and often is, at least in the c++ world

gr4vityWall4• 8 months ago

The Debian stable model of having a distro handle common dependencies with a full system upgrade every few years looks more and more sane as years pass.

It's a shame some ecosystems move waaay too fast, or don't have a good story for having distro-specific packages. For example, I don't think there are Node.js libraries packaged for Debian that allow you to install them from apt and use it in projects. I might be wrong.

kykat2• 8 months ago

It is possible to work with rust, using debian repositories as the only source.

embedding-shape2• 8 months ago

For example, I don't think there are Node.js libraries packaged for Debian that allow you to install them from apt and use it in projects

Web search shows some: https://packages.debian.org/search?keywords=node&searchon=na... (but also shows "for optimizing reasons some results might have been suppressed" so might not be all)

Although probably different from other distros, Arch for example seems to have none.

o11c3• 8 months ago

Locally, you can do:

  apt-cache showpkg 'node-*' | grep ^Package:

which returns 4155 results, though 727 of them are type packages.

Using these in commonjs code is trivial; they are automatically found by `require`. Unfortunately, system-installed packages are yet another casualty of the ESM transition ... there are ways to make it work but it's not automatic like it used to be.

noosphr2• 8 months ago

Never mistake motion for action.

An eco system moving too quickly, when it isn't being fundamentally changed, isn't a sign of a healthy ecosystem, but of a pathological one.

No one can think that js has progressed substantially in the last three years, yet trying to build any project three years old without updates is so hard a rewrite is a reasonable solution.

gr4vityWall4• 8 months ago

No one can think that js has progressed substantially in the last three years

Are we talking about the language, or the wider ecosystem?

If the latter, I think a lot of people would disagree. Bun is about three years old.

Other significant changes are Node.js being able to run TypeScript files without any optional flags, or being able to use require on ES Modules. I see positive changes in the ecosystem in recent years.

mkoubaa3• 8 months ago

The stable model usually implies that your app has to target both the old and the new distro version for a while. That is a bit too much to ask for some, unfortunately

l9o• 8 months ago

I wonder if LLMs can help here to some extent. I agree with others that cooldowns aren't helpful if everyone is doing it.

I've been working on automatic updates for some of my [very overengineered] homelab infra and one thing that I've found particularly helpful is to generate PRs with reasonable summaries of the updates with an LLM. it basically works by having a script that spews out diffs of any locks that were updated in my repository, while also computing things like `nix store diff-closures` for the before/after derivations. once I have those diffs, I feed them into claude code in my CI job, which generates a pull request with a nicely formatted output.

one thing I've been thinking is to lookup all of those dependencies that were upgraded and have the LLM review the commits. often claude already seems to lookup some of the commits itself and be able to give a high level summary of the changes, but only for small dependencies where the commit hash and repository were in the lock file.

it would likely not help at all with the xz utils backdoor, as IIRC the backdoor wasn't even in the git repo, but on the release tarballs. but I wonder if anyone is exploring this yet?

robszumski• 8 months ago

We built and launched this product about 2 months ago, HN thread here: https://news.ycombinator.com/item?id=45439721

Totally agree that AI is great for this, it will work harder and go deeper and never gets tired of reading code or release notes or migration guides. What you want instead of summaries is to find the breaking changes, figure out if they matter, then comment on _that_.

perlgeek• 8 months ago

Some scattered thoughts on that:

* If everybody does it, it won't work so well

* I've seen cases where folks pinned their dependencies, and then used "npm install" instead of "npm ci", so the pinning was worthless. Guess they are the accidental, free beta testers for the rest of us.

* In some ecosystems, distributions (such as Debian) does both additional QA, and also apply a cooldown. Now we try to retrofit some of that into our package managers.

exasperaited• 8 months ago

* If everybody does it, it won't work so well

Indeed, this is a complex problem to solve.

And the "it won't work so well" of this is probably a general chilling effect on trying to fix things because people won't roll them out fast enough anyway.

This may seem theoretical but for example in websites where there are suppliers and customers, there's quite a chilling effect on any mechanism that encourages people to wait until a supplier has positive feedback; there are fewer and fewer people with low enough stakes who are willing to be early adopters in that situation.

What this means is that new suppliers often drop out too quickly, abandon platforms, work around those measures in a way that reduces the value of trust, and worse still there's a risk of bad reviews because of the reviewer's Dunning-Kruger etc.

I think the mechanism is important for people who really must use it, but there will absolutely be side effects that are hard to qualify/correct.

layer82• 8 months ago

"If everybody does it, <some outcome>" is rarely a good argument, because the premise rarely becomes reality.

dugidugout• 8 months ago

If you're prescribing a public practice you intend to be taken seriously, you should contend with what happens at scale, especially when evaluating competing alternatives, where comparative scale effects become decisive. Given the article's push for ecosystem-wide support, there's good reason to probe this hypothetical.

rmunn2• 8 months ago

So true.

Things that everybody does: breathe. Eat. Drink. Sleep. And a few other things that are essential to being alive.

Things that not everybody does: EVERYTHING else.

K0nserv• 8 months ago

This is not how `npm install` works. This misunderstanding is so pervasive. Unless you change stuff in `package.json` `npm install` will not update anything, it still installs based on package-lock.json.

Quoting from the docs:

This command installs a package and any packages that it depends on. If the package has a package-lock, or an npm shrinkwrap file, or a yarn lock file, the installation of dependencies will be driven by that [..]

cxr3• 8 months ago

What everyone should all be doing is practicing the decades-old discipline of source control. Attacks of the form described in the post, where a known-good, uncompromised dependency is compromised at the "supply chain" level, can be 100% mitigated—not fractionally or probabilistically—by cutting out the vulnerable supply chain. The fact that people are still dragging their feet on this and resist basic source control is the only reason why this class of attack is even possible. That vendoring has so many other benefits and solves other problems is even more reason to do so.

Stacking up more sub-par tooling is not going to solve anything.

Fortunately this is a problem that doesn't even have to exist, and isn't one that anyone falls into naturally. It's a problem that you have to actively opt into by taking steps like adding things to .gitignore to exclude them from source control, downloading and using third-party tools in a way that introduces this and other problems, et cetera—which means you can avoid all of it by simply not taking those extra steps.

(Fun fact: on a touch-based QWERTY keyboard, the gesture to input "vendoring" by swiping overlaps with the gesture for "benefitting".)

dbdr• 8 months ago

Doesn't vendoring solve the supply chain issue in the same way as picking a dependency version and never upgrading would? (especially if your package manager includes a hash of the dependency in a lock file)

cxr3• 8 months ago

Mm, no.

The practice I described is neither "never upgrading" your dependencies nor predicated upon that. And your comment ignores that there's an entire other class of problem I alluded to (reproducibility/failure-to-reproduce) that normal, basic source control ("vendoring") sidesteps almost entirely, if not entirely in the majority of cases. That, too, is without addressing the fact that what you're tacitly treating as bedrock is not, in fact, bedrock (even if it can be considered the status quo within some subculture).

The problem of "supply chain attacks" (which, to reiterate, is coded language for "not exercising source control"—or, "pretending to, but not actually doing it") is a problem that people have to opt in to.

It takes extra tooling.

It takes extra steps.

You have to actively do things to get yourself into the situation where you're susceptible to it—and then it takes more tooling and more effort to try and get out. In contrast, you can protect yourself from it +and+ solve the reproducibility problem all by simply doing nothing[1]—not taking the extra steps to edit .gitignore to exclude third-party code (and, sometimes, first-party code) from the SCM history, not employing lock files to undo the consequences of selectively excluding key parts of the input to the build process (and your app's behavior).

All these lock file schemes that language package manager projects have belatedly and sheepishly began to offer as a solution are inept attempts at doing what your base-level RCS already does for free (if only all the user guides for these package managers weren't all instructing people to subvert source control to begin with).

Every package manager lock file format (requirements file, etc.) is an inferior, ad hoc, formally-specified, error-prone, incompatible re-implementation of half of Git.

1. <https://news.ycombinator.com/item?id=25623388>

jhatemyjob3• 8 months ago

That's what I do for some of my dependencies. But it's only for legal reasons. If licensing isn't a concern, vendoring is superior.

lijok2• 8 months ago

Vendoring hasn’t been feasible since CI became free for OSS.

You cannot vendor yourself out of a nuclear waste pile that is the modern OSS ecosystem.

cxr3• 8 months ago

Not sure what any of what you wrote has to do with the subject at hand. Your comments are vague and come across as non-sequiturs at best.

The only thing I've claimed is that keeping dependencies under source control neutralizes the supply chain attacks that the author of the post describes.

They each belong to two totally different genres of comment.

If you have something concrete to say about the relationship between supply chain attacks and dependencies being excluded from source control in lieu of late-fetching them right at/before build time, then go for it.

callamdelaney• 8 months ago

Anything to save us from not being able to apply because of a CVE which is only relevant if you're doing something which we don't.

pimlottc• 8 months ago

This assumes that most exploits are discovered by pro-active third-party security vendors, instead of being noticed in deployed projects. Is this actually true?

woodruffw9• 8 months ago

Is this actually true?

I don't know, but it's the claimed truth from a lot of vendors! The value proposition for a lot of supply chain security products is a lot weaker if their proactive detection isn't as strong as claimed.

nicoburns• 8 months ago

I would like to see a variant of this that is based on a manual review/audit process rather than a time-based cooldown.

Something like, upgrade once there are N independent positive reviews AND less than M negative reviews (where you can configure which people are organisations you trust to audit). And of course you would be able to audit dependencies yourself (and make your review available for others).

testplzignore• 8 months ago

I've been wanting something like this for years. It's simply impossible for millions of companies to individually review the thousands of updates made to their thousands of dependencies every day.

Imagine a world where every software update has hundreds of signoffs from companies across the industry. That is achievable if we work together. For only a few minutes a day, you too can save a CVSS 10.0 vulnerability from going unpatched :)

DrScientist3• 8 months ago

The think I find most odd about the constant pressure to update to the most recent and implied best version is that there is some implicit belief that software get's uniformly better with each release.

Bottom line those security bugs are not all from version 1.0 , and when you update you may well just be swapping known bugs for unknown bugs.

As has been said elsewhere - sure monitor published issues and patch if needed but don't just blindly update.

switchbak• 8 months ago

I remember this used to actually be the case, but that was many moons ago when you'd often wait a long time between releases. Or maybe the quality bar was lower generally, and it was just easier to raise it?

These days it seems most software just changes mostly around the margins, and doesn't necessarily get a whole lot better. Perhaps this is also a sign I'm using boring and very stable software which is mostly "done"?

DrScientist3• 8 months ago

The question is do you take control of your update cycle - or outsource it.

That's balancing the effort and risk of managing it yourself versus the risk and busy work generated from dependency churn.

What evergreen update policies have done is increase development velocity ( much easier to make breaking changes if you assume everyone is upto date everywhere ) - but also increases churn.

cesarb2• 8 months ago

Bottom line those security bugs are not all from version 1.0, and when you update you may well just be swapping known bugs for unknown bugs.

One great example of that is log4shell. If you were still using version 1.0 (log4j 1.x), you were not vulnerable, since the bug was introduced in version 2.0 (log4j 2.x). There were some known vulnerabilities in log4j 1.x, but the most common configuration (logging only to a local file or to the console, no remote logging or other exotic stuff) was not affected by any of them.

jbkgujklgui3• 8 months ago

The infamous log4j vulnerability was actually a feature when it was introduced. It was only several years later it was considered a security vulnerability. Countless of other techs has had the same problem, for example, ActiveX. Same with most downgrade attacks, they were most often considered a good thing (better compatibility) when introduced, only much later were such features considered non-good.

DrScientist3• 8 months ago

Funny you should mention that - amid the general log4j panic the mere presence of old version on disk would send IT security into a frenzy, irrespective of whether the old version was vulnerable or was deployed in a way where the vulnerability could be accessed.

I had to point out it was because we hadn't updated was the reason our stuff wasn't vulnerable.

edoceo• 8 months ago

I do this with my pull-thru-proxy. When new stuff shows up, I know, makes it easy to identify if urgent update is needed. I usually lag a bit. Make a dependency update a dedicated process. Update a few at a time, test, etc. slower but stable.

programmertote• 8 months ago

I know it's impossible in some software stack and ecosystem. But I live mostly in the data world, so I usually could get away from such issues by aggressively keeping my upstream dependency list lean.

P.S. When I was working at Amazon, I remember that a good number of on-call tickets were about fixing dependencies (in most of them are about updating the outdated Scala Spark framework--I believe it was 2.1.x or older) and patching/updating OS'es in our clusters. What the team should have done (I mentioned this to my manager) is to create clusters dynamically (do not allow long-live clusters even if the end users prefer it that way), and upgrading the Spark library. Of course, we had a bunch of other annual and quarterly OKRs (and KPIs) to meet, so updating Spark got the lowest of priorities...

jhatemyjob3• 8 months ago

This is a decent approach, another approach is to vendor your dependencies and don't update them unless absolutely necessary. The only thing you'd need to react to is major security vulnerabilities.

compumike• 8 months ago

There's a tradeoff and the assumption here (which I think is solid) is that there's more benefit from avoiding a supply chain attack by blindly (by default) using a dependency cooldown vs. avoiding a zero-day by blindly (by default) staying on the bleeding edge of new releases.

It's comparing the likelihood of an update introducing a new vulnerability to the likelihood of it fixing a vulnerability.

While the article frames this problem in terms of deliberate, intentional supply chain attacks, I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.

On the unintentional bug/vulnerability side, I think there's a similar argument to be made. Maybe even SemVer can help as a heuristic: a patch version increment is likely safer (less likely to introduce new bugs/regressions/vulnerabilities) than a minor version increment, so a patch version increment could have a shorter cooldown.

If I'm currently running version 2.3.4, and there's a new release 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm probably better off waiting N days, or until 2.4.1 comes out and fixes the new bugs introduced by 2.4.0!

woodruffw9• 8 months ago

Yep, that's definitely the assumption. However, I think it's also worth noting that zero-days, once disclosed, do typically receive advisories. Those advisories then (at least in Dependabot) bypass any cooldown controls, since the thinking is that a known vulnerability is more important to remediate than the open-ended risk of a compromised update.

I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.

Yes, absolutely! The overwhelming majority of vulnerabilities stem from normal accidental bug introduction -- what makes these kinds of dependency compromises uniquely interesting is how immediately dangerous they are versus, say, a DoS somewhere in my network stack (where I'm not even sure it affects me).

mik3y• 8 months ago

Could a supply chain attacker simulate an advisory-remediating release somehow, i.e., abuse this feature to bypass cooldowns?

hinkley• 8 months ago

Defaults are always assumptions. Changing them usually means that you have new information.

Too• 8 months ago

You should also factor in that a zero-day often isn’t surfaced to be exploitable if you are using the onion model with other layers that need to be penetrated together. In contrast to a supply chain vulnerability that is designed to actively make outbound connections through any means possible.

throwawayqqq112• 8 months ago

Thank you. I was scanning this thread for anyone pointing this out.

The cooldown security scheme appears like some inverse "security by obscurity". Nobody could see a backdoor, therefor we can assume security. This scheme stands and falls with the assumed timelines. Once this assumption tumbles, picking a cooldown period becomes guess work. (Or another compliance box ticked.)

On the other side, the assumption can very well be sound, maybe ~90% of future backdoors can be mitigated by it. But who can tell. This looks like the survivorship bias, because we are making decisions based on the cases we found.

goalieca• 8 months ago

I’d estimate the vast majority of CVEs in third party source are not directly or indirectly exploitable. The CVSS scoring system assumes the worst case scenario the module is deployed in. We still have no good way to automate adjusting the score or even just figuring false positive.

lambdaone• 8 months ago

The big problem is the Red Queen's Race nature of development in rapidly-evolving software ecosystems, where everyone has to keep pushing versions forward to deal with their dependencies' changes, as well as any actual software developments of their own. Combine that with the poor design decisions found in rapidly-evolving ecosystems, where everyone assumes anyting can be fixed in the next release, and you have a recipe for disaster.

OhMeadhbh• 8 months ago

I'm not arguing that cooldowns are a bad idea. But I will argue that the article presents a simplified version of user behaviour. One of the reasons people upgrade their dependencies is to get bug fixes and feature enhancements. So there may be significant pressure to upgrade as soon as the fix is available, cooldowns be damned!

If you tell people that cooldowns are a type of test and that until the package exits the testing period, it's not "certified" [*] for production use, that might help with some organizations. Or rather, would give developers an excuse for why they didn't apply the tip of a dependency's dev tree to their PROD.

So... not complaining about cooldowns, just suggesting some verbiage around them to help contextualize the suitability of packages in the cooldown state for use in production. There are, unfortunately, several mid-level managers who are under pressure to close Jira tickets IN THIS SPRINT and will lean on the devs to cut whichever corners need to be cut to make it happen.

[*] for some suitable definition of the word "CERTIFIED."

nine_k• 8 months ago

There is a difference between doing

  $ npm i
  78 packages upgraded

and upgrading just one dependency from 3.7.2_1 to 3.7.2_2, after carefully looking at the code of the bugfix.

The cooldown approach makes the automatic upgrades of the former kind much safer, while allowing for the latter approach when (hopefully rarely) you actually need a fix ASAP.

andrewla• 8 months ago

I'm not convinced that this added latency will help, especially if everyone uses it. It may protect you as long as nobody else uses a cooldown period, but once everyone uses one then the window between the vulnerability being introduced and it being detected will expand by a similar amount, because without exposure it is less likely to be found.

cmckn• 8 months ago

I don’t think it’s bad advice, it really just depends on the project, its dependencies, and your attack surface. I so badly want this era of mindlessly ticking boxes to end. Security is a contact sport! “Best practices” won’t get you to the promised land, you have to actually think critically about the details day to day.

abalone• 8 months ago

Sure, but there's an obvious tradeoff: You're also delaying the uptake of fixes for zero-day vulnerabilities.

The article does not discuss this tradeoff.

woodruffw9• 8 months ago

The article assumes that engineers have the technical wherewithal to know when they should manually upgrade their dependencies!

Clearly I should have mentioned that Dependabot (and probably others) don't consider cooldown when suggesting security upgrades. That's documented here[1].

[1] https://docs.github.com/en/code-security/dependabot/working-...

awesome_dude• 8 months ago

Ye Olde "Cache Invalidation" problems really

Instead of updating the cache of dependencies you have immediately, the suggestion is to use the cooldown to wait....

As you point out, this means that you have a stale cache member has a critical fix applied.

Next week's solution - have a dependency management tool that alerts you when critical fixes are created upstream for dependencies you have

Followed by - now the zero day authors are publishing their stuff as critical fixes...

Hilarity ensues

jimbob45• 8 months ago

Isn't this more or less what the Microsoft Long-Term Support (LTS) versus Short-Term Support (STS) is meant to do? LTS only receives critical updates but eschews all experimental/beta features. STS gets everything for the people that couldn't care less if their app gets hacked (e.g. apps like calculators, sandboxed internal tools, etc).

I know Ubuntu and others do the same but I don't know what they call their STS equivalent.

Marsymars• 8 months ago

Depends which Microsoft products you're referring to. If you're talking about .NET versions, MS explicitly says "The only difference is the length of support."

xg152• 8 months ago

It's a good idea, but not without weak points, I think.

One of the classic scammer techniques is to introduce artificial urgency to prevent the victim from thinking clearly about a proposal.

I think this would be a weakness here as well: If enough projects adopt a "cooldown" policy, the focus of attackers would shift to manipulate projects into making an exception for "their" dependency and install it before the regular cooldown period elapsed.

How to do that? By playing the security angle once again: An attacker could make a lot of noise how a new critical vulnerability was discovered in their project and every dependant should upgrade to the emergency release as quickly as possible, or else - with the "emergency release" then being the actually compromised version.

I think a lot of projects would could come under pressure to upgrade, if the perceived vulnerability seems imminent and the only point for not upgrading is some generic cooldown policy.

__MatrixMan__• 8 months ago

Along those lines: If you're packaging an exploit, it's probably best to fix a bug while you're at it. That way people who want to remove their ugly workarounds will be motivated to violate the dependency cooldown.

mewpmewp2• 8 months ago

How would they create that noise?

xg152• 8 months ago

Depends on the level of infiltration I guess. If the attacker managed to get themselves into a trusted position, as with the XZ backdoor, they could use the official communication channels of the project and possibility even file a CVE.

If it's "only" technical access, it would probably be harder.

gbin• 8 months ago

Feels like the tragedy of the commons: I don't want to look at the change, I don't want to take responsibility, somebody else will take care or it, I just have to wait.

Ok if this is an amazing advice and the entire ecosystem does that: just wait .... then what? We wait even more to be sure someone else is affected first?

Every time I see people saying you need to wait to upgrade it is like you are accumulating tech debt: the more you wait, the more painful the upgrade will be, just upgrade incrementally and be sure you have mitigations like 0 trust or monitoring to cut early any weird behavior.

tempestn• 8 months ago

You're not taking on any meaningful tech debt by waiting a week after a new version goes public to adopt it. As the OP says, there are services that scan popular open source tools for vulnerabilities as soon as they are released; even if a large percentage of the user base is waiting a week to update, many will still be caught in that period. And for various reasons some will still upgrade immediately.

bongodongobob• 8 months ago

This is just completely wrong. If you are talking about a sizeable number of devices, you're not getting anything updated immediately even if you wanted to. You roll out to groups over a period of time because you don't want to break everything if there are unintended consequences. Your personal device? Sure whatever, but any fleet of devices absolutely does not get immediate updates across the board.

catlifeonmars2• 8 months ago

You’re implicitly assuming that it’s exposure to downstream consumers that causes the malicious packages to be discovered, but we haven’t actually seen that in the last couple of major supply chain attacks. Instead it just buys time for the maintainers to undo the damage.

andix3• 8 months ago

Even if less consumers will notice a compromise and report it, it still gives additional time for security researchers to analyze the packages, and for maintainers to notice themselves they got compromised

There are a lot of companies out there, that's scan packages and analyze them. Maintainers might notice a compromise, because a new release was published they didn't authorize. Or just during development, by getting all their bitcoin stolen ;)

PoignardAzur• 8 months ago

You could do it like a brownout, and have a random per-user cooldown going from a day to a month.

Not enough to accumulate tech debt, enough to mitigate the potential impact of any supply-chain vulnerability.

morshu90013• 8 months ago

The update tech debt tends not to compound or even accumulate at all. Usually you'd pay the same to update v1 to v2 as you would for v1 to v3. Maybe skipping v2 has a negative cost even.

cesarb2• 8 months ago

Maybe skipping v2 has a negative cost even.

I've seen this plenty of times: v1 of some library has one way of doing things, v2 of that library changes to a new incompatible way, and then v2.1 introduces a few extra changes to make it easier to port from the v1 way. If you wait a while, you have to do less work to update than if you had updated immediately.

One example is Python 3. After the first few Python 3.x releases, a few "useless" features were introduced to make it easier to port code from Python 2.7 (IIRC, things like reintroducing the u'...' syntax for unicode strings, which had been removed by Python 3.0 since normal '...' strings are now always unicode strings).

aoeusnth1• 8 months ago

Your comment assumes that security issues are only discovered because someone downloads the package. This premise is incorrect.

BrenBarn2• 8 months ago

The culture of constant updates is a cancer on the software world.

ChrisMarshallNY• 8 months ago

I'm an advocate for safe dependency usage.

I can't in good conscience, say "Don't use dependencies," which solves a lot of problems, but I can say "Let's be careful, out there," to quote Michael Conrad.

I strongly suspect that a lot of dependencies get picked because they have a sexy Website, lots of GH stars and buzz, and cool swag.

I tend to research the few dependencies that I use. I don't depend lightly.

I'm also fortunate to be in the position where I don't need too many. I am quite aware that, for many stacks, there's no choice.

jrowen2• 8 months ago

A “cooldown” is exactly what it sounds like: a window of time between when a dependency is published and when it’s considered suitable for use.

My understanding of a cooldown, from video games, is a period of time after using an ability where you can't use it again. When you're firing a gun and it gets too hot, you have to wait while it cools down.

I was trying to apply this to the concept in the article and it wasn't quite making sense. I don't think "cooldown" is really used when taking a pie out of the oven, for example. I would call this more a trial period or verification window or something?

bauble• 8 months ago

You're correct; it's a misuse of the term. Better terms would be embargo, quarantine, or just delay.

333c• 8 months ago

The dependency has just come out of the oven, so it's too hot and must cool down ;)

jrowen2• 8 months ago

I addressed this though lol. Nobody says "my pie is on cooldown."

largbae• 8 months ago

Wouldn't this be a great use case of any agentic AI coding solution? A background agent scanning code/repo/etc and making suggestions both for and against dependency updates?

Copilot seems well placed with its GitHub integration here, it could review dependency suggestions, CVEs, etc and make pull requests.

andix3• 8 months ago

One reason for cooldowns is not mentioned: maintainers often notice by themselves they got compromised.

The attacker will try to figure out when they are the least available: during national holidays, when they sleep, during conferences they attend, when they are on sick leave, personal time off, ...

Many projects have only a few or even only a single person, that's going to notice. They are often from the same country (time zone) or even work in the same company (they might all attend the same conference or company retreat weekend).

kazinator• 8 months ago

Cooldowns won't do anything if it takes wide deployment in order for the problem to be discovered.

If the code just sits there for a week without anyone looking at it, and is then considered cooled down just due to the passage of time, then the cool down hasn't done anything beneficial.

A form of cooldown that could would in terms of mitigating problems would be a gradual rollout. The idea is that the published change is somehow not visible to all downstreams at the same time.

Every downstream consumer declares a delay factor. If your delay factor is 15 days, then you see all new change publications 15 days later. If your delay factor is 0 days, you see everything as it is published, immediately. Risk-averse organizations configure longer delay factors.

This works because the risk-takers get hit with the problem, which then becomes known, protecting the risk-averse from being affected. Bad updates are scrubbed from the queue so those who have not yet received them due to their delay factor wlll not see those updates.

alphazard2• 8 months ago

For some reason everyone wants to talk about all the solutions to supply chain attacks except designing languages to avoid them in the first place.

Austral[0] gets this right. I'm not a user, just memeing a good idea when I see it.

Most languages could be changed to be similarly secure. No global mutable state, no system calls without capabilities, no manual crafting of pointers. All the capabilities come as tokens or objects passed in to the main, and they can be given out down the call tree as needed. It is such an easy thing to do at the language level, and it doesn't require any new syntax, just a new parameter in main, and the removal of a few bad ideas.

[0] https://austral-lang.org/

nextaccountic• 8 months ago

How does Austral avoids supply chain attacks?

alphazard2• 8 months ago

The typical attack tries to get the victim to load a module which accesses information on the filesystem or in another module's memory, then exfiltrate that information using network access. Other attacks may install backdoors used for infiltration once the service is deployed.

If the module provides some small helper function like `appendSmileEmoji(x: String) -> String` then it shouldn't be accessing the network or the filesystem, and the only memory it should have access to is the string passed into the function. The programmer already provided enough information to explain to the compiler that the function should not do these things but poor language design allows them to happen anyways.

Components which do have access to the network should be vetted thoroughly because they can facilitate infiltration and exfiltration. Most supply chain attacks aren't introduced directly in a high profile project like a web framework, they are introduced deeper in the dependency chain in the long tail of transitively imported modules.

In Austral (or any language using these good ideas) a small poorly vetted function like `appendSmileEmoji` wouldn't be called with access to the filesystem or network. Without access to any resources, the worst it can do is append a frowning face or a null character or something annoying. You can tell by its function signature above, it doesn't take a capability for the network or filesystem, just a String. So trying to ship a change that secretly accesses the filesystem would produce a "variable not found" error. To even get the new version to compile, you would have to add an argument to take a filesystem capability. `appendSmileEmoji(x: String, fs: FS) -> String`. That breaks any dependent modules because they have the wrong number of arguments, and looks very suspicious.

rmunn2• 8 months ago

One guy I vaguely know (only know him online, never met him IRL) often says "If you're using NPM you deserve what's going to happen to you one of these days." (He means the package ecosystem, so he would say the same thing about PNPM). I don't agree with him — he goes way too far with his opinion — but I can't deny that he's at least partly right. Two major supply-chain attacks in the past month (Shai-Hulud, and that tea-farming attack earlier this month which AFAIK doesn't yet have a clever name) have served to demonstrate that he definitely has a point in there, even if he takes it too far. NPM has a long way to go in terms of hardening (they've started doing so, thankfully), and is a very popular target.

kachapopopow• 8 months ago

pick your poison:

- you are vulnerable for 7 days because of a now public update

- you are vulnerable for x (hours/days) because of a supply chain attack

I think the answer is rather simple: subscribe to a vulnerability feed, evaluate & update. The amount of times automatic updates are necessary is near zero as someone who has ran libraries that are at times 5 to 6 years out of date exposed to the internet without a single event of compromise and it's not like these were random services, they were viewed by hundreds of thousands of unique addresses. There was only 3 times in the last 4 years where I had to perform updates due to a publically exposed service where these vulnerabilities affected me.

Okay, the never being compromised part is a lie because of php, it's always PHP (monero-miner I am sure everyone is familiar with). The solution for that was to stop using PHP and assosiated software.

Another one I had problems with was CveLab (GitLab if you couldn't tell), there has been so many critical updates pointing to highly exploitable CVE's that I had decided to simply migrate off it.

In conclusion avoiding bad software is just as important as updates from my experience lowering the need for quick and automated actions.

torginus• 8 months ago

I just had a thought about dependency chain attacks - has there been any example where instead of overtly compromising the code, they injected an exploitable bug?

Though I guess it'd be hard to prove intent in this case.

Jweb_Guru• 8 months ago

These are just backdoors as a vulnerability class. It does happen occasionally even in high quality widely used libraries, but it's very rare and usually quite sophisticated (e.g. https://en.wikipedia.org/wiki/XZ_Utils_backdoor in 2024). A lot of these attacks, even when highly sophisticated, only really work when the attack payload can be hidden in a binary--it's very very hard to insert a useful backdoor into the source code of an actively maintained project that receives regular code reviews without anyone noticing (which is not to say it's impossible).

chuckadams2• 8 months ago

It's also worth noting that the xz exploit didn't even live in the xz source, it was added to the source by the attacker before uploading it to the package build farms. With most of the current package repositories, there's not even an expectation that the bundles you get are derived from a particular tag or commit hash, let alone a detailed chain of custody. I remember RPM in the old days made a big deal out of "pristine sources" to which patches would be applied, but it still has no way to prove or enforce that claim.

nrhrjrjrjtntbt2• 8 months ago

Didnt know this had a name i.e. cooldowns but I totally agree. You want tooling to generate upgrade PRs but on a safe cadence. For urgent patches, i.e. zero days, a security team (internal or external can judge)

weinzierl2• 8 months ago

The article assumes that the duration of the attack window is fixed and independent of the cooldown period. It's not. Once everyone waits to update the time until the vulnerability is found increases and the attack window will grow.

presentation2• 8 months ago

What if cooldowns were implemented by a package manager somewhat randomized, so that it’s more of a gradual rollout instead of a fixed cooldown period?

xmddmx• 8 months ago

Why not do an empirical A/B test: Set up two honeypots (or perhaps 2000 for statistical significance). A gets zero updates, B gets all updates immediately. See which ones get pwned faster.

presentation2• 8 months ago

If everyone uses cooldowns would the discovery time potentially get pushed back?

pigbearpig• 8 months ago

Probably, because researchers/vendors/maintainers aren't going to catch everything, but you have less exposure too.

yyyk• 8 months ago

This assumes the direct mode of exploitation. The indirect mode (where an attacker inserts a vulnerability) does not necessarily have the 'short time span' issue. So not a complete solution by any means.

Also, there should a way to distinguish between security updates and normal updates for this. If there is, a cooldown is a useful idea in general for normal updates, since (presumably) the current version works and the new version may introduce bugs.

acdha2• 8 months ago

This assumes the direct mode of exploitation. The indirect mode (where an attacker inserts a vulnerability) does not necessarily have the 'short time span' issue. So not a complete solution by any means.

The short time span isn’t just because exploits get attention: it’s to allow the groups which do automated analysis time to respond. Significantly increasing the challenge level for an attacker to introduce a vulnerability is a meaningful improvement even if it doesn’t prevent that class of attack entirely.

bandrami• 8 months ago

I always remind my team that users of Debian Stable missed heartbleed entirely because their OpenSSL was too old. A lot of people and tools have the attitude that regressions never happen.

joshdavham• 8 months ago

Also interesting to note that the author is involved with the uv python package manager, homebrew and is a member of pypa (Python Packaging Authority). Very impressive!

ivh• 8 months ago

He mentions uv in a footnote. I will try to remember to add --exclude-newer $(date -I -d '2 weeks ago') the next time I use uv lock --upgrade.

illusive4080• 8 months ago

The Noir iPhone extension won’t darken this white page because the author has explicitly told Noir to not darken the page. Why not just make a dark mode?

unsungNovelty• 8 months ago

I always liked semver. Never have I liked just churning version numbers. It tells you exactly what a release is. Dont even need to look at release notes (which some doesnt have).

It helps a lot with these dev cycles.

Recently, I saw even EmberJS moved away from it. I still think they could've done it without leaving semver. Maybe I'm wrong.

josephcsible• 8 months ago

Isn't this one of those things that can only work if most people don't do it? If everyone did it, wouldn't something like https://xkcd.com/989/ result?

OakNinja• 8 months ago

On dependencies, a friend of mine once said ”The early bird might get the worm but its mouse number two that get’s the cheese”.

m0002• 8 months ago

Isn't the title kind of self-conflicting? Dependency cooldowns work because not everybody is using them. The more people start using dependency cooldowns, the less effective they become.

macintux• 8 months ago

There are comments elsewhere in the discussion indicating that these attacks tend to be discovered by automated scanners and the maintainers themselves, not by customers negatively impacted.

teaganga• 8 months ago

i had the same thing in mind, though, is not that part of the process until the release of available? should not the release already have the maintainers input?

jappgar• 8 months ago

If we all did it, it wouldn't work.

Quizzical4230• 8 months ago

Could you please elaborate on this?

alkallio• 8 months ago

Because nobody would test the updates.

morshu90013• 8 months ago

Heh, my stuff is always pinned to old versions just cause I don't want to deal with breakages, so... way ahead of you. Or behind.

Also whoever keeps changing the entire API of node-redis, please stop.

snowwrestler• 8 months ago

When I admin’ed a bunch of Wordpress and Drupal websites (open source CMSes), I was very quick to install security patches, and relatively slow to install normal updates—for this reason.

It’s one of the main reasons I used popular open source software: so I could ride the coattails of the rest of the community. Basically everyone else could be my beta tester.

A regular update was an input to the community security practice, so I would let it settle for a while. A security patch was an output of the community security practice, so I would install ASAP, even if it meant breaking a feature temporarily.

I also manually managed dependencies as commits to the main codebase, meaning my entire site was one deployable object from a single Git repo. The “modern” practice today seems to instead favor a minimal repo and resolving and pulling dependencies at deploy time. Personally I think this is a bad idea that has amplified the risk of supply chain attacks.

And now the idea is apparently back to: give it a little while. Tell the automatic dependency puller to chill out and wait.

cryptonector• 8 months ago

If every dependent implements a dependency cooldown, then the cooldown might as well not exist.

jbkgujklgui3• 8 months ago

Also protects against when a library developer suddenly goes rogue and enshittifies the library, to make a buck or two.

coastalpuma• 8 months ago

I really wish homebrew supported this feature. There are a bunch of terminal tools I have installed through it (and wouldn't make sense to sandbox) that are basically one-person projects, and I imagine subject to supply chain compromises. I don't urgently need the very latest versions for any of them, as they are unlikely to have exploitable security vulnerabilities on my machine (due to not being involved with any network facing service).

Seattle3503• 8 months ago

vendors are still incentivized to discover and report attacks quickly, but are not as incentivized to emit volumes of blogspam about “critical” attacks on largely underfunded open source ecosystems.

How?

TZubiri• 8 months ago

Nice. Additionally we should be using dependency limits and dependency moderation.

Half of my job when reviewing work is telling junior devs to try to do something without the dependency. Usually they just learn about a very basic skill.

"Ok so you downloaded a thing called apscheduler, what does that do?" "So it's like cron?" "Can you use threads, or even better just use a separate process?" "Cool"

My (least) favourite example is the amount of juniors that download a dependency from a random indian dev to store and read credentials from .env . Just open() and read() the security critical file my dude.

ElevenLathe• 8 months ago

Maybe the repositories should do this by default?

mattwilsonn888• 8 months ago

The issue with this model in the most general sense is that it is zero-sum, and at the limit it doesn't provide hardly any security.

I delay the use of updated software by a week, and anyone that doesn't takes the risk. Therefore I, the user of the cooldown, enjoys reduced risk at the expense of everyone not implementing a cooldown.

If everyone simply delays their updates, then there is nobody to suffer an attack which notifies users of the cooldown (in this case, everybody).

The blog post makes the argument that the vendors are incentivized to discover these attacks in this time, but that's an entirely different argument and if that were true, they would already be doing that.

In fact, auditing updates for vulnerabilities is the general solution. The whole appeal of the cooldowns is that you don't have to do that - the cost is that it's a zero-sum game reliant on the suffering of those less wise.

time4tea• 8 months ago

Been saying this for years!

All the security theatre!

False urgency is red flag when it comes to emails and sms.. I dont see why people don't see the same when it comes to dependency alerts.

The vast vast majority of alerts are totally useless, when it comes to your actual deployed system, unexploitable and irrelevant.

Rushing to update causes make-work, churn and loss of focus on the real.

Don't get me wrong, I love secure systems, but dependency theatre doesn't make it.

Very very occasionally, you'll get a high priority signal, and then yes act on it immediately. Your deployment systems should be designed so that this is just another release. No expedited short cuts, just another day.

Reducing dependencies, defense in depth, validation, type-safe, null-safe languages etc all good.. endless dependabot PRs with no understanding of the reason or impact, bad.

We should all be using dependency cooldowns

Related Stories