Why has software supply chain security exploded?

https://opensourcesecurity.io/2022/09/06/why-has-software-supply-chain-security-exploded/

by pabs3 • 4 years ago

71 85 4 years ago

gbrindisi• 4 years ago

log4j introduced supply chain security to the wider corporate world where most production run on Java and a lot more people finally realized how complicated it is to do incident response for these kind of situations.

rcarmo• 4 years ago

i have a personal take on this, which is that people have yet to realize that pulling new shiny dependencies off GitHub (or entire unaudited container images off Docker Hub) and putting them directly into production is not necessarily a good idea…

spyremeown• 4 years ago

I mean, it's a speed v. security thing. Every engineer (and I don't mean software engineers here) will tell you this is insane. You can't trust your bricks and yet, you're building a cathedral.

I know that automating stuff to check for CVEs and vulnerabilities of sorts is getting a lot of traction recently, but I don't think this is really sufficient. Who do you trust? Are we going to see some companies creating a "Security Seal of Approval" for packages?

I'm not sure what's the answer here, but certainly it isn't what we have been doing for the last couple years.

qikInNdOutReply2• 4 years ago

You can trust those who depend upon a lifelihood from you.

You can trust who you pay properly.

Who entrust the survival of theire family to your continued existence.

Whos bread i take, whos song i sing.

Thats the supply chain.

Turns out there is no free lunch to be had even in open source.

freeqaz3• 4 years ago

There are a few companies in this space that are trying to do the "Security Seal of Approval" thing to various degrees.

Tidelift is one company that has a bunch of "catalogs"[0] of packages. I'm not sure how their package metadata is generated though -- maybe semi-manually?

There is also Bytesafe[1] which is supposed to help give you a way to "firewall" yourself from unapproved dependencies. I don't think they sell data though. Just tools.

A company based on the Open Source project, packj[2], is Ossilate[3] which is trying to use automated analysis for analyzing 3rd party packages.

And another that's in this space is Socket.dev[4] which tries to warn you in your PRs about bad packages.

I know about this space because I work on a project[5] that's also related to supply chain security. It's a bit different from all of the above since we're focused on patching known vulns, but the idea of "vetted packages" has crossed my mind before.

Are there any other services in this space that I missed?

0: https://tidelift.com/solutions/catalogs

1: https://bytesafe.dev/

2: https://github.com/ossillate-inc/packj

3: https://ossillate.com/

4: https://socket.dev/

5: https://github.com/lunasec-io/lunasec/

selfmodruntime• 4 years ago

This is only insane in the engineering world because actual human lives are depending on the structural integrity of your cathedral.

Even if you face a supply chain attack, the only value lost may be money for most businesses, and not enough that worrying about supply chain attacks is worth it.

ciupicri4• 4 years ago

Isn't this what Linux distributions like Red Hat Enterprise Linux are doing? Providing a secure (among other things) platform on which you can build on?

staticassertion2• 4 years ago

At least github supports 2FA, signing of commits, SSO, etc. It can be a lot more trustworthy than a package distro.

have_faith• 4 years ago

Or people do realise the risk and are balancing the risk of speed of dev with the likelihood of something going wrong (and them being blamed for it). Imperfect systems often produce a lot of value, even if it means playing a game of spinning plates to keep it going. It's very possible that the value produced by this system overshadows the lost value from an ocassional black swan event as a result of the systems failure modes.

mnd999• 4 years ago

Security snake oil merchants figured out a new way to make a quick buck.

qikInNdOutReply2• 4 years ago

I run around the office with scissors air gapping your cables, pay me or i shall destroy productivity to the point security is the least of your concerns. Once i get your employees to work around me, violating contract, your problems are "over".

rswail• 4 years ago

Of course they will, but the actual problem is there.

I've been delivering systems where we had to deliver the source environment (ie what is used to build production) into escrow with a lawyer as part of the contract. It effectively required us to prove that what we delivered (both hardware and software) into escrow, would be able to reproduce what was in production, reliably and provably.

Reproducible builds are really hard, especially with people encoding CVS/SVN/git tags into the build, compilers and linkers not producing equivalent results between runs (eg caches warm/cold can make a difference) etc etc.

It's a real problem and a real issue.

Rygian• 4 years ago

There was never a supply "chain" to begin with.

If you're looking for a more precise metaphor, think of it as a large pot of slightly overcooked spaghetti and meatballs.

Where anyone can throw stuff into the pot.

ben_w• 4 years ago

At the top level, that sounds apt.

But people have automated the throwing in of stuff, and even when the original stuff was excellent, stuff sources are sometimes bought out by bad actors.

And sometimes the stuff has ingredients, where the ingredients are also so managed, so I would say that there is still a chain.

yosito• 4 years ago

I think you're oversimplifying spaghetti and meatballs. There are supply chains that go into that as well. Nothing in the modern world is built from raw materials. In theory, if it weren't for supply chain security, someone could poison you in your kitchen by going to a chicken farm, feeding toxins to chickens, who lay contaminated eggs which are sold to a factory that makes pasta, which is sold to a grocery store where your coworker bought it to make spaghetti and meatballs that he brought to your potluck. Even I'm over simplifying it. The point is, without supply chain security, anyone already can throw stuff into any pot.

_int3_2• 4 years ago

Business depends on trust, there is risk involved in doing business. There is always chance somebody will either not pay or not deliver, or deliver lower quality products than ordered (like this is the case with supply chain attack). And there is not much you can do about it. Later you can sue them or not do business with them anymore, but you don't know that in advance.

perlgeek• 4 years ago

I really wonder why supply chain security wasn't such a big issue before, and why there weren't really many supply-chain attacks on software before.

Too few attackers understood it well enough? Or were other attack vectors easier?

_int3_2• 4 years ago

Programming style has changed over time to include more dependencies from 3rd party. Look at style of programming from 70s, they would do everything from OS , filesystem to application. Today there are many dependencies.

EDIT: It's much like building a tower. Upper floors depends on lower floors. Taller the construction more unstable it gets.

whatever1• 4 years ago

Because C does not have package managers. We were copy pasting libraries as .h files in our projects and never updating them.

taneq3• 4 years ago

I still find it mind boggling that businesses have build processes for their core properties that rely on pulling hugs swathes off third party code in from the internet. How can that ever possibly be secure and robust?

DrBazza4• 4 years ago

Same with C++. Dependencies were boost, qt, gtk, and a few others and that was pretty much it. Since vcpkg, cmake, conan, and a few others, the C++ "ecosystem" has started growing far more in the last 10 years, than in the previous 20.

Ygg2• 4 years ago

Looking at https://wiki.alopex.li/LetsBeRealAboutDependencies

Perhaps the reason is package managers for C are called distro maintainers (for Linux/BSD)

staticassertion2• 4 years ago

Attackers do what makes money. A lot of that is market driven - you buy what exploits are available, what tools are available, etc. You monetize in a fairly standard way. Attackers aren't all just random people thinking "oh I could hack that" - that's a tiny minority of them.

Eventually, due to a lot of factors, threat landscapes start shifting and attackers start moving their targets and tooling. One of the bigger shifts would be the huge, rapid improvement to both OS and browser security that occurred within a few years, radically increasing the cost of attacking desktop users via malicious websites. Another would be crypto, where 'account takeover' attacks that could lead to wallet access are now easier to monetize + crypto itself as a tool for transfers.

With regards to supply chain, enough of these changes occurred that some attackers took the leap and have started looking at this area. There are probably a lot of reasons why - prevalence of dependencies, increased interest in tech companies, etc.

If it continues to prove viable (it's obviously viable from an attack perspective, unclear if it's something attackers will rally around to monetize) we'll see it escalate and get better tooling around the attacks.

Jach• 4 years ago

There's a copycat effect at play too. Once something becomes well-known enough, it enters as an idea into more people's minds that it's a possible thing, and some will attempt it. Before it was smashing the stack, now it's supply chains etc. As for what you can do with an attack vector to motivate beyond a simple satisfaction in breaking something, an obvious popular choice now is ransomware. Ransomware attacks would still continue even if the world somehow got rid of cryptocurrencies, they would just use less convenient but more traditional forms of payment. Because ransomware became popular.

Melatonic• 4 years ago

Probably because the CIA was doing tons of it and did not want to tell people :-D

diamondo25• 4 years ago

How about all the throubles with hijacked NPM packages? These attacks are still "small", but have had big effects, multiple times.

bobsmooth• 4 years ago

Because it was bound to happen eventually.

chalst• 4 years ago

The hardwiring of dependencies in a Nixpkg-based deployment seems to be good for generating SBOMs.

Cf. https://discourse.nixos.org/t/generating-software-bill-of-ma...

ciupicri4• 4 years ago

Assuming you're using only Nixpkg and not doing pip install, npm install, cargo install and so on like many people suggest and do.

ThePadawan• 4 years ago

Vendors had to start checking the "ran supply chain audit" box since investors demand they do.

Boom, new industry of snake oil.

Not suggesting that the investors have stakes in any companies doing supply chain audits or building automated solutions for it, of course.

hannob• 4 years ago

The answer is "too many dependencies".

Package manages made it easy to have dependencies. Every added dependency adds risk. That doesn't mean you should never add dependencies - there are good reasons to do so - but you should weight their pros and cons. Adding dozends, hundreds or thousands of dependencies (as it is common in the npm ecosystem) adds lots of risk. None of that is particularly surprising.

TAForObvReasons• 4 years ago

For-profit package managers like NPM touted download and package counts as if they were KPI. Deep dependency trees is the natural result.

freeqaz3• 4 years ago

Installing react-scripts is ~1400 dependencies and around 2 million "edges" (where one package requires another). This blew my mind recently when "yarn audit" was failing with an OOM error!

Jenk3• 4 years ago

That's ret-conning a correlation. Npm aren't _generating_ packages themselves. The community at large is generating them.

Nextgrid• 4 years ago

This is especially a problem in languages which have a chronic lack of decent standard library and everything relies on a third-party package.

DrBazza4• 4 years ago

I built zellij (rust tmux clone) recently. 193 dependencies that cargo downloaded. When did rust turn into javascript? Are there supply chain attacks in Rust?

sylvain_kerkour• 4 years ago

Yes :) https://kerkour.com/rust-crate-backdoor

ehnto• 4 years ago

I would wager that the PHP standard lib isn't all that comprehensive, but it doesn't suffer anywhere near as much from nesting dependencies. I think the culture is that a library takes care of itself for the most part.

bob10292• 4 years ago

This is the #1 reason we use .NET for our banking apps. Everything we need is baked into the Microsoft-owned dependencies. Certainly, you could argue of transitive attacks still being possible but from a liability perspective we get to point our finger at Microsoft if something goes wrong (while we hastily rebuild with their patch)

Even for my side project - building a custom 3D rendering engine from scratch - I use libraries like System.Numerics to calculate the camera perspective transform and perform all the linear algebra for me. I was wasting a lot of time trying to reinvent this, which is a typical but lethal risk to any project.

Jenk3• 4 years ago

As head of eng in a ftse100, this is so far from the truth. .Net is a heavily targeted ecosphere, possibly moreso than npm, precisely because of its relation with finance.

We have to contend with not only nuget and the sheer abundance of packages available (with their own transitive dependencies similar to the problems with npm - yes, even Microsoft packages have this risk) posing problems, but also the tooling itself.

VS/Code is a fucking minefield of potential threats. Their extensions market place(s) are automatic RCE risks, and a developer/admin/whomever box getting pwned is a monumental risk.

Zero-Trust applies to everyone and everything. Microsoft don't get a pass just for being Microsoft.

jimrandomh• 4 years ago

It used to be that adding library dependencies to a project had a lot of fraction, and if you were writing a library yourself, then imposing indirect dependencies on users was something you'd try fairly hard to avoid.

Then npm came, along with some ideology about code-reuse, and the friction went away. But the friction was serving an important function: if adding a library is annoying, you'll add a small number of large libraries rather than a large number of small ones, and you avoid getting an ecosystem where major libraries have hundreds of tiny dependencies. This is important because the friction of adding a dependency is only a small portion of its true cost: the main cost is that you're trusting too many different developers and developers' computers.

ElevenLathe• 4 years ago

This is a good framing of it.

Can we keep the benefits of automatic dependency resolution while adding some other, artificial deterrent? What if NPM hosted packages for free if they have less than five (or whatever number) of transitive dependencies, but above that charged some number per additional one? Even if it were a small amount, this might put significant downward pressure on the size of dependency trees.

egberts13• 4 years ago

It all started with glib/compilers/LLVM/Windows toolkit not being able to reproduce a build, byte-for-byte.

egberts13• 4 years ago

As a QA can see, the inability to procure a reproducible executable checksum values allowed all sorts of non-validated (and un-validatable) things to creep into production.

egberts13• 4 years ago

Of course, timestamp is the biggest offender of recreatable (deterministic) build.

https://blog.conan.io/2019/09/02/Deterministic-builds-with-C...

pabs32• 4 years ago

Indeed, tons of other issues prevent deterministic builds though:

https://reproducible-builds.org/docs/

ciupicri4• 4 years ago

What difference does it make if the build is compromised in an identical way byte-for-byte?

acuozzo2• 4 years ago

https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

pabs32• 4 years ago

These folks are working on fixing that problem:

https://reproducible-builds.org/ https://bootstrappable.org/

dusted• 4 years ago

npm install

DrBazza4• 4 years ago

At a previous job we were building something significant in the Java stack, and as part of the CI/CD we have a jar checking stage which flagged that an emoji library, that wasn't an emoji library, suddenly was coming in to our build transitively. Presumably somewhere down the line, someone wasn't using a fixed version in gradle or maven.

ThePhysicist• 4 years ago

These days it's always hard to say how much of the hype in areas like supply chain security is driven by the problem actually becoming worse and how much is due to VC funded startups with enormous marketing budgets that try to convince everyone of the nearing apocalypse (but no worries, their tools can save us).

Looking at integrated vulnerability reporting as found e.g. in npm (and soon Golang, I think) it seems that 99 % of alerts are just false positives, as they affect build dependencies (e.g. the tar library) somewhere down the chain that will never run in production or even touch anything sensitive, and that cannot be easily exploited even by sophisticated actors. In my opinion the "high severity" threat category is thrown around way too lightly by these tools, which leads to everyone panicking.

If anything good could come out of this whole trend it might be the realization that open-source software needs better funding to address supply chain issue, my expectation though is that most of the money will go to tool vendors that simple tape over the underlying problems by creating an artificial "security layer" between the maintainers and users.

UncleMeat• 4 years ago

The CVE landscape has been broken for ages. In a past life I did a lot of work on a bug bounty program and the amount of "I ran some simple tool that says CVE-XYZ is present in your codebase" reporting we received when the relevant code was either dead, ran in a trusted environment, or ran on trusted input was outrageous. Then people would whine about not getting paid.

I do think that organizations should have a process for identifying and triaging bugs in dependencies. But "there are vulns in dependencies" is not itself a compelling narrative because there are vulns in your business' code too. And people sure as hell are getting updates more frequently when using npm or whatever then in the alternative world of "link in some open source code and never update ever for all time."

To me, the more interesting problem is the "malicious actor takes over an npm account and distributes malware" that is more unique to the npm ecosystem. This one is hard. On the one hand developers are expensive and a robust library ecosystem is basically a superpower. On the other hand you really need a trust relationship with your software vendors if you want to be confident you are delivering safe code.

pornel3• 4 years ago

I think "npm bad" is a lazy take.

• There used to be a focus on attacking endpoints directly, but eventually OSes, browsers, and servers stopped being easy targets, so attackers looked for another way in.

• OS (especially Linux) distributors acted as gatekeepers. Getting into a distro requires having some reputation, and the risk of getting caught with a backdoor is a deterrent (although I don't think distros are as much of security barrier as people assume, since maintainers already have too much work to do and their source code review is at best at "LGTM" level).

• New tooling gave more visibility into dependencies. Transitive and build-time dependencies in precompiled libraries typically aren't visible, so it's easy to think you have one or two deps, but transitively there are still tons of them: https://wiki.alopex.li/LetsBeRealAboutDependencies

• Exploits became easier to monetize. Thanks to crypto miners and crypto wallets any exploit is almost a direct payout. Way easier and safer than trying to sell a botnet, adware or even stolen CC numbers, so it lowered barriers to entry to this "business" as well as made it more attractive.

wikfwikf2• 4 years ago

I agree with this.

The example of 1000s of packages having a transitive dep on a package which left-pads strings looks bad, however the two obvious alternatives to this are:

1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues.

2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.

galdosdi• 4 years ago

1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues.

You sure? You can write it, with no security issues, with acceptable performance for most cases, in the same time than it takes to find the dependency. In fact, you'll save time since you don't have to vet the dependency.

Uh, you do like, vet your dependencies right, whenever you add them, right? As well as every time you update to a new version, right?

If you're not doing any of that and just adding dependencies willy nilly then I could see how it could seem easier, but this is an illusion. You need a dependency to do more than just left padding to make it worth adding, unless you don't give a crap about stability or security.

2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.

Yeah this is how languages have traditionally done it, and when done well it is very nice. As you say, it is hard to change the contract because it gets depended on, and you know what? This is a feature, not a bug. Things that are really simple and settled like leftpad don't need to constantly change. This is also why emerging languages try to stay "alpha" without guarantees about breaking changes as long as possible early on, so by the time they have to really freeze the standard library, a lot of experience has helped polish it.

A similar model is something like Java, which has done a fantastic job of evolving its standard library over the years while staying very (but, mostly, not excessively... mostly...) backwards compatible. Mostly through the trick of letting users create their own popular libraries and then eventually integrating it with the JEP process once one gets really big. Kind of like a lot of the stuff from Guava, or Jodatime.

fomine3• 4 years ago

3. Have a package of common utility functions maintained by organization rather than an individual, like Apache Commons Lang.

aragilar2• 4 years ago

I'm not sure "new tooling" (if you're contrasting language-specific with linux distro tooling) does give more visibility though. How well does "new tooling" track root certificates being up-to-date and available for example (ca-certificates is explicitly mentioned in the vlc dependencies in the link as an example)? Or across language/ecosystem boundaries? It would be interesting to compare vlc dependencies on Windows, MacOS and one of the BSDs, as unlike linux distros, there is a explicit and somewhat complete core (which would likely remove or at least reduce the number of "system" dependencies on the vlc list).

I do think better tooling can exist (and is being developed), but it needs to track the whole system, not just a single language.

pornel3• 4 years ago

Yes, I mean something like Cargo vs apt.

If you depend on libcurl, you count that as 1 (one) dependency. Nice, and restrained.

But curl has everything from gopher, FTP to SMTP and IMAP! It has a half a dozen TLS libraries, SSH, auth methods, multiple compression algorithms, socks proxies, etc, It's a lot of code, and a lot of transitive dependencies that you don't even think about.

OTOH when you add `reqwest` in Cargo, Cargo will visibly fetch everything needed to build it from scratch, and it will look like 140 dependencies, not 1. But in total it's still only a subset of what libcurl contains. There's no Samba or POP3. It's just a very detailed list showing you it contains mundane things like a CLI arg parser and base64 encoder, and a wrapper for ca-certificates too.

Please don't interpret it as a criticism of curl. It's just to contrast that you may see libcurl as 1 dependency, because it's one precompiled package, but you may see a smaller amount of code as extravagantly bloated shocking 140 dependencies, because it's just been split into tiny pieces and rubbed in your face.

verisimilitudes• 4 years ago

I'll take the advice of this article lacking substance, and complain.

Well, companies used to have employees write code, rather than stitch together random garbage written by random dipshits who could be tricked into using loose licenses. That's one cause for concern.

Open source won because everyone worked together.

No, ``open source'' has ``won'' because it helped corporations defang Free Software and get gratis labor.

Supply chain security will happen because everyone works together. If you try to do this alone, you will fail.

This isn't food, but perfectly uniform applied mathematics. There's no ``working together'' with corporations. Doing it alone is the only reasonable option, which means aggressively reducing the size of this foul shit is necessary.

Thank goodness others had the foresight to create our current SBOM formats.

Yeah, MicroSoft totally *<3* ``open source'' guys.

Long ago all the software supply chain work was done by hand.

Oh, right, all of this new security theatre is always about trust and reputation, and not trusting those disgusting lone programmers such as me or other silly things; it's always really about doing anything but truly auditing that yucky code.

oxff• 4 years ago

There is more money to be earned in exploiting the supply chain.

m3047• 4 years ago

I quit a gig recently because the CI/CD pipeline had automatic scanning of dependencies for vulns (yay!) and no tangible process for dealing with them.

It was up to the dev to "fix" this "defect" before code could be merged.

There was no pinning policy. There was no actual security review (where do we use this? are we vulnerable? why do we use this? are there alternatives?).

Meanwhile it's running in production. (have we discussed this with our MSP? can I discuss this with our MSP? has our MSP notified us of any issues?)

Although we say ignorance is no excuse, having spent a decade in cybersecurity and 20 years before that as a quality-focused consultant, I'd say ignorance lets people get away with a lot. But I know better and I would never be allowed to claim "I vass only followeeeng zee orders!" if the shit hit the fan.

nokya• 4 years ago

I felt clickbaited after reading the article. It's interesting to read, but if you're actually looking for the answer to the question then it's not in the article.

mmaney13• 4 years ago

I think it has a lot to do with 2 things:

1. A conscious shift in focus from triaging risks when they occur to stopping threats before they arise.

This is a natural next step from contemporary security and disaster response. Threat response and continuity planning which both incorporate plans that respond to threats were once the primary objective of organizations. They are still valid, but a more modern and proactive approach includes mitigating the risk at the source.

2. A forced increase in material spend toward securing dev and devops ecosystems in a time where they are one of the most targeted parts of the organization.

One only has to watch the news to see this play out... unfortunately after decades of deployment and intranet security emphasis, hackers have recognized that IP and source code are the best way to get money out of a company, and that both are ironically some of the least protected assets.

Why has software supply chain security exploded?

Related Stories