We're Drowning

https://snarfed.org/2022-03-10_were-drowning-software-dependencies

by mgrayson • 4 years ago

187 157 4 years ago

lmm• 4 years ago

Funnily enough this becomes much less of a problem once you use a real type system instead of having to write manual tests. Both because whether your code compiles becomes a pretty good (not perfect, just as your test suite is not perfect) indication of whether this dependency upgrade is breaking for you or not, and because those ecosystems tend to have a lot less need for new versions to fix things or change behaviour in the first place.

gyulai6• 4 years ago

I don't think a type system will save you.

The problem is that, since the introduction of dependencies has become so easy, people have become dependency hogs.

Every time you introduce a dependency, you need to weigh the benefits against the cost. But to the person introducing a dependency, at that moment when they make that decision, the cost is just "pip install x". So they think the cost is negligible, but, of course, the true cost doesn't start to show until much later, when it's too late to undo the decision.

One slighly masochistic thing I've done for myself is that I don't allow myself to ever do "pip install x". I force myself to (1) download a source distribution for the package, or wheel if it's pure (2) cut the internet connection (3) do "pip install --no-deps" or "setup.py", building from source from the local package (4) see if it works, or if the absence of dependencies causes actual breakage (5) if there's breakage, go to 1, repeating the process with the missing dependency -- when I've spent more time on this process than I think the package is worth, given the benefit it brings to my project, then stop the process and try to live without the dependency.

Interestingly: Forcing yourself to build everything from source also tells you a lot about the state of health of a package you're about to introduce as a dependency. Like: Does it depend on something that's just a wrapper around a perl module that hasn't been touched in 15 years; that kind of thing.

Smaug123• 4 years ago

You may be duplicating a lot of work that's already been done in Nixpkgs, by the way.

reidrac• 4 years ago

I disagree. I work with Scala and I can see myself in that post 100%.

If things compile and your test pass you might be OK (sure), but the part where "those ecosystems tend to have a lot less need for new versions to fix things or change behaviour in the first place" is not there, unfortunately. If you are using Scala Steward, you'll have a good number of PRs almost every week.

drewcoo2• 4 years ago

instead of having to write manual tests

Write automated tests instead! Welcome to the 21st century, human.

keyle• 4 years ago

It seems there is one of those posts on HN every 6 weeks.

And yet there is no solutions besides yolo-live-with-it.

I thought it sucks too, I'm in the same boat, but I considered the opposite for a while: what if we didn't have all the code reuse and boatloads of libraries at our fingertips?

That would be a TON of code added to our applications and overall - shock full of bugs and lack of optimizations. These libraries, when popular are battle tested, cover much more grounds than we do today (but might need tomorrow) and have been looked at if slow.

This ecosystem relies mostly on everyone acting like adults; Libraries to respect semver and not making large, breaking API changes on minor versions; and an ecosystem that is "safe", where bad actors can't pretend to be someone else.

Ideally there would be an open "group" of people running penetration tests and security overview of most packages where the collective could donate against their work. Large donations would allow specifying which packages you'd want them to review, and fixes would flow money to the maintainers...

But I'm dreaming. We're all building on a gigantic pile of hacks written for someone's portfolio to get their next job, which won't be in the same language and the thing will become abandoned.

mellavora• 4 years ago

That would be a TON of code added to our applications

wait, doesn't using a library also add that code to your application?

And if the library has to cover more than your immediate needs, isn't is much much more code added?

mojzu2• 4 years ago

I think they mean that it's code they're not having to write/test/maintain themselves because presumably if you pick a good enough library the quality of code/features are better then what one individual or team could do

While there are issues with this approach like the article outlines, personally I believe it to be better then the alternative of countless developers reimplementing the same feature sets over and over because honestly that just seems like a waste of human time and talent

avidphantasm3• 4 years ago

It’s all about balance. Use libraries where it makes sense (e.g., if I’m reading HDF5 files, I’m going to use the HDF5 library, not roll my own), but don’t forget that you (ostensibly) know how to program on your own and like implementing some things yourself (rather than just lashing together other people’s code). Using a language with a decent standard library helps.

jerf• 4 years ago

"And yet there is no solutions besides yolo-live-with-it."

There is a solution: Stop accounting for dependencies as zero cost. Start accounting for their costs, in their need to be updated, understood, and possibly significant overcompelexity for your needs, plus all these costs are transitive into the dependencies they pull in.

This is not a magic solution. It doesn't mean you just think these magic thoughts and all your problems go away. But if you actually do this, it will solve your problems. It will turn the wide wonderful world of existing libraries back into a benefit instead of so often a net loss.

I still use a lot of libraries, but I've been noticing lately I use more "fundamental" libraries. I use a YAML/JSON parser for my config, but I don't worry so much about all the fancy wrappers around config that try to make it super nice to have environment variables or maybe a config file or maybe map it to the command line or maybe get it from the network... I just use a config file parser and commit to having that. My dependency trees are not completely pristine exactly, but they end up with a lot of these sorts of things in them but not so much all-singing, all-dancing "frameworks" or text templating libraries that are so amazingly helpful that they have entire embedded scripting languages or other such things.

pjmlp3• 4 years ago

A solution is vetting libraries, CI/CD only has access to internal repo, and libraries only get added to the internal repo after being validated.

This is part of my work for the last 10 years.

I only use whatever I feel like on personal projects and small prototypes.

irrational• 4 years ago

You’ve probably seen run-of-the-mill web applications with hundreds of direct dependencies

This is one of the reasons I like working on legacy web apps. Other than maybe jquery, they typically don’t have any dependencies. It is so easy to follow the code logic because it is all right there and nothing is hidden 10 layers deep in dependency hell.

Semaphor• 4 years ago

I work on a legacy webapp (created 1996, updated to a more modern codebase in 2001), we even have closed source dependencies.

throwawaaarrgh• 4 years ago

The solution is to do more with less. Devs are taking advantage of a wealth of abstraction and being buried under it. But they actually don't need it.

You do not need some 3rd party library to write a command line interface for your python code. You do not need three different projects just to maintain your list of installed packages or virtual environment. And you don't need a package to wrap around loading environment variables.

Just learn how to do these things in a simpler way, without the extra deps. Will there be some more boilerplate code? Yes. But it will be your code, and it will be so simple that you will learn to write it very well. In addition: make your code loosely coupled and composeable, and overload existing code with additional functions when feasible, rather than starting a new project from scratch. Press your company to let you contribute back to OSS, as this lets somebody else do more of the maintenance, and has the side effect of reducing the total number of software projects. (It immediately saves you money and time. Kind of stupid that more of us don't do this already.)

Simplifying is hard. But when you get good at it, you find you're much more buoyant.

drewcoo2• 4 years ago

You do not need some 3rd party library to write a command line interface for your python code.
Just learn how to do these things in a simpler way, without the extra deps.

That's known as reinventing the wheel. Commonly espoused by people with NIH syndrome.

https://www.techopedia.com/definition/3848/not-invented-here...

gyulai6• 4 years ago

Imagine some point in history where the wheel hasn't yet been invented. But someone figured out that an octagon works kind of reasonably well as something that you might want to put under your chariot, and the whole world of engineering is under a culture where "reinventing the octagon" is some kind of cardinal sin. Where would we be then?

Also: Your reasoning is logically a fallacy. From "people making mistake X tend to do Y" it does not follow that "doing Y is a mistake".

shafyy2• 4 years ago

There's a fine line between NIH (not invented here) syndrom and overusing third party libraries. The classic example being the is-even[0] npm library, which seems like a joke, but has ~200k weekly downloads. It just seems that many devs too quickly reach for a third party library before taking a few minutes to think if it would make more sense or not to write the code themselves.

0: https://www.npmjs.com/package/is-even

frou_dh• 4 years ago

Does anyone here actually use Python? Because "not [using] some 3rd party library to write a command line interface for your python code" should mean: Use the argparse module that's in the Standard Library. Not coming up with something homegrown.

zppln• 4 years ago

In Python you can insist on pulling in Click... or just use argparse which is available in the standard library.

ekianjo• 4 years ago

I assume you missed the fact that you can already do a lot with the standard library in Python for example.

gyulai6• 4 years ago

You do not need some 3rd party library to write a command line interface for your python code. You do not need three different projects just to maintain your list of installed packages or virtual environment. And you don't need a package to wrap around loading environment variables.

Couldn't agree more.

I'm going through the experience now of switching from Python to Lua and this experience of trying to do more with less is really eye-opening. I find that lots of things where Python has trained me to expect them to be one-liners are now idioms of half a dozen lines of code. ...but as I use them, I find myself introducing a lot of variation around each half-dozen-loc-idiom. Then the thought occurs to me: Should I make this into a utility function? And the answer is usually no: Because it would have to support all this variation and create a complex interface. Having to understand the interface would introduce as much complexity for the person trying to read/maintain my code as having to tackle half-dozen-loc-idioms where there could be one-liners.

Languages that default to simplicity and make-your-own-batteries force that thought process on developers, making them better developers in the process, able to produce more maintainable code.

Languages that default to batteries-included try to spare developers the work of having to think about such things, and the end result is way more messy.

moviewatcher333• 4 years ago

My favorite aspect of lua is that it gives you all the tools to implement anything and do it simply. Python, C#, and many languages push you to find the “right” way to do things, and spend time searching through documentation to find the shortcomings of each option and which method may be optimal for your situation. It’s like walking into a hardware store with a specialized tool for everything—you spend your time worrying if you’re really using the right thing. Lua feels like a garage workshop with all the basic tools you need to quickly implement something on your own—master a few and only go hunting for something new when you’ve got no other choice.

jay_kyburz• 4 years ago

I've also fled to Lua. LuaJIT is "finished". Love2D has a few dependencies but they are big and slow. Love does releases only every couple of years. I occasionally grab some code from the internet, but I vendor it in, and now its my code.

bsenftner• 4 years ago

You just described why I preferred Assembly early in my career, and C/C++ for the majority. Making your own batteries enables exponential opportunities to improve and optimize a project, simply because one knows all the parts, they are not externally sourced "batteries-included" features.

nequo• 4 years ago

Is the natural conclusion of this argument that we should be writing assembly because that has the fewest batteries included? Or is there a counterveiling force?

tonto2• 4 years ago

overload existing code with additional functions when feasible, rather than starting a new project from scratch.

why? are you making a jab at small packages? ill just rant on that and say i think small packages are great. they are extremely purpose built and lovely to use. they do become transitive dependencies often leading to the "issue" in this blogpost but im in the yolo upgrade camp

avidphantasm3• 4 years ago

The churn required to keep up with frequent updates to many dependencies can work for some projects, but I would argue only for projects with continuous funding. If a project is only funded every couple of years and has many dependencies, odds are, the next time it is funded, there will be an inordinate amount of work to patch/update the code. For such projects, fewer or no dependencies may be better.

oweiler• 4 years ago

And because every dev has a different idea about how to simplify things, you throw any sort of standardization out of the window.

gyulai6• 4 years ago

Standardization and pluralism both have their respective merits. It's a "cathedral vs bazaar" thing or corporate culture vs hacker culture.

mysterydip• 4 years ago

As opposed to the standardization of a random collection of github libraries?

amelius2• 4 years ago

But it will be your code

And your colleagues will have great difficulty figuring out how it works because there is no documentation and no community support.

fileeditview• 4 years ago

Which is true for most libraries pulled from Github also.. but additionally you cannot just ask your colleague.

avidphantasm3• 4 years ago

Then you flog mercilessly people in your organization who write code like this. When I was coming up, the greybeards were bastards, but they taught me the right way to go about things.

randomdata3• 4 years ago

Code is only hard to figure out if you don't understand the business goals. Third-party libraries/frameworks will almost never explain the business to you.

However, there is a practice known as testing that is supposed to document the business goals so that your colleagues aren't left in the dark. Hopefully you haven't skimped there, regardless of where the functional pieces came from.

Test0129• 4 years ago

I've rarely had a hard time understanding code that was written for a purpose rather than importing a library. Except in rare cases (like dates, big APIs) a library is often unnecessary cruft. Should I go pull a fat logger when all I need to do is send data over TCP to rsyslog? As long as I consider all of my use cases, and properly defend the cases I wont use I can write faster, often times more understandable, code. An example of a great, but often unnecessary library, is Pandas. I've seen Pandas imported for trivial CSV work, simple data manipulation, etc just because it was "familiar" rather than the right tool. Pandas is so, so, so heavy that if you're not using it completely for data science oriented work you are probably wasting your time...people seem to not understand this distinction.

I've had people cut juniors/data scientists/etc loose on a "be-all-end-all" framework and ended up with the worst code I've ever seen. Edges so sharp just looking at them will cut you. I've seen code that imports trivial things like line counting! I've even been on the response team for a security incident involving several internal packages that imported the wrong thing from the package managers! I've had to re-write important library functions for a package that was imported and deprecated, and the author just went to do something else. Libraries are great, but they are not a fire-and-forget tool. When you use them, you must use them sparingly, correctly, and securely. Importantly, you need to pray a library will be maintained over the life of your software.

"Documentation and community support" is worth absolutely, positively, nothing if the usage of the library enabled the writing of poor code. Hire better developers, and create strong coding standards at your company. Let software engineers do what they do best and choose the right place to include a library after evaluating all options. Experience level matters. Late seniors and staff+ understand the difference. New seniors, TLs, and of course braindead management love to bring in the universe on every project.

annowiki5• 4 years ago

I come from a Python background so libraries were always just a pip install away.

In the past year, however, I've been working on a large C++ codebase (couple million lines) and the result has been a considerably greater amount of "roll your own."

This has filtered back into my Python. If it's not in the standard library, I don't install it unless I really need it.

My boss always tells me that you can probably write it faster than a library, something I never used to believe. Until I tried it. I needed to check which of two version strings, of XX.XX.XX.XXXX format, were bigger. I tried the most recommended version number library, then I tried writing my own solution, the simplest solution I could think of:

    def version_compare(v1: str, v2: str) -> int:

        """
        Compares two versions.

        Parameters
        ----------
        v1: str
        v2: str

        Returns
        -------
        int
            1 if v1 is greater, -1 if v2 is greater, 0 if equal.
        """
        for el1, el2 in zip(v1.split('.'), v2.split('.')):
            if el1 != el2:
                return 1 if int(el1) > int(el2) else -1
        return 0

My code was faster by like 20x. Libraries are bloated and you probably only need a small subset of the functionality, so write your own code has become my mantra.

thyrsus• 4 years ago

TIL python type hints: https://docs.python.org/3/library/typing.html

Thanks.

pc86• 4 years ago

Another benefit, that IMO is as big if not bigger, is that you can easily make changes to this code whereas changes on the library side may be hidden behind configuration, impossible at all, or god forbid even require a wrapper anyway. In the above example, if you wanted this instead to return the version number string that is larger, that's an easy change and one that's very obvious from the PR what it is doing.

bruce343434• 4 years ago

Little bit off topic perhaps but why encode the types a second time in the comment when you already have the annotations?

tharkun__• 4 years ago

Now think about how (some) of these libraries came to be:

Someone like you wrote something like the above for their needs. Then they needed it again in another program. Then in another. They figured, why not put this into a library I can use across all my projects. Maybe this was internal only. Maybe at some point they thought: why not push this out there to github and upload it to a public repo and let other people use it.

Then someone (maybe the same person) needs not just XX.XX.XX.XXXX with actual numbers. For whatever reason their standard is vXX.XX.XX.XXXX[-RCX] or other variations that aren't just a simple number format.

Of course if you never ever have a need for that kind of version number (or whatever other "problem area" we apply this principle to) and only ever write one piece of software, you're fine with roll your own. But this is (one way) how libraries get "bloated". The fact that they are libraries (or frameworks if we think a little wider) need to take care of "all the problems" and not just a very specific one. Especially if it's open source and not an in-house only library which attracts lots of people with lots of different variations on the same problem. It's very hard to have a very opinionated stance of "this is what you get, change your version number standards, I'm not changing my library". Of course you can do that, but your library will probably end up in obscurity. The "can deal with anything you throw at us" is probably going to be more popular and if there's no Linus type guy doing code reviews before accepting contributions, you end up with sub-optimal code very quickly.

I've done the opposite before. I came to a code base that had various variations on sending email notifications for batch jobs. Like literally 20 jobs with 20 variations (well maybe it was 15 different ones with different types of bugs and quirks). I extracted them into an in-house library for sending a standardized version of these emails. In the end it had accumulated quite a few features for adjusting the email template, attaching various formats of output, automatically zipping it up etc. I'm pretty sure you might call it "bloated". In reality it made things better (consistent) for the users receiving those, easier for us to understand errors, faster to code new jobs and just calling one standard function that everyone knew how to use and that all bugs had been ironed out from over time. Import library, call function. Instead of deciding which other project to copy the email sending code from, finding out where in the mess of code it was, throwing it away again because "Oh yeah right, that project's version of it can't deal with this type of attachment etc.

remram2• 4 years ago

This returns 0 for '1.2.3' and '1.2'. Might be fine if you are sure they have the same number of components, but otherwise is worth checking.

poulpy1232• 4 years ago

why do you split the strings instead of comparing them directly ?

srcreigh• 4 years ago

This was one of the interview questions for my first ever programming internship at 19.

fluoridation2• 4 years ago

As a long time C++ developer, the modern trend of library package managers has always seemed insane to me. In C and C++ integrating with a dependency is no trifling matter. Beyond just simply making your executables bigger, you're also making the build process more complex, you're locking yourself to an interface, and you probably need to keep special considerations in mind; for example, some libraries need you to call a function when you load them. Therefore, people don't make trivial (i.e. things anyone can do in a few minutes) libraries, and you don't add dependencies unless you really need to.

fragmede2• 4 years ago

Back in my day, we coded in assembly and looked up opcodes in paper books. And we liked it! At least we weren't still on punch cards.

Eg, I use the python docopt library because it makes life easier and more straight-forwards once you understand it. Which, to be fair, not everyone wants to put in the time to learn. But I like it more than having to code up an argument parser from scratch using string matching, and then having to do that every single time... I don't need it, but I also don't anything more than nasm, anything more than that is luxury! Kids these days...

commandlinefan2• 4 years ago

we coded in assembly and looked up opcodes in paper books

Don't threaten me with a good time.

P5fRxh5kUvp2th2• 4 years ago

Most people use abstractions so they don't have to.

There's a difference between someone using an ORM who knows SQL and someone using an ORM so they don't have to learn.

The fact is, most people use these abstractions so they don't have to understand it.

---

Back in my day ... it was accepted that you needed to understand atleast 1 level of abstraction below where you're working at. Nowadays, that attitude is only used by older developers.

chasd00• 4 years ago

Simplifying is hard. But when you get good at it, you find you're much more buoyant.

it's not finished when there's nothing left to add, it's finished when there's nothing left to take away.

Supermancho2• 4 years ago

Code can always be improved. How you evaluate tradeoffs both varies between individuals and your own views from day to day.

Code is never finished, until it's forgotten or feared.

collyw• 4 years ago

I try to convince them to keep stuff simple at my work, but we still go creating a shitshow of epic proportions. If our app was done the traditional way it could probably run on a Rasperry Pi. Instead they are pushing for microservicces, one database per service has been mentioned. A SPA app for the frontend. All for what amounts to a standard crud app that sends some jobs to the cloud.

lake_vincent• 4 years ago

Just a quick shout-out to Python Poetry (https://python-poetry.org/) which makes package, version, and virtual environment management extremely simple!

I love it so much that I can't imagine having to work on a python project without Poetry now, lol.

jmillikin• 4 years ago

In the "old" days ("old" being .. what, ten years ago?) packages were released as libraries, which bundled large amounts of functionality into a single atomic version. You could depend on OpenSSL for all your crypto code, GLib for argument parsing or atomics or UUIDs, libiconv for character encoding. There was some expectation (usually met) that the developers of these libraries would have some sort of basic testing strategy, and that they would try to maintain API compatibility over long periods of time.

A large program might have less than a dozen external dependencies, and they released every couples months at most, so keeping up to date was easy. And most of the time you didn't even need to update unless someone discovered a security vulnerability, which was not that common despite all the libraries being written in C or C++.

The approach of NPM (etc) is fundamentally different in that it's considered normal for a project to have hundreds or thousands of dependencies. To manage the update cadence, the users of these package repositories try to invent new schemes for version numbers -- or write quirky gif-laden blog posts about how they burn the monthly budget of a small country on testing in CI.

But the core problem is that they've locked themselves into a stupid and unsustainable paradigm. You can't do the sort of software development that involves writing code if you've decided that your job is to haphazardly glue together other people's code, especially when your're willing to depend on packages written by anything and anyone.

I honestly think a lot of the problem of over-dependency comes from the new package managers that make it too easy to depend on stuff recursively. If NPM required you to type out `npm add-project-dependency left-pad/1.0.0 --checksum sha256:a1b2c3...` then that would have been a strong forcing function against a huge number of tiny packages, because at some point your fingers get tired.

For example, in my personal projects I use Bazel as a build system, which makes me a bit more aware of dependency hell because each dependency needs to be registered.

There are projects like QEMU that are straightforward to build in Bazel because their dependency tree is bounded, but some of the stuff coming out of JavaScript land is just unavailable to me because I don't have the patience to chase down hundreds of packages to run a "hello world".

I've found I avoid certain parts of the Rust ecosystem because they're JS-ish, but there's other parts that have a more C/C++-ish style and those packages work fine for me. Similarly for Python. Go has largely avoided the issue, though I don't have an intuition as to why (different culture?).

juped• 4 years ago

A problem with Rust is that the compilation unit is the crate, which pushes developers into splitting something that is one library into multiple crates to get incremental compilation, and it's hard for the user to tell the difference (you can synchronize their version numbers, sure, but the user has to put each crate into their list of dependencies individually).

pferde3• 4 years ago

It's not even in the "old" days, what you described is still the situation today, for C and C++ languages - as I'm sure you are aware. You have to know about all your dependencies, and curate them all manually. This despite all the misguided "package manager for C projects" attempts that pop up here on HN every now and then.

BirAdam• 4 years ago

Well, C was made with UNIX in mind. To that end, UNIX/Linux is both the package manager and dev environment for C. Further C package managers are just redundant.

gernb• 4 years ago

I went through dependency hell today.

I wanted to start a new react project and someone who I was going to collaborate with said they wanted try storybook. Ok, follow the tutorial. It immediately spits out all these warnings

warning " > @testing-library/user-event@13.5.0" has unmet peer dependency "@testing-library/dom@>=7.21.4". warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-syntax-flow@^7.14.5". warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-transform-react-jsx@^7.14.9". warning "@storybook/builder-webpack5 > fork-ts-checker-webpack-plugin@6.5.2" has unmet peer dependency "typescript@>= 2.7". warning "@storybook/addon-essentials > @storybook/addon-docs > @babel/plugin-transform-react-jsx@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0". warning "react-scripts > eslint-config-react-app > @typescript-eslint/eslint-plugin > tsutils@3.21.0" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta". warning "@storybook/addon-actions > react-inspector@5.1.1" has incorrect peer dependency "react@^16.8.4 || ^17.0.0". warning " > @storybook/addon-essentials@6.5.7" has unmet peer dependency "@babel/core@^7.9.6". warning "@storybook/addon-essentials > @storybook/addon-docs > @mdx-js/react@1.6.22" has incorrect peer dependency "react@^16.13.1 || ^17.0.0". warning "@storybook/addon-interactions > @devtools-ds/object-inspector > @devtools-ds/themes > @design-systems/utils@2.12.0" has unmet peer dependency "@types/react@". warning " > @storybook/preset-create-react-app@4.1.2" has unmet peer dependency "@babel/core@". warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin@1.0.2-canary.6.9d540b91e815f8fc2f8829189deb00553559ff63.0" has unmet peer dependency "typescript@>= 3.x". warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin > react-docgen-typescript@2.2.2" has unmet peer dependency "typescript@>= 4.3.x". warning " > @storybook/react@6.5.7" has unmet peer dependency "require-from-string@^2.0.2". warning "@storybook/react > @babel/preset-flow@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0". warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1". warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react-dom@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1".

So that failed IMO. There should be zero warnings! A pile of warnings has no meaning because no will notice when it goes from 18 to 19 warnings.

I ended up aborting that and trying other things. It's been 10hrs now and I still have nothing working. Every path has lead to some kind of out dated article and/or dependency issue.

tralarpa• 4 years ago

I knew React would show up here :)

dvh• 4 years ago

The thing I hate the most is when some JavaScript library uses some obscure build system (or any build system really) but on GitHub they don't publish the built version. What could have been simple download now turns into several hours exercise in trying to understand and make work yet another build system, downloading intermediate bullshit from random websites. Just publish the damn library, thanks.

cudgy2• 4 years ago

True, but then how do you know the library was built with the code in the repository? This could introduce a security risk.

rini172• 4 years ago

All these build dependencies don't introduce security risk?

sinuhe69• 4 years ago

For simple, once used projects, I don’t concern much with these aspects. But for a big projects, choosing your framework and dependency is always a big concern and should do it carefully. Investigate all your dependency is a must, not just for the security aspect. And when you do it, you will convince yourself less is more and your team will re-create many things just to reduce the dependency.

amelius2• 4 years ago

I'm personally amazed that we see so few security issues with all the dependency chains that we build on.

walderf• 4 years ago

that doesn't mean they aren't there.

fmajid• 4 years ago

Read also this excellent article by Russ Cox (of Go fame) on the issue:

https://research.swtch.com/deps

This is at heart an economics problem: how to provide a public good (software security) in the presence of free riders. The way society handles this kind of risk is through insurance, but there are a number of things that need to happen to enable a sustainable economic basis to fund the necessary dependency-vetting to happen.

Edit: I wrote up my thoughts on the way forward here:

https://blog.majid.info/supply-chain-vetting/

bob10292• 4 years ago

If you are really struggling, I'd suggest trying out the latest .NET before abandoning all hope.

I was recently able to build a 3d software rasterizer and streaming web service using .NET6 and I did not have to consume a single 3rd party dependency.

Even things like SIMD-optimized projection matrix calculations can be found in the box (System.Numerics).

There is obviously the Microsoft aspect to this equation, but I feel like you should at least try the porridge before throwing it away. The empire has a hell of a Death Star these days.

ajuc• 4 years ago

Most languages were like that when they were new. Famously Python. It doesn't last.

Deukhoofd• 4 years ago

C# is 22 years old.

andyp-kw3• 4 years ago

C# is old enough to drive.

I don't think any languages are feature-full when they're new. It takes time for the ecosystem to develop.

trinovantes• 4 years ago

Did you need to build a GUI? It feels like Microsoft invents some new Windows UI library every 5 years or so and deprecates the previous one. The last I used .NET, it was WPF but I hear WinUI is the hot thing now.

pjc503• 4 years ago

Yes, I was wondering where they were rasterizing to. Although of course the most long-lived way is to just P/Invoke straight into user32.dll to bring up a window.

(The last time I wrote a 3D rasterizer was in 1996, and I had to type my dependencies in from paper into Borland Turbo C)

thomascgalvin• 4 years ago

GUI frameworks are subject to more churn than, I think, any other programming domain.

On the desktop, you have WPF, WinUI, QT, GTK, Swing, JavaFX, Electron, and a dozen more. On the frontend, you have React, Angular, Svelte, Ember, Vue, and probably a thousand more, plus roughly as many CSS toolkits as there are atoms in the universe.

And honestly? None of them are that great. Most of them are perfectly functional, but at some point you hit a limit of their abstractions, and start fighting against the framework, rather than fighting alongside it.

This is probably true of any domain if you go deep enough, but UI development seems significantly more fraught with danger and churn than the backend.

P5fRxh5kUvp2th2• 4 years ago

This is because the current UI solutions are windows only and they're trying to create something that works cross-platform.

electroly2• 4 years ago

I've been writing Windows Forms applications for 21 years. It's still officially supported and maintained on the latest .NET. I don't think it will ever die. Microsoft invents new libraries but they deprecate nothing.

pjmlp3• 4 years ago

The hot new thing its Blazor the WebAssembly framework. They are pushing it everywhere, even inside Web widgets on MAUI.

Then you have MAUI, which is basically the rebranding of Xamarin Forms, but it does required a bit of rewriting, and apparently there are some bugs still. And on the Mac they have taken the shortcut of building on top of Catalyst to reuse iOS Xamarin.

Then you have the WinDev team pushing WinUI 3.0, still behind UWP capabilities and moreso behind WPF.

Currently on .NET your safest bets are Windows Forms, WPF, the 3rd party projects like Avalonia and Uno, or if the use case is possible, Web.

I would wait for MAUI to mature a bit more, consider WinUI a implementation detail of MAUI on Windows, and ignore it for everything else.

tedk-42• 4 years ago

We're not drowning. The boat was not made to be waterproof and yet we think it is.

Any software you build, like everything else needs maintenance. It's a fallacy to think that just because math is correct (boolean algebra) that your code will tick along fine indefinitely.

andyp-kw3• 4 years ago

This is a good point. However some ecosystems and languages might require less maintenance than others.

openfuture• 4 years ago

How about building on solid ground for once? Datalisp.is the thing I keep posting here and it keeps getting overlooked but a solid foundation means longer lasting structures, just look at the 3-4-5 triangle; it still has a right angle.

prepend• 4 years ago

Who’s drowning? I feel better than ever. This is a great problem compared to the old way of stuff being out of date and it being a giant pain to use libraries and stuff.

Also (knock wood) I haven’t experienced any dependency related outages and have been able to run Java, ruby, Python, and javascript stuff for many years without problems. When I need reproducibility I pin to specific versions. And I vet what packages I use and don’t just randomly grab whatever is in the stackoverflow copy and paste.

imwillofficial• 4 years ago

This type of overwhelming amounts of info has made me focus my efforts more and narrow my focus

joeyguerra• 4 years ago

Not me.

tobyhinloopen• 4 years ago

Stop using NodeJS.

Solved.

mindaslab• 4 years ago

agreed 100%

codegeek• 4 years ago

Not that simple. You also have to stop using some popular frameworks/libraries even in CSS. For example, Tailwind is the hotness these days. I like it but in production, they recommend installing using npm and not CDN. Yes it is for performance and size reasons but still.

spion• 4 years ago

We're really bad at code reuse.

commandlinefan2• 4 years ago

Another downside of the deluge of dependencies that I don't see discussed often enough is bloat. If you import something that imports something else that imports something else ad nauseam, you're importing a lot of stuff you're not actually using and won't ever actually need. Tree shaking was supposed to help with that, but it barely moves the scale.

The author says that "Reusing external code is a huge advantage, and when everyone does it, it’s also a competitive necessity", but when I've taken it upon myself to "reinvent" something that there's already a moderately decent solution for, I've found that it really wasn't that much extra effort and the end result was insanely more flexible in the ways that I needed it to be than the supposedly off-the-shelf alternative.

forgotmypw17• 4 years ago

Every library is code I have to maintain without even knowing what's inside.

I only use libraries for something I absolutely can't write myself, like PGP.

I also avoid languages and technologies which introduce breaking changes.

e.g. Python will never be a top-level language in my project, only an optional add-on for scripting, with failure checks.

shp0ngle• 4 years ago

ahh yeah the good old days of getting a library from SourceForge

fleddr• 4 years ago

It's basically a batteries not included problem.

Say you create a new React app. The fact that you do is an indicator that the core tech (web tech in this case) does not satisfy your needs out of the box. So you grab a framework from "user land", one of many.

React though is not self-contained. To even get it to build anything, it depends on various other projects that each have deep dependencies. Webpack, linters, a choice of CSS framework, etc.

At this point, you app itself has zero functional dependencies. You haven't even produced a hello world, all of this stuff is needed just to get it to do anything at all.

As you'll then start to build actual functionality, you'll notice that React doesn't have facilities or an opinion on the most basic of stuff. Loading data, managing forms, standardized UI, the essentials are not there. So you go for robust/popular libraries for each of those gaps, which in turn have their own dependency trees.

This way even the simplest of apps require thousands of dependencies. It's not due to developers too lazy to write "leftpad". It's a tech stack issue where nothing is standardized or included out of the box.

mikkergp• 4 years ago

Isn't this the consequence of a large, free, open, diverse system? I remember the constant complaints about their being "too many javascript" frameworks a few years ago, and how their are always new ones.

I'm less concerned with the moral or practical considerations here, but can this really be a bad thing? Or, perhaps more to the point, isn't it a sign of a healthy ecosystem?

It seems as the world gets more interconnected this problem of "too much"; it's an interesting one to solve... And I'm not saying we shouldn't, but I guess I'm curious about the philosophical question of complaining about complexity in a decentralized public ecosystem. Is there a solution to the ecosystem as a whole? Or will sort of curated subgroups of the ecosystem be birthed that attempt to create a walled garden. It doesn't seem possible to create a shared walled garden without stifling innovation, becoming more closed, all the things that I think most open source communities don't want to do.

electroly2• 4 years ago

I lightheartedly object to C# being mentioned in the same breath as the rest. I've been writing Windows Forms applications for 21 years. Most C# applications have fewer than 10 dependencies, and they tend not to have their own transitive dependencies. It's bliss. We just really don't have the same level of problems in this ecosystem.

sdflhasjd• 4 years ago

Yes, having originally been a C# developer and now diversified and had some experience in other tech, I realised that we are spoiled by one of the most robust standard library and suite of first party libraries that nothing else I've ever used has stood up to it.

pjmlp3• 4 years ago

Java is the only ecosystem on comparable level, hence why both of them are my daily tools.

vikingerik2• 4 years ago

I have to disagree, C# can be as bad too. I work on a medium-sized (~500k loc) C# codebase, and we get tangled up in dependency chains frequently. It's a web app with a number of integrations and API calls for things like Google/Facebook/Bing ads that each use some library from them, and other stuff like cloud storage calls and FTP file deliveries. They all tend to bring in more dependencies, like Newtonsoft Json parsing, Excel reader/writer classes, HTTPS and certificate management libraries, log4net logging. I spend a non-trivial amount of time sorting out version conflicts among these, even with the Nuget package manager doing most of the work.

andyp-kw3• 4 years ago

I don't understand how any of the libraries you mentioned would cause dependency issues unless you're referencing them in a weird way.

Nuget packages are usually compiled and self contained with their dependencies.

codalan• 4 years ago

It depends. I've encountered C# apps with very few NuGet packages.

And then there are the ones that pull in all sorts of logging libraries and abstractions, 3rd party DI frameworks, 3rd party ORMs, Newtonsoft JSON, and more!

Fortunately, the trend is moving back to more centralized, MSFT maintained packages and libraries (EF Core, ASP DI, System.Json, etc.), which makes future maintenance less of a headache.

The main issue I have with 3rd party packages (particularly with obscure ones) is vetting them out. Arguably, the same is true for pulling in MSFT libraries, but when serious security issues happen, you can expect MSFT to (eventually) resolve it. I don't really get that guarantee from 3rd party packages, which might have only 1-2 unpaid maintainers if you are lucky. If you are unlucky, the code might be abandoned and completely unreadable.

I try to limit this risk by only using well-known, well-maintained packages, e.g. Serilog, and keeping package dependencies as minimal as possible.

OmarIsmail• 4 years ago

I'm going through this right now. At dayjob we have a lot of internal dependencies and libraries that are created sufficiently far from me that I treat them as third party and when things go wrong (which they do a lot) it's a nightmare to debug and work through. Easily half a day gone.

So for sideproject I'm going mostly bespoke and using very little 3rd party libraries. The result of that is there's sooooooo much boiler plate to write and it's very boring and it's hard to be motivated and productive to get through it. What's keeping me going is that once the boilerplate is done - it's mostly the foundational level of data queries + endpoints - then I shouldn't really have to touch that stuff again.

My conclusion is that there's no great answer.

Using lots of libraries is like riding a horse. It'll start moving right away but you don't have full control and you gotta find the right way to get it to do what you want, and may have to fight it if it really doesn't want to. Vs building most things yourself is like building a car. You don't go anywhere for a long time as you're building it, but once you've built it you can move very quickly and if anything goes wrong you can easily pop the hood and fix/change things up.

I think for long lived projects it's better to build the car, but also be disciplined in writing very good code + documentation. The initial build out sucks hard but that pain will get amortized over a very long future and ultimately be worth it.

And for absolutely required dependencies go with paid services. Specifically paid services where the company's primary focus _is_ providing that service. They are financially and existentially motivated to give good service, and are usually good about keeping backwards compatibility while usually staying on top of security/modernization upgrades without requiring work on your own end. A managed database is a good example of this kind of paid dependency.

mperham• 4 years ago

This path was paved when NPM and Node.js decided that lots of small dependencies are a good thing. All of this damage and JavaScript's reputation as a maintenance nightmare is a result of that terrible, terrible decision.

We're Drowning

Related Stories