Dependency Sprawl
"For all the talk of the power of leverage and the virtues of using tools and systems built by others, there seems to be little conversation being had about how to decide what to learn or do yourself."
"I like to pick on Node.js because you have the npm package system where everything just sprawls out to infinity... You install one thing and suddenly now you've got 100GB of dependencies. And even in Python we have to be really careful with that. What does our dependency sprawl look like?" —Adam Guenther, from The Changelog Podcast: Episode 605
I love this concept of Dependency Sprawl.
It captures the nuanced tradeoff inherent in relying on others in any domain. There is opportunity cost to doing everything yourself, building things from scratch, learning new skills... But there are also diminishing returns to over-reliance on someone else. Over-outsourcing. Over-dependence.
Guenther currently focuses on software reliability at OpenAI but was speaking here about his previous role involving software development for a satellite company. He was working with applications whose dependencies—the software libraries built and maintained elsewhere but imported—carried different consequences and considerations depending on whether they were required for satellite software remotely accessed in space or coordination software more directly accessed on Earth. There's upload/download time delay to factor in but also speed of updates and download size.
For all the talk of the power of leverage and the virtues of using tools and systems built by others, there seems to be little conversation being had about how to decide what to learn or do yourself.
- When is it best to embrace dependence in order to do something more quickly and when should one instead invest the time and resources to build?
- What are the weightings of each factor and how do those values change over time?
- Where is the tipping point into Dependency Sprawl? What does that look like?
- How quickly would Dependency Sprawl accelerate? What would be the consequences?
Importing packages and libraries into software projects is vital and helpful, but it's good to remember that it can quickly get out of hand and compromise a project with the bloat of the unnecessary. The same principle applies to home DIY projects or ongoing business relationships. Scale and leverage come from depending on others but can also plant the seeds of laziness and vulnerability. It's a balancing act.
Perhaps the best way forward looks less like a complicated decision tree of if/else statements to be followed, and more like a garden to be intermittently tended to and pruned. We don't build our gardens so much as we manage them, letting their nature find a balance between growth and health. Protection and resilience. Dependency and self-reliance.
It's usually time to step in only when things start to sprawl.
