The C-Shaped Hole in Package Management

Summary

System package managers and language package managers are both called package managers. They both resolve dependencies, download code, and install software. But they evolved to solve different problems, and the overlap is where all the friction lives. If you drew a venn diagram, C libraries would sit right in the middle: needed by language packages, provided by system packages, understood by neither in a way the other can use. As Kristoffer Grönlund put it in 2017: “Why are we trying to manage packages from one package manager with a different package manager?” apt, dnf, pacman, and the rest started as application delivery mechanisms. Users needed to install Firefox or LibreOffice, and decomposing applications into shared libraries emerged as an implementation detail. The dependency graph is there to serve patching and security updates for end users, not developers. A package like python3-requests isn’t there because the Python ecosystem wanted it. It’s there because some GUI application or system tool needed it, and the distro maintainers packaged it. System package managers generally keep only one version of each package at a time. This massively simplifies dependency resolution, but it means getting a newer or older version is hard for individual users without upgrading the whole system. It’s a stop-the-world model. Hacks like naming packages python3 and python2 exist to work around this when you really need multiple versions, but they’re exceptions. npm, pip, cargo, and gem went the other direction. They’re dependency assembly tools that help developers build projects. They keep every version around indefinitely, letting projects pin exactly what they need. They’re also cross-platform by design: pip doesn’t know if it’s running on Debian or Fedora or macOS, so it can’t just shell out to the system package manager to install C dependencies even if it wanted to. The fact that you can pip install httpie and get a working command-line tool is a side effect, not the ...