You've got to look at this in a context where platform package managers like apt are simultaneously 1) platform-specific 2) jealous, insisting that every language has to conform to their way of doing things and 3) fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package. Platform package managers like maven have been far more successful because they provide a much better user experience, and frankly I suspect this is if anything because they're so thoroughly isolated from the platform package manager.
That's quite a negative view of the situation. The debian maintainers definitely don't insist that everything has to do things the apt-way. They only insist that packages for the official repos do so, which seems sensible enough to me. Everyone is still free to build their own deb packages or distribute their applications through install scripts or app images or any one of a hundred thousand different ways to distribute programs.
But if you want a package in the official repos you have to play by the rules they set, the primary one being that official packages ought to only depend on official packages. It'd simply be impossible to have any guarantees about the quality of a package otherwise.
Here here. And the only thing that makes Debian an interesting brand is the quality consistency of the official repository.
If someone doesn't like apt & the main repository they shouldn't use Debian because that is pretty much the only big thing separating Debian from the tens to hundreds of other distros out there. Take apt and the main repo out of Debian and it is basically Linux From Scratch with a bigger team of bugtesters. Distributions aren't 'jealous' about packet management, they are the packet management. It is right there in the name "Debian distribution". The Debian brand, and all linux distro brands, are deeply linked to how they execute packet management to distribute software.
It's not just a question of being in the official repos. Even if you're running a private repository, the deb tooling is not suitable for building many kinds of software: it's unwilling to play nice with "external" dependencies, and still single-platform and limited in its functionality.
> unwilling to play nice with "external" dependencies
But that's a policy decision - the dependencies should themselves be packages, because that's the only way to guarantee reliability.
The inability to install multiple versions is probably the only serious problem with the dpkg model. Even then we might argue that the dependent software should be fixed so that it's not so fussy about specific dependency versions.
Installing multiple versions is possible, given proper semver and good arguments for it: There are quite a few libfoo5, libfoo6, libfoo7 packages. What isn't and shouldn't be possible is installing libfoo 6.6.5 and 6.6.6 at the same time, because those should be compatible. And not being easy to replace (i.e. ABI- and API-compatible) means that security and bug fixes will be missed.
Of course this is work for application as well as library developers, designing and keeping compatibility is hard. Most would rather build the new shiny, consequences be damned.
I think these are noble but naive approaches - we have been developing software for a few decades now and sometimes the software is not compatible without the developer knowing it. You should be able to use whichever version of the software you need.
"Whichever version you need" comes with the responsibility of providing security and bugfix upgrades in a timely fashion. Almost all developers fail spectacularly at this, with latencies in the order of months. This is clearly not viable.
This was my experience as a developer on a SOA team that provided APIs for the rest of the organization (a large regional healthcare system). Even changes that didn't officially break the API could break our customers in ways we didn't anticipate.
When I left that position, we were in the middle of an OpenShift (k8s) deployment, and one of the things motivating the change was the ability to more easily run separate versions of services for different customers if the need arose. Yes, this would put a lot more burden on us as the service provider, but it would also allow us to iterate faster while maintaining stability.
Yeah, but who decides if it's a defect? Maybe the defect is subtle and only manifests in rare cases and the upstream rightfully decides that after a risk/payoff analysis, it's not worth fixing it.
Or maybe he agrees and fixes it and it's included in distributions 7 years later.
I've changed 5 jobs in 7 years (because of life circumstances) and I'm not even a job hopper... How is any commercial shop going to plan around 7 year time frames?
Well, within a debian distribution the Debian developer responsible for the downstream package would get the dependency fixed, either themselves or by badgering the maintainer, and use the fixed Debian version. This gives Debian a consistent set of packages that work together. Yes, it's a lot of work and it's why Debian are behind other distributions and don't include Hadoop. But it reduces the unpleasant surprises.
It's effectively part of insisting that it's properly Free software - if you can't maintain your own bugfixed fork, but have to keep going to an external organisation for their version, is it really Free?
That's unlikely. Worst case: examine where the package will put its files, and prep your system with symbolic links to directories residing on a file system on the other disk(s).
But that raises the question: why are your file systems sensitive to disk drives? Use a layer (or more) of indirection through LVM and RAID so to make the drive placement opaque and robust.
> why are your file systems sensitive to disk drives?
A couple of systems I have at home have a 16GB primary disk, so it is useful to be able to put things on an external disk. I have yet to see any package manager that can actually handle that.
> Use a layer (or more) of indirection through LVM and RAID...
Yeah, that's a great idea, just introduce more complication and abstraction to make up for a shortcoming in how Linux developers think about applications.
Or, just have your programs be self contained in a single file or folder and put them wherever it makes sense.
Hear hear. If anything .debs dont care enough where dependencies come from.
As long as the package name matches you are good.
I have no complaints about debian packaging. I've used lots of different ones. All official packages are signed and its easy to sign your own packages.
Language-specific package managers are also "jealous" as you put it. They insist that every platform conform to their way of doing things. They're also promiscuous, in the sense that most have really poor dependency management ("let's build the internet today just in case") and practically no security. The computing world would be a better place if the people forcing their language-specific idioms on the rest of the world had instead spent their time improving the platform package managers that already existed at the time.
> They're also promiscuous, in the sense that most have really poor dependency management ("let's build the internet today just in case") and practically no security.
Not criticisms that apply to maven. Binary dependencies on released versions are the norm, and all packages in the central repository are signed with GPG, just as with apt.
> The computing world would be a better place if the people forcing their language-specific idioms on the rest of the world had instead spent their time improving the platform package managers that already existed at the time.
If those platform package managers had been open to being improved, which I don't think they were (and conversely, deb/apt originated because Debian insisted on doing things their own way rather than improving RPM). Apt's closed-world assumptions seem like a policy decision. Apt not being cross-platform is definitely policy. And surely it's occurred to people in Debian that the ability to have per-user installs, and more importantly non-shared/non-overlapping installs of libraries that are depended on, would be useful; the insistence on single system-wide installs of libraries can only be a policy decision, and one that's turned out to result in a less usable system.
> jealous, insisting that every language has to conform to their way of doing things
Let's ignore for a moment all your baseless assertions and focus on the following question: how do you ensure that a build is reproducible if you have no control on which required dependencies are downloaded by a build system which requires root access?
Is jealousy the only explanation that comes up to you?
Nix is a non-sequitur. Nix requires specific packages and package versions to be specified, and Nix requires the build process to be "free from side effects". That fails to address both problems stated by the OP, and actually try to reimplement what Debian's package building process already does. Other than trying to publicize Nix, your comment adds nothing to the discussion.
>how do you ensure that a build is reproducible if you have no control on which required dependencies are downloaded by a build system which requires root access?
By having control over which required dependencies are downloaded. As Nix does. Why would you settle for not solving that problem?
Sorry to hear you don't think my comment adds to the discussion, but I'd disagree
> By having control over which required dependencies are downloaded.
Again, Nix adds nothing to the discussion. The problem stated by the OP was caused by the (broken) way that the custom build system of a specific package was designed to work, which failed to adhere to the standard practices enforced by Debian's packaging process.
If the packagers of said software project followed Debian's practices then the problem wouldn't exist.
You're parroting that Nix also enables users to specify dependencies. Ok, so it works just like any other package system. What does that have to do with the problem being discussed? Nothing.
It feels like you're being unnecessarily hostile, here. He didn't address your specific problem as stated, but he did add to the discussion.
I don't think it's fair to say that Nix is 'just another package manager'. It solves many of the stated problems with distro package managers (overlapping versions, user-specific packages, strict build environment rules), and provides many of the benefits of Docker and it's ilk (perfectly reproduceable environments for packages to run, including dependencies that don't fit well into traditional package managers, like JARs). Because it doesn't rely on the standard POSIX filesystem layout, it runs happily on any Linux or OSX system, alongside whatever package manager your system uses.
If people started using Nix recipies instead of Docker or janky bash scripts for deployment of Hadoop and other complex software, most of this article's complaints would disappear. And Nix is, if anything, better from a sysadmin point of view than apt-style packaging systems. It makes a fine distro package manager (see: NixOS).
> It feels like you're being unnecessarily hostile, here. He didn't address your specific problem as stated, but he did add to the discussion.
You're right, sorry for the tone. The thing is, it sounded an awful lot like a blatant attempt at derailing the discussion by shoehorning span to promote a build tool. Nix does not solve anything, particularly as it was being proposed as a solution to a problem that plain old Debian packages do not have. So if Nix solves nothing and Nix adds nothing to the discussion then why waste everyone's time by adding noise by selling a tool that does and solves nothing wrt plain old Debian packaging?
> If people started using Nix recipies instead of Docker or janky bash scripts for deployment of Hadoop and other complex software, most of this article's complaints would disappear.
...or simply build a plain old Debian package?
Is it that hard to simply follow the happy path of packaging for Debian?
Why is suddenly Nix the only option on the table, specially as it brings absolutely nothing to it wrt what plain old Debian packages already provide for decades now?
I like Apt a lot, but it does have some shortcomings.
First, of course, it requires an apt-based distro, so software distributors need to have apt alongside all the other packaging alternatives. Or they just provide a bash script. Ew.
Second, apt doesn't elegantly handle different versions of the same package. That's rarely an issue for well-established C libraries, but it's a big issue for Java and most of the dynamic languages. So you end up with a host of language-specific package managers.
Third, there's stuff beyond simple files that falls outside the wheelhouse of apt. Networking, configuration, whatever. The paradigm for apt is very much to have a small number of systems, manually curated by dedicated sysadmins. When you start scaling up to tens, hundreds, and thousands of hosts, you end up writing and maintaining long scripts to initialize a freshly-installed system and put it in the right state. And those script will break as packages evolve. Getting a system into a known-working state is difficult.
A specific example: I installed and set up GitLab on a Debian system a while back, and it was a huge pain. It's not just a package, after all, it's web code, a couple daemons, a sql db, a redis db, git repos on the filesystem, and more. The install guide was pages and pages long. I never quite got it working right (something about SSL certs, IIRC one of the daemons wasn't using the system CAs?).
So I tried docker for the first time, and had GitLab up and running in about 10 minutes.
And if I ever wanted to migrate to a different host, spin up another node for load balancing, or do backups and restores, you bet your ass I'd use docker.
Apt is great for carefully curated, individual systems. It was perfect for the world circa, say, 2005, and the world would be better of if we'd all standardized on it then. But even if we had, somebody would've invented something like docker in the meantime, for managing complex software (like GitLab) on tens, hundreds, or thousands of hosts.
But docker has all the issues pointed out in the article above, and more besides (every image is hundreds of megs, because it contains a full, running Linux system...that's just crazy).
Nix can do the package management thing that apt does so well, and it can also do the reproduceable, holistic system build thing that Docker does. It can also make management of language dependencies (i.e. Java JARs) much more clean and elegant.
> That fails to address both problems stated by the OP
This is wrong and unfair. One of the major problems the OP has with Docker is that many software builds are unreproducible, encouraging many people to deploy binaries of dubious quality. Nix tries to solve precisely this problem through better tooling which makes it easy to ensure that its packages are reproducible and its dependencies easily verifiable. This is also what makes Nix distinct from Debian, which tries to improve package quality through policies and community collaboration.
> Nix requires the build process to be "free from side effects"
I disagree that this makes Nix irrelevant to the discussion at hand. Another gripe the OP had with Docker is that it sandboxes entire apps, making it a blackbox. Nix does sandboxing on a more granular level, which provides more transparency into individual packages.
> Reproducible builds were a non-goal for apt for most of its existence.
Your comment sounds very disingenuous, as Debian was first released in 1993 (about 26 years ago) and has been pushing for reproducible builds since 2000.
Your own comment is extremely disingenuous. Your own link states that there was very little enthusiasm for reproducible builds when mentioned in 2000 and 2007, and the first serious effort towards reproducible builds started in 2013.
That doesn't dismiss its immense benefits. Besides, what's the source of your claim? Package quality, ease of install, and security were always high-priority topics for Debian AFAICT.
Decades of experience using Debian. Package quality and ease of install were indeed always a priority (and security may have been claimed as one), but reproducible builds were not seen as necessary for any of those.
you can use a tool like snap or appimage to pull down the newest full release of your software.. at least then, all the dependancies are 'bundled' into the image, and your not having people go randomly install crap from around the internet with no real way to update.
Jealousy is the characterisation of exactly that behavior/aspiration to have everything managed through the OS's package manager. It's a nice effort, but for people not firmly within the ecosystem it's often a hard effort to justify to package something specifically for every distribution.
> 2) jealous, insisting that every language has to conform to their way of doing things
Well, programming languages clearly should not be in business of software distribution. That creates unnecessary tight coupling between language, build system and distribution, causes proliferation of language-specific package managers incompatible with a platform way of doing things.
Something Java and the Java ecosystem is terrible at. I've had project where we needed all of 7,8,9 and 10 [0] for the data stack: Hadoop, Scala, Spark, HDFS, zookeeper, pyspark. Good luck setting that up on a single machine, containers all the way. The java path nightmares I still have. I'm probably one of the few people in the word who know this much about the JVM without ever having written any Java.
[0] We should have also been following 11 but just didn't have the resources to even think about doing it.
Scala breaks binary compatibility approximately every 2-3 years, which is fast by JVM standards but slow compared to most programming languages (e.g. Python releases an incompatible version every 1-2 years). You can blame Scala for at most one incompatibility in that long list; older versions of Scala are incompatible with Java 9+ (this is true for many JVM frameworks too, as Java 9 made breaking changes).
Scala's version is epoch.major.minor, not major.minor.patch. I though 2.12.(x+1) is generally binary compatible with 2.12.x. Or at least I personally never ran into that issue with Scala itself.
Right, but my experience is that Java (and a few others) have succeeded in this space. I actually wish that I could use maven to install system software rather than having to rely on something like apt.
No I don't. I use Debian derivatives because I can apt get all my stuff without thinking hard, because those package maintainers have done the hard work.
Well, clearly not, since you can't install Hadoop. Before you blame that on Hadoop, remember that it only requires some really quite basic things from its package manager, which apt is nevertheless completely unable to do: use libraries in the normal recommended way, play nice with Java, work cross-platform.
Whatever. (Great practices those, being stuck in a decade old JVM and requiring root to build! But well, whatever.)
If I can not install on Debian in a clean way, I will just try to avoid using the software (and everything that comes on the ecosystem). It may be unavoidable, but if there is an alternative, it will be preferred. If one alternative appears after I am dealing with it for some time, it is still preferred (because the work with out of distro software never ends).
An OS is a tool for running applications, not vice versa. Packaging an application should indeed be the responsibility of people working on that application, which is why language package managers like maven are successful. Debian explicitly rejected this philosophy in favour of one where OS maintainers are responsible for packaging all applications (thus the OS-specific packaging format and closed-world assumption of their package manager).
> Debian explicitly rejected this philosophy in favour of one where OS maintainers are responsible for packaging all applications
That assertion is quite wrong. Just because the OS has its official package repository, and just because OS maintainers volunteer their time to package some software projects, obviously that does not mean that maintainers are responsible for anything. You are confusing offering a convenience service with being responsible for packaging each and every software under the sun.
In fact, your baseless assertion ignores two basic facts: packages are proposed and adopted by volunteers, thus what you've described as "OS maintainers" is pretty much any random person who simply wants a software to be available for download in the distro's official repositories, and packaging systems such as Debian's apt supports private package repositories, where anyone can make available their packages to the world.
Another fact that you missed is that build systems such as Maven or msbuild or cmake or Gradle or whatever do support download packages only for a reason: convenience. It has absolutely nothing to do with where lies the responsibility of providing packages. It's convenient that a build fulfills all build dependencies. However, the responsibility of packaging and distributing packages of a software product lies exactly where it always was: those working on the software product.
> packages are proposed and adopted by volunteers, thus what you've described as "OS maintainers" is pretty much any random person who simply wants a software to be available for download in the distro's official repositories
The entirety of the Debian project are volunteers, random people who simply wanted xyz. And Debian's raison d'etre is as a "distribution" of existing software.
More to the point, Debian packagers explicitly overrule "upstream" application developers; see the history of cdrecord or the ssh key generation bug for particularly spectacular examples. How can you say it's the responsibility of those working on the software project when they don't get to make the final decisions?
> packaging systems such as Debian's apt supports private package repositories, where anyone can make available their packages to the world.
Not really, because apt's approach to dependencies requires a central naming authority. In practice any private repository is necessarily a dead end: you can package additional software that depends on software from the central system, but not vice versa.
> For Linux distros, the primary responsibility for packaging is actually with the distro maintainers, since packaging is distro-specific.
No it is not. Although distros might pick and adopt some packages and include them in their official repository, it's obvious to anyone that as a distro maintainer you are not responsible for packaging each and every software package under the sun.
Moreso, linux distros such as Debian rely on volunteers to propose and adopt packages, which obviously also includes people affiliated with each software project.
Additionally, it's patently obvious that the responsibility of packaging and making a software available to the public lies on the people involved in developing the software project.
> That said, a well-written app should not present any difficulty for the package maintainer.
That is true, particularly as the primary package maintainers are in fact those actually developing the software project.
Once you get used to this method of packaging it’s actually really empowering for the end user. You can download the source for anything in the system, modify it, recompile, and install it all with a standard set of commands (no one-off custom build steps for every package).
I feel it’s the closest thing to the spirit of “free software” we have today: software designed to empower the end user to read, understand, and modify their system without a huge amount of obfuscation.
It increases the burden on the packagers to reduce the burden on the users. I would be interested to see how often these features actually get used though. I wish distributions would put this front and center.
In my opinion there’s simply too many dependencies to begin with. Some of the simplest things put into a GitHub repository to be “shared” with the world is great—as a gist file. Software needs to be a little more self-contained. Software reuse these days is honestly taking “not reinventing the wheel” to lazy extremes.
System package management systems do things just fine. Ask yourself, “when did I last properly package a Debian library? Do I even know how to do it properly?” Most people won’t be able to answer that because they didn’t.
But truth be told, having each programming language a unique package manager sucks. I’d rather have a consistent way of managing packages on a given system so that I can use various programming languages.
> Software needs to be a little more self-contained. Software reuse these days is honestly taking “not reinventing the wheel” to lazy extremes.
The three cardinal virtues of a programmer are laziness, impatience and hubris. I've never seen a decent argument against using ever-smaller libraries (as long as our dependency management is good enough to track them); all of the usual arguments for reusing libraries still apply, even if we take them further than their originators imagined.
> System package management systems do things just fine. Ask yourself, “when did I last properly package a Debian library? Do I even know how to do it properly?” Most people won’t be able to answer that because they didn’t.
Well, as we're seeing, Debian is struggling to keep up with packaging of everything that users want. I have actually created some (unofficial) debian packages; it's fine, but it's not a particularly inspiring experience either. Creating a maven/pip/... package is generally a nicer experience.
> But truth be told, having each programming language a unique package manager sucks. I’d rather have a consistent way of managing packages on a given system so that I can use various programming languages.
In principle I agree. But I'd definitely need to be able to install independent copies of the same library, different packages for different users, and fundamentally just have a nice development experience when working on that package manager. And that's not something I see the sysadmin tradition being able to come up with.
>In my opinion there’s simply too many dependencies to begin with. Some of the simplest things put into a GitHub repository to be “shared” with the world is great—as a gist file. Software needs to be a little more self-contained. Software reuse these days is honestly taking “not reinventing the wheel” to lazy extremes.
All true. Did we learn nothing from the left-pad incident?
But platform managers like maven are 1) language-specific, 2) jealous, insisting that every OS and extra-linguistic dependency has to conform to their way of doing things, and 3) fundamentally, not very good, particularly with regard to multiple versions of the same package (technically, OSGi could do that...), repeatable builds, and the desire for do-what-I-mean behavior.
Platform managers have been successful because a) they have the same interface across operating systems, b) they're the only game in town if you want to work in that language, and c) everybody loves to re-write existing software in their favorite new language.
> 2) jealous, insisting that every OS and extra-linguistic dependency has to conform to their way of doing things
Not really. Language-specific package managers are usually capable of integrating with "system" dependencies from outside the language (certainly maven has decent support for this) - they kind of have to be after all.
> 3) fundamentally, not very good, particularly with regard to multiple versions of the same package (technically, OSGi could do that...), repeatable builds, and the desire for do-what-I-mean behavior.
Maven is just as opinionated, jealous, and terrible, but its impact is restricted to JVM packages, mostly (hopefully) in the development stage, and so the consistency issues don't come up.
I ultimately agree that the system package managers do a poor job of wrapping language-specific package managers. But you lost me when you suggested Maven has a good user experience compared to most OS package managers. If that was a sly joke, then it was a good one.
> jealous, insisting that every language has to conform to their way of doing things
Not at all. The large majority of packaging systems have reasonable and similar requirements for upstreams, like not bundling dependencies or not hardcoding paths.
> fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package
That's completely by design, and for good reasons.
Who are you, package manager, to decide that I shouldn't be able to use different versions of the same package? Do you know more about my context than I do?
> Who are you, package manager, to decide that I shouldn't be able to use different versions of the same package?
That's one of main point of a distribution, rather that simply throwing software on a hard drive.
The maintainers have to work hard to guarantee that a specific set of packages and version work well together. Well enough for most users to deploy 99% of the packages without surprises.
And then backport security updates (often faster than upstream), for 3 years, often 5.
And various companies provide longer term updates and maintenance.
Oh, and provide license checking and vetting.
Turns out that it's a lot of work and no serious distribution would consider supporting multiple version of packages while guaranteeing the same level of quality.
> Do you know more about my context than I do?
Given your statements about distributions... most likely yes.
> Given your statements about distributions... most likely yes.
Hello there, ad hominem.
Linux distributions are overreaching. They should offer those services for core software. I should be able to install Inkscape 0.91 and 0.92 at the same time, easily.
I don't see the security angle either. You could design the system in a way that makes it able to force updates into the individual user's broken packages. Heck, it shouldn't even be connected to a user - it's just that every user should be able to install a package, and what install might mean is
* compartmentalized side-effects (not per-user even -
users might decide they want multiple compartments for
different apps) and
* a shared storage area for all the packages and their
versions (i.e. Nix)
The design results with flawed user experience and the users will move past the restrictions one way or another. It's up to system package managers to realize why the old way of doing things is simply not good enough; otherwise, people will just move on.
I'm sad that this is not happening since in the process we're also losing the good bits of system package managers as per this article.
> Platform package managers like maven have been far more successful because they provide a much better user experience, and frankly I suspect this is if anything because they're so thoroughly isolated from the platform package manager.
This is just wrong. Package managements support simple models by design so you don't end up with a mess of conflicting and potentially insecure dependencies in deployment environments.
Use maven in your build pipeline, fine, but distribute a minimal package using tarball/RPM/APT/etc. so that only the minimal runtime dependencies install on the deployment target. In fact, I built my first RPM using the maven rpm plugin. It takes just minutes to set up and use.
