How updating all your packages at once is a good idea?

[danielo515]

After a year of using Nix for managing my personal macOS laptop, I started to notice one annoying pattern, and that is updating specific packages that I was interested in.
Everything in Nix came from a single source, which is Nix packages. And while the flake lock pins everything to specific versions, I can’t update specific packages to the latest version that I’m interested in. I am forced to pull a new version of all the packages from Nix packages.
How is this a good idea? I’m used to get interested in one program or package new version, read their changelog, and see if I’m interested or not, and if I trust it or not. Nix, on the other hand, forces you to update the entire system just to get one package to the latest version.
Am I missing something? This looks like a crazy to me.
How can this be a good model for managing a system dependencies?

There are, of course, alternatives, like using a specific nixpkgs source for that package or even build from source, but that gets tedious very fast, so I don’t think that is a solution.
Also, sorry if I sound dumb, but I’m a TypeScript developer, and I’m used to have a declared set of packages that my package manager gets and pins to the versions that I specify. I’m having trouble translating that way of working to Nix.

Thanks for clarifying this nob’s mind

[Zellrus]

For updating only specific packages, you can use nix profile. Yes, you lose declarative package management, but you still get access to the largest package repository available.

[danielo515]

I don’t like the idea of nix profile.
I want my system to be reproducible. I don’t want to have to build the same system over and over again.
I moved from normal dotfiles management to nix just to prevent this kind of situation.
If I wanted that I will just use brew, which is native to mac and usually has all the packages that I need to use.

Thanks for the suggestion, but it doesn’t look like a solution to me.

[Mapybara]

The point, I think, is for packages to be tested together. For some combinations of packages this is super important (e.g. ZFS and the kernel). For your choice of packages, maybe it isn’t.

[danielo515]

That makes sense.
However, how does testing certain packages together forces you to only use one specific set of packages?
If you always apply the same 5 packages to your system, those are the ones that you know work for you, hence tested toghether.

Maybe it’s because the same mechanism is being used for system-wide critical packages that doesn’t update often and user packages/apps that are a completely different story.

[Mapybara]

Sorry - I should have said that I meant all the testing that’s done by the maintainers and the CI system.

@danielo515

Maybe it’s because the same mechanism is being used for system-wide critical packages that doesn’t update often and user packages/apps that are a completely different story.

Yes, I think that’s a big part of it, especially for NixOS - maybe less so for you.

[gamebeaker]

@danielo515

There are, of course, alternatives, like using a specific nixpkgs source for that package or even build from source, but that gets tedious very fast

What do you mean with tedious?

I use 3 different nixpkgs sources nixpkgs, nixpkgs-unstable and nixpkgs-legacy. I overlay unstable and legacy like this:

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
    nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
    nixpkgs-legacy.url = "github:NixOS/nixpkgs/nixos-26.05";
    ...
  };
outputs = { self, nixpkgs, home-manager, nix-flatpak, nixpkgs-unstable, nixpkgs-legacy, ... }@inputs:
    let
      # This is the overlay logic for unstable
      overlay-unstable = final: prev: {
        unstable = import nixpkgs-unstable {
          inherit system;
          config.allowUnfree = true;
        };
      };
      ...
    in {
      nixosConfigurations."nixos" = nixpkgs.lib.nixosSystem {
        specialArgs = { inherit inputs; };
        modules = [
          {
            nixpkgs.overlays = [ overlay-unstable overlay-legacy ];
          }
        ...

with that i can simply write in my config:

pkgs.legacy.android-studio
pkgs.unstable.antigravity-fhs
pkgs.piper

depending on what i want.
If i want to update i use:

nix flake update nixpkgs home-manager nixpkgs-unstable nixpkgs-legacy

and before i enter it i think for a bit what i want to update and remove the ones i don’t want to update.

[sotormd]

@danielo515

Nix, on the other hand, forces you to update the entire system just to get one package to the latest version.

You are free to pin a specific commit of nixpkgs for any package you want, either using builtin fetchers like fetchTarball or using flake inputs. You can, of course, pin a branch like what @gamebeaker has done or even an exact commit.

[danielo515]

@gamebeaker

What do you mean with tedious?

precisely this:

@sotormd

You are free to pin a specific commit of nixpkgs for any package you want, either using builtin fetchers like fetchTarball or using flake inputs.

Not sure if it is clear what I’m doing here. The intention is to answer @gamebeaker question with the quote I added below. In your example you have just 3 nixpkgs versions. The scenario I’m describing is more like what sotormd suggested. Every-time I want a package specific version I need to find in which nixpkgs commit it is present, and then add another pkgs source with that commit and then use that for that package. Scale this to 15 or 30 apps and you have a mess to deal with.

I am hoping for something like package.json, where I specify the version I’m interested on, and the pkg manager takes care of fetching it and adding it to my project/system

[woile]

I think this is one of the reasons for flakes to exist. You can consume flake packages from independent modules, and update flakes independently.

they provide a uniform structure for Nix projects, allow for pinning specific versions of each dependencies, and sharing these dependencies via lock files, and overall make it more convenient to write reproducible Nix expressions.

Flakes - Official NixOS Wiki

[gamebeaker]

For packages.nix that i develop/ test i make it like this:

1. create a subfolder in the same dir with the package name
2. in your case the package.nix file already exists so you can just copy this file from github. You can search here: NixOS Search the package and than click on source to copy the relevant files (don’t forget other files like patches etc. or just copy the whole folder.) You can use the github history feature to show all commits to this file if you want a specific version of the program.
3. add (pkgs.callPackage ./<your package name>/package.nix { }) to your packages

The problem is that the inputs aren’t pinned for this package.nix so should a breaking change be introduced to the inputs it could still break.

To pin the input version of the package you could wrap it with a flake for this specific package.
That doesn’t really makes it simpler it just offloads the pinning to a sub dir so your main config doesn’t get so large.

[withakay]

It feels like you misunderstand nix’s value proposition and methodology.

nix alleviates certain reproducibility concerns by defining each package (in nix terms, “derivation”, which can deliver binaries but also other files) in terms of it’s dependencies (in nix terms, “inputs”). That means that pkg-3.2 that depends on foo-1.0 that depends on baz-0.5 is not the same as pkg-3.2 that depends on foo-1.0 that depends on baz-0.6. If any dependency changes, it is a new package from nix’s standpoint. Knowing that a given /nix/store/<hash>-<pkgname>-<ver> specifies it’s dependency chain like this is the core value proposition of nix. pkg-3.2 is not a useful specification from the perspective of nix.

If you change any dependency, then the downstream derivations will change, requiring them to be rebuilt. That is the cost of the input-addressed method.

nix is a source-based distribution, but it has a binary cache which allows certain builds to be substituted instead, improving convenience and bit-for-bit style reproducibility. It’s just not possible to build every combination of every possible dependency and resulting downstream packages. The build and storage for that would scale exponentially. Therefore, a tractable approach is to have specific versions that get evaluated and built together, then cached. It’s an imperfect solution, but one that scales (better).

You really can’t compare TypeScript lockfiles at all - a single dynamic language is not a good comparison for a way to do something similar with packages written in any possible language and compiled, linked, and resolved in multiple different ways.

I get that you may not like the compromise that nix and specifically nixpkgs makes in this regard, but you have the freedom to override definitions and build yourself if you want to. Take for comparison Arch’s pacman, and getting old versions is hard enough – and completely forget trying to specify a group of system packages easily and ensuring they don’t conflict with other packages on your system. Arch is another system that also does not really believe in partial upgrades, but the difference is that they tradeoff ability to have input-addressed reproducibility and concurrent deployments of conflicting packages for ability to quickly deliver packages and reduced build/storage costs. It’s a reasonable trade, depending on your needs. Every technology is a series of tradeoffs for use case. If nix serves your use case, great! If not, also totally cool, and I hope you do find a suitable technical solution for your problem domain.

I’ll also say that I prefer to keep my system dependencies pretty lean. Instead I write project-specific shells and packaging for specific projects. When I stop working on that project, I’m fine with the /nix/store files to be garbage collected, knowing that the definition files will still allow me to recreate the dev environment or project-dependencies in the future.

[juipeltje]

For me personally i only worry about it when a package won’t build and then i pin it through a flake input. Otherwise i trust the distro and the packages they ship. It does make me curious though since this sounds more like a trust issue, how did you deal with this on other distros? Cause as far as i’m aware most distros advice to update your entire system and not to do partial updates, which is what you would be doing if you only update specific packages.

[danielo515]

@gamebeaker

• create a subfolder in the same dir with the package name
• in your case the package.nix file already exists so you can just copy this file from github. You can search here: NixOS Search the package and than click on source to copy the relevant files (don’t forget other files like patches etc. or just copy the whole folder.) You can use the github history feature to show all commits to this file if you want a specific version of the program.
• add (pkgs.callPackage ./<your package name>/package.nix { }) to your packages

This reads a lot like vendoring a package, right? I guess the package definition will include the version or git commit hash that I can change to fit my desired version. Correct?

You’re right that this doesn’t make it any more ergonomic but gives me more control

[danielo515]

@juipeltje

Cause as far as i’m aware most distros advice to update your entire system and not to do partial updates, which is what you would be doing if you only update specific packages.

I think I’m making things more confusing by using the term packages.
I use nix to define my user configuration, programs and settings in my Mac.
What I usually want to update are the apps, but a lot of those apps are just CLIs that I consume as nix packages, that’s why I used the term package.

@withakay

I’ll also say that I prefer to keep my system dependencies pretty lean. Instead I write project-specific shells and packaging for specific projects.

I do the same, but there are certain “dependencies” that I want to use system wide. For example atuin (a terminal history manager whose latest version is in fact what drove me here), exa as a replacement for ls and some other apps that I always want on all my MAC systems to be installed. Everything else is project specific, even git.

@withakay

You really can’t compare TypeScript lockfiles at all - a single dynamic language is not a good comparison for a way to do something similar with packages written in any possible language and compiled, linked, and resolved in multiple different ways.

True. I should have compared it to Brew. I really wanted nix to be a declarative, reproducible alternative to brew, but it is clearly not.

Thank you for explaining in such detail what the value proposition of Nix is. I kind of have that same idea in mind, but it is always good to read different ways of expressing the same concept. I’m the kind of guy that wants to use the tool he likes for everything, and a lot of my problems just arise from that.

[withakay]

@danielo515

True. I should have compared it to Brew. I really wanted nix to be a declarative, reproducible alternative to brew, but it is clearly not.

I do think it is a pretty good declarative, reproducible alternative to brew, it’s just that declarative + “input-address” reproducible are not a zero-cost abstraction. I’m not really sure how to reliably implement on a way to produce what you seem to want: a declarative, reproducible package manager that not only allows you pin/upgrade individual packages but also has a community binary cache for all of those.

We know it’s a pain point, especially the build cost as nixpkgs continues to grow. I’m sure the community would be interested in tractable ideas if you think there’s a better architecture to explore for this.

@danielo515

What I usually want to update are the apps, but a lot of those apps are just CLIs that I consume as nix packages, that’s why I used the term package.

How long does it take to compile the CLIs? I’m sure some can be quite lengthy, but I assume most would be relatively quick builds and I’d probably just use a .overrideAttrs for this and eat the build cost.

[drakon]

You also don’t need Flakes to achieve this. npins, lon, and tack, also all achieve this.

[whispers]

in the same vein as what withakay was saying, it’s worth keeping in mind that Nixpkgs is not (though it does a decent job at pretending to be) a collection of independently built packages, in the way that some other distros are. there is no such thing as a definition of one package in isolation; it’s a heavily interlinked and monolithic codebase. it’s sort of getting at the reason Nixpkgs is a monorepo, instead of one repository for each package; the functions and definitions used in the code are shared widely throughout the project, and updating them in sync is a huge part of how Nixpkgs works. when Nixpkgs e.g. updates glibc, the reason ≈every package is rebuilt is because (to the Nix evaluator), the fundamental definition of what that package is has changed. updates to one part of the codebase propagate immediately to other parts. thus, it doesn’t really make sense as a concept for Nixpkgs to distribute itself as individual packages, because that’s not what it is.

[phaeseeKe5Ee]

Is devenv what you looking for? Its for project, not the whole system though.

[phaeseeKe5Ee]

Have you tried overlay the package with the src and version you want?
e.g. Overriding a version, just override the src and version attribute to anything you want.

nurl is a helper to speed up this process.