Transitioning from imperative to declarative package management with nix alone

iFreilicht · June 3, 2023, 11:13pm

I know we want to move people away from just installing packages with nix-env or nix-profile, and so I was searching for a good stop-gap before moving to something like home-manager that just used first-party tooling. In the process, I found that we have some disparate documentation around setting up simple declarative package management and none of it is satisfyingly easy to understand.

Right now, there really seems to be no good way of maintaining a declarative profile, period. All the options have significant drawbacks in some way.

I collected my research and the options I found, hopefully they will be useful for driving discussion about how to continue this topic.

This is a very long post. If you have any idea how I could format it better, please let me know.

Only using `nix-env -i` / `nix profile install`

This is what most users start with and what we want them to move away from.

$ nix profile install direnv
$ nix profile install asdf-vm
$ nix profile install grc
$ nix profile install autojump

Disadvantages

Not declarative.

Advantages

Well-documented, easy to understand

The existing documentation only covers nix-env, but I would offer to write an additional version that explains the same procedures for nix profile. They are quite similar in this imperative use-case.

Works reliably, no surprises

Every added package creates a new profile generation and you can roll back to your heart’s content.

Uninstalling a package is easy and only uninstalls a single package, just like you would expect.

$ nix profile remove '.*grc' # or nix-env -e grc

Side note, the new regex syntax really needs an example in the docs. Right now it looks like ‘.*’ is a special case. I only found out it wasn’t during this research accidentally.

`nix profile list` actually does what you expect:

0 flake:nixpkgs#legacyPackages.aarch64-darwin.direnv github:NixOS/nixpkgs/ac5281cce24fac6d98773c55c038f654b8e84cd4#legacyPackages.aarch64-darwin.direnv /nix/store/7gkqw1267ccc9lvfjx3ajgmqf6mq8xhp-direnv-2.32.2
1 flake:nixpkgs#legacyPackages.aarch64-darwin.asdf-vm github:NixOS/nixpkgs/ac5281cce24fac6d98773c55c038f654b8e84cd4#legacyPackages.aarch64-darwin.asdf-vm /nix/store/hymd5xkvyjr76ppw3pwf2plnzyp0yspa-asdf-vm-0.11.3
2 flake:nixpkgs#legacyPackages.aarch64-darwin.grc github:NixOS/nixpkgs/ac5281cce24fac6d98773c55c038f654b8e84cd4#legacyPackages.aarch64-darwin.grc /nix/store/1mknlhg6ymbqwnf22s6zf3q7p5abp68p-grc-1.13
3 flake:nixpkgs#legacyPackages.aarch64-darwin.autojump github:NixOS/nixpkgs/ac5281cce24fac6d98773c55c038f654b8e84cd4#legacyPackages.aarch64-darwin.autojump /nix/store/z45mi3fbwwnwrqkdrzdj1ysc8f04pxw8-autojump-22.5.3

It lists every package that you installed, just like the docs claim. Marvelous.

Upgrades are simple

nix profile upgrade '.*' # or nix-env --upgrade

Using `nix-env -if` / `nix profile install -f`

I never saw this documented anywhere, but it seems like the simplest way to write and install a declarative profile is to use the --file option with a file containing a list of packages:

# tools.nix
{ pkgs ? import <nixpkgs> {} }:
with pkgs; [
    direnv
    asdf-vm
    grc
    autojump
]

$ nix profile install -f tools.nix # or nix-env -if tools.nix

Advantages

Easy to understand, low on syntax

Just one file with a list of packages, one command to install them all. You can basically discard the first two lines as small boilerplate if you just want to get started right away.

`nix profile list` still works. Somewhat.

It’s not quite as nice, but at least you can still see all the packages:

0 - - /nix/store/53k1jp9wks4psq0yii98cm4ia81md94f-grc-1.13 /nix/store/88yzfgm6aan038qk1j0n81zkypp4vac3-autojump-22.5.3 /nix/store/n7hb3zmgk9d3ggv57xm0ngi79pxx32f1-direnv-2.31.0 /nix/store/rsk4xgdx8xdfi36mg4gxp3gq6r1g0fli-asdf-vm-0.9.0

This does become quite unwieldy with large numbers of packages.

`nix profile remove` is surprisingly concise

Quickly glossing over the fact that `nix profile remove ‘.*’ doesn’t work anymore:

$ nix profile remove 0
removing 'asdf-vm, autojump, direnv, grc'

It seems like there is some implicit conversion from a list of derivations to a single derivation built into nix?

Disadvantages

Implicitly switches new CLI back to legacyPackages

The output of nix profile list shows that the packages are not installed as flakes anymore. This breaks updates:

$ nix profile upgrade '.*'
warning: '.*' does not match any packages
warning: Use 'nix profile list' to see the current profile.
$ nix profile upgrade 0
warning: '0' is not a valid index
warning: Use 'nix profile list' to see the current profile.

More generally, it breaks the installables matching.

Installing the same file multiple times doesn’t result in an error

This means removing a package from the file and running nix profile install again will not result in the package being removed! It will also never be garbage collected.

A big gotcha.

Actually still imperative → upgrades are weird

To upgrade, you will have to nix-channel --update, nix profile remove <number> and then nix profile install -f tools.nix again, otherwise you get errors like

error: packages '/nix/store/rsk4xgdx8xdfi36mg4gxp3gq6r1g0fli-asdf-vm-0.9.0/bin/asdf' and '/nix/store/020v04f7d5h92sm3md5v8ysivrnvr2iv-asdf-vm-0.9.0/bin/asdf' have the same priority 5; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' to change the priority of one of the conflicting packages (0 being the highest priority)

Using `pkgs.buildEnv`

This is what the nixpkgs the nixpkgs docs recommend, but there’s multiple ways of achieving the same thing. For simplicity, let’s use a separate file like above:

# myEnv.nix
{ pkgs ? import <nixpkgs> {} }:
with pkgs; buildEnv {
  name = "my-env";
  paths = [
    direnv
    asdf-vm
    grc
    autojump
];
}

$ nix profile install -f myEnv.nix

This works exactly as it did before. I don’t see any advantages in this.

Disadvantages

Everything from the previous “list of packages” approach, plus:

`nix profile list` doesn’t work as expected anymore

It will only output one package now:

0 - - /nix/store/n4gl63cd4y2x7sn1c9cbhdrkjkm2gw5a-my-env

You can come up with a bash oneliner to find the packages, sure:

$ nix show-derivation $(rg "/nix/store/n4gl63cd4y2x7sn1c9cbhdrkjkm2gw5a-my-env" /nix/store -l --max-depth 1 2>/dev/null) | jq -r '.[] | .env.pkgs' | jq -r '.[] | .paths | .[]'
/nix/store/n7hb3zmgk9d3ggv57xm0ngi79pxx32f1-direnv-2.31.0
/nix/store/rsk4xgdx8xdfi36mg4gxp3gq6r1g0fli-asdf-vm-0.9.0
/nix/store/53k1jp9wks4psq0yii98cm4ia81md94f-grc-1.13
/nix/store/88yzfgm6aan038qk1j0n81zkypp4vac3-autojump-22.5.3

But that is not user-friendly at all.

Putting `pkgs.buildEnv` into a flake

I haven’t seen this documented explicitly anywhere either, but it seems like the best solution right now.

# my-env/flake.nix
{
  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
    in {
      packages.${system}.default = pkgs.buildEnv {
        name = "my-env";
        paths = with pkgs; [ direnv asdf-vm grc autojump ];
      };
    };
}

Then install with

$ nix profile install path:/home/my-user/my-env

Alternatively, one could also use nix flake init and git add to create a git-based flake, which avoids one of the gotchas below.

Advantages

Works reliably, enforces reproducibility

This is mostly caused by flakes themselves, but still good.

Updates work very well

If you make any change to the file, just run nix profile upgrade '.*'.

Disadvantages

Still no error when installing a flake multiple times

I guess this is kind-of by design in nix, as you absolutely can install multiple versions of a derivation side-by-side, but it still feels icky that you can make that basic mistake.
This might be somewhat solvable by having nix profile upgrade '.*' shown in the beginner’s guide.

More boilerplate

This is also somewhat inherent to flakes themselves and probably unavoidable. Especially when you start adding flake-utils for compatibility with all systems, this becomes a lot.

`nix profile list` is still unhelpful

0 path:/home/my-user/my-env#packages.aarch64-darwin.default path:/home/my-user/my-env?narHash=sha256-GjhHsdw+357Uq8ELDRdGF6Rou%2fh5zUd2hFYiwor52pI=#packages.aarch64-darwin.default /nix/store/yvm01clbiwy8bw7mywaipds517q1bgnc-my-env

This is of course expected as we just installed the one package and the others were just dependencies of it, but it’s still a bit of an issue.
However, as the exact path of the flake is now contained within the output, one can easily find the file and even the exact version using the locked URL:

$ nix flake metadata 'path:/Users/feuh/.dotfiles?narHash=sha256-GjhHsdw+357Uq8ELDRdGF6Rou%2fh5zUd2hFYiwor52pI=' --json | jq -r '.path'
/nix/store/pvdr7mp0r36kah63wsqvikbbyba153zp-source

So that’s a lot better!

Requires flakes

Flakes are great, but there’s no definitive roadmap to stabilization yet. Primary resources should probably not encourage the use of experimental features, though they absolutely should document them properly and guide the user in their adoption.

Final remarks

This final solution was the result of a lot of research, trial, and error.
It is the best one I’ve seen so far, but it’s not properly documented anywhere.

I know that we want to move people away from nix-env and nix profile somewhat, and I believe this could be a good way of transitioning from imperative to declarative package management before going all-in and managing your entire OS or home folder with nix.

For resources, I only used the official NixOS manual, the nixpkgs manual, and fastthanlime’s fantastic articles on flakes. Thanks to everyone who is working on those!

iFreilicht · June 5, 2023, 9:45am

I realize I forgot to add the caveat to the flakes method I was talking about:

Upgrades won’t work when using a relative path

This is somewhat inherent to flakes as they’re implemented right now.

In my first attempt, I just used nix profile install path:.dotfiles to install the flake. I use a bare repository to store my dotfiles, so I can’t use the regular .dotfiles installable, as this is expanded to git+file:///home/my-user/.dotfiles, which fails, as the flake commands do not support setting a git worktree manually. (Which is fair IMO, this is pretty obscure functionality.)

It seems like the docs incorrectly indicate that this . or any other relative path for that matter would be interpreted as path:/home/my-user/.dotfiles. Not sure if that would be the better behavior. I assume that’s how it was handled in the past and it was changed for a good reason.

Additionally, as the docs explain:

path generally must be an absolute path. However, on the command line, it can be a relative path (e.g. . or ./foo ) which is interpreted as relative to the current directory.

However, relative paths are not automatically normalized, so using path:.dotfiles as an installable works only in nix profile install, but subsequent calls to nix profile upgrade '.*' will fail:

error: cannot fetch input 'path:.dotfiles' because it uses a relative path.

So you have to normalize it yourself when calling install:

$ nix profile install path:$PWD/.dotfiles  # Now expands to an absolute path
$ nix profile upgrade '.*'  # Now works and returns 0

This is a bit of a gotcha. I guess this could be resolved on the nix side by always normalizing paths in path: installables. It’s already done this way for git: installables.

waffle8946 · December 11, 2024, 1:40pm

Not sure if you saw this already, but figured I’d share:

gist.github.com

https://gist.github.com/lheckemann/402e61e8e53f136f239ecd8c17ab1deb

0-readme.md

# Expression for a buildEnv-based declarative user environment

This is one way of managing your user profile declaratively.

Alternatives include:
 - an attrset-based `nix-env`-based environment, installed using `nix-env -ir` rather than `nix-env --set`. LnL has an [overlay](https://gist.github.com/LnL7/570349866bb69467d0caf5cb175faa74) which shows a way of doing this.
 - [home-manager](https://github.com/rycee/home-manager), which provides NixOS-like config for your `$HOME`

Note that this is incompatible with regular imperative use of `nix-env`, e.g. `nix-env -iA nixpkgs.hello`. It has the advantage of allowing the installation of multiple outputs of the same package much better than `nix-env`'s builtin profile builder does.

This file has been truncated. show original

buildenv.nix

/*
Save this file to ~/default.nix or your preferred path (make sure to
change the location in the update-profile script if you choose a
different place).

Install using `nix-env -f ~ --set`, from then on use `update-profile`.
*/
{ pkgs ? import <nixpkgs> {}
, name ? "user-env"
}: with pkgs;

This file has been truncated. show original

iFreilicht · December 11, 2024, 9:08pm

Ah, that’s interesting, thank you! Something along these lines is also what flakey-profile does.

I like the idea of deliberately breaking the manifest.nix to prevent additional imperative changes, that’s clever. That’s possible with manifest.json as well, but of course you can’t embed as nice of an error message then.

Transitioning from imperative to declarative package management with nix alone

Only using nix-env -i / nix profile install

Disadvantages

Advantages

Well-documented, easy to understand

Works reliably, no surprises

nix profile list actually does what you expect:

Upgrades are simple

Using nix-env -if / nix profile install -f

Advantages

Easy to understand, low on syntax

nix profile list still works. Somewhat.

nix profile remove is surprisingly concise

Disadvantages

Implicitly switches new CLI back to legacyPackages

Installing the same file multiple times doesn’t result in an error

Actually still imperative → upgrades are weird

Using pkgs.buildEnv

Disadvantages

nix profile list doesn’t work as expected anymore

Putting pkgs.buildEnv into a flake

Advantages

Works reliably, enforces reproducibility

Updates work very well

Disadvantages

Still no error when installing a flake multiple times

More boilerplate

nix profile list is still unhelpful

Requires flakes

Final remarks

Upgrades won’t work when using a relative path

Only using `nix-env -i` / `nix profile install`

`nix profile list` actually does what you expect:

Using `nix-env -if` / `nix profile install -f`

`nix profile list` still works. Somewhat.

`nix profile remove` is surprisingly concise

Using `pkgs.buildEnv`

`nix profile list` doesn’t work as expected anymore

Putting `pkgs.buildEnv` into a flake

`nix profile list` is still unhelpful