How do nix profiles and flakes fit together?

[woile]

I’ve been using nix profile install on Mac for some time already. Mostly to replace brew. I created a package along the way that helped me remember the command and as a way of learning. I got a question recently that I haven’t thought about. How do I make the packages I install through nix profile install reproducible?

The only thing I see from profiles is a manifest.json. Shouldn’t they be in a .nix file? Can someone help me make sense of how are these things supposed to behave together?

Thanks

[TLATER]

Here’s the documentation for nix profiles: nix profile - Nix Reference Manual

All of the profile contents are in the nix store, as you see, so they’re read only. There’s no way to replicate a profile from those paths yet to my knowledge, and I don’t really think there should be either.

You could try home-manager, or an approach similar to the old packageOverrides, i.e. writing a flake with a single package that just has all the ones you want combined, and installing that package instead.

[arianvp]

Why shouldn’t there be? Turning an existing profile manifest into a flake sounds like a trivial mapping and quite useful. Maybe “nix profile freeze” command?

[abathur]

I have thought, broadly, that a good share of Nix users would gain leverage from having more imperative → declarative workflows. (But I’m also curious why you think there shouldn’t be, @tlater?)

It seems to me like there’d be some general leverage for everyone, but especially for newer users facing the blank-page problem?

Cross-linking some related issues:

• Option to create shell.nix from nix-shell arguments · Issue #1967 · NixOS/nix · GitHub
• Imperative package management as a high-level interface for a declarative format · Issue #4482 · NixOS/nix · GitHub

[TLATER]

Just feels a bit backwards to me to create a declarative configuration from an imperative one, but that’s the point I suppose.

I’ve not looked at it from @abathur 's perspective somehow, maybe I should just look past my displeasure with anything imperative

It would be nice if it created a proper flake at least, looks like the manifest contains enough data to actually create the inputs set, guess it would just take a new type of output?

[abathur]

Hehe. It’s fair, and I don’t even really disagree. I’ve certainly wished that nix-env wasn’t the first recommendation many people encounter when they’re trying to figure out how to install stuff. (Link is a comment in a thread I made a while back about how Nix’s affordances map to language and how that affects new users flailing around to figure out how their human motivations map onto the toolkit.)

As long as we’ve got these imperative workflows, I think it’d be nice for them to readily flow into “best practice” whenever possible.

I also suspect it would be a win for the “learning journey” (cc @zmitchell) if every “imperative” tutorial and guide could end with a final step or a link out to how you could ~lift the imperative work you just did into a declarative format.

[TLATER]

Maybe we should start a #dev thread about this

@woile, since I’ve actually made it to a computer now, here’s an example for a flake.nix that would work in the mean time:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11"; # Or just `nixpkgs` if you want to use the registry
  };

  outputs.packages."x86_64-darwin".default = let
    pkgs = nixpkgs.legacyPackages."x86_64-darwin";
  in pkgs.buildEnv {
    name = "my-packages";
    paths = with pkgs; [
      aspell
      bc
      coreutils
      gdb
      ffmpeg
      nixUnstable
      emscripten
      jq
      nox
      silver-searcher
    ];
  };
}

You can then just install it with nix profile install .. If you put it on github or such, you can even install it with e.g. nix profile install github:woile/packages.

Obviously this isn’t the automated freeze we’re talking about, so you’d need to manually pick out all the packages you’ve installed so far.

It’s also not exactly the same thing, since you’re now installing this combined package, instead of all the packages individually, which is why I think flakes would need a new type of output to support a hypothetical freeze command.

[woile]

I think that could be a good idea, to add the packages to a flake, and then install the flake.
Though it would probably add git as a dep to keep track of the hash, right?

We can see a lot of tools that do something like this: npm, poetry, cargo, etc. I think it’s a welcoming pattern. And in Mac I haven’t seen a way to declaratively install deps.

I’ve been working on this to help move away from brew on Mac, and if I can put your deps in a flake.nix file, once people get familiar with nix, they would already have all their OS deps in a flake.

GitHub - woile/npt: Nix Package Tool. A (humble) successor to linux's apt,...

[TLATER]

@woile

I think that could be a good idea, to add the packages to a flake, and then install the flake.
Though it would probably add git as a dep to keep track of the hash, right?

Not necessarily; nix keeps track of hashes without git as well (by naively hashing the whole directory, if I’m not mistaken). If you want to use git to track your flake’s history that’s up to you.

Nix just has support for using git repositories as input, via the various git protocols. It does indeed shell out to git for these features, but I don’t think it’s a mandatory dependency - assuming you can live without git inputs, which I don’t think is realistic

Note that if you do use git, nix does use information from it to track versions, as your flake will implicitly be treated as a git input. So if you use git you need to make sure you git add everything you intend to be accessible from the flake.

npm/poetry/cargo etc. also don’t directly interface with git for version tracking, by the way. All of them, including nix, operate on “lock” files, which contain the hashes of all your inputs to ensure correct input versions. These are typically decoupled from git, because ultimately svn and other version control systems are still out there - and tarballs and such need to be usable as inputs as well. Unlike nix, those package managers make the lock file optional though.

@woile

I’ve been working on this to help move away from brew on Mac, and if I can put your deps in a flake.nix file, once people get familiar with nix, they would already have all their OS deps in a flake.

You might be interested in some of the stuff @vlinkz has been doing: GitHub - snowfallorg/nix-software-center: A simple gtk4/libadwaita software center to easily install and manage nix packages - and in particular, GitHub - snowfallorg/snow: A commandline wrapper for the Nix package manager, which has very similar goals to you I think.

They’re spearheading the muggle-friendly-nixos initiative - some MacOS presence in the area might be welcome: https://snowflakeos.org/

[alper]

@TLATER

You can then just install it with nix profile install . . If you put it on github or such, you can even install it with e.g. nix profile install github:woile/packages .

And if you add/remove something in the file, do you have to run the install command again and it’ll be updated?

@TLATER

You could try home-manager

I looked at this but didn’t have the stomach for it.

[TLATER]

@alper

And if you add/remove something in the file, do you have to run the install command again and it’ll be updated?

So, if you run nix profile install . for the first time, nix will download the latest version of the nixos-22.11 branch from https://github.com/nixos/nixpkgs, and write its hash into another file it creates called flake.lock. Then it will use the definitions from the repository to compute what to download from hydra (or build if not available yet) and finally install for you.

If you run the command again, it will see that you already have the packages installed, because it will download the exact same packages as the first time you ran the command, because flake.lock will point to the hash of the version that you first installed (and in fact, the repository will likely be cached locally).

If you actually wanted to update your packages, you would need to first update the hash in flake.lock. You do this by just running nix flake update, which will repeat the first part of the installation process.

And indeed, after that you can install the new versions of the packages with another nix profile install ..

In this case I think nix profile upgrade might also work, after you update the lockfile. I’m not 100% sure though, it might even work independent of the lockfile and be a bit of a mess. I’ve not used nix profile much myself.

@alper

@TLATER

You could try home-manager

I looked at this but didn’t have the stomach for it.

That’s a shame! It’s less complex than it looks, and if you have a use case for actually installing configuration as well as packages, incredibly useful.

Maybe try this suggestion for a bit, and when/if you get comfortable with it I’d be happy to run you through some setup for home-manager. It’s basically just a fancier version of the pkgs.buildEnv function

[NobbZ]

Personally I think adding import and export capabilities to a nix profile would massively benefit the ecosystem.

In my opinion this is not related to a transition from/to declarative, this is just something that other package managers support for quite a while…

Create a list of package names, which at some point in time have been installed, to be able to recreate the system “best effort” after troubleshooting.

nix profile could make this even better, as it doesn’t only know which packages have been installed, but it knows exactly how they have been installed and from where, down to the full dependency chain (through the source flakes exact commit).

So at the end, this is just some means of “backup” flow, that some poeple are more used to than to other flows.

As much as I personally dislike this, but we have to face the truth, nix is used as an imperative and rootless package manager a lot on the Mac and on some linuxes, to get fresh and up to date packages on the “stable” distros that don’t bump versions for decades.

Not everyone can or want use HM, darwin or the recent addition of system manager. This is something we have to accept. And we have to do some efforts to accomodate them.

[alper]

@TLATER

That’s a shame! It’s less complex than it looks, and if you have a use case for actually installing configuration as well as packages, incredibly useful.

It’s not the complexity. It’s the awkward abstraction together with “Words of warning” and the fact that there is no good documentation on it. (This isn’t it.)

I’ll note it down but at this point (and probably for the next years) it seems smarter to use a minimal homebrew to manage global packages and have a zero chance of anything getting messed up and then using flakes to create specialized shells for projects.

[TLATER]

Yeah, the words of warning are a bit of a thorn in my side. They’re somewhat overly dramatic, but to be fair, lots of people with no programming experience, let alone nix experience, try out home-manager and get confused. While the things the warning talks about are true, it fails to explain why.

I don’t think you’ll struggle if you actually understand how your configuration works, and so far you seem like the kind of person who does.

So important caveats to the words of warning:

• I don’t think the advice of reading the nix pills is good. They’re notoriously complex and don’t explain the actual ecosystem, so good luck new users.
• home-manager can’t always detect if it will overwrite something. This is very uncommon though, only really applies to database-based configs like dconf (which isn’t a thing on MacOS afaik), and you’re in full control - if you know how the application stores its configuration this should be transparent to you.
  • When in doubt, you can look at the module implementations to see if they’re doing something odd. And even then, the module authors do their best not to mess with things
  • If you still don’t trust the configuration modules, none of this code runs if you don’t enable them, so you should not need to fear other applications breaking. Though if you’re using home-manager only to manage packages just using a flake like I described is probably better.
• The warning about incompatibility is mostly targeted at the situations in which some non-user owned things interact with the user-owned ones. Think X11 or graphics drivers.
• You can do rollbacks, you just don’t have the support of the home-manager cli tool for them. I think you only rarely need rollbacks for home config too, git works fine. They’re only for when git stopped working.

Generally I find home-manager works great for TUI tools, and a bit less well for GUI ones.

It doesn’t just thrash about on the system without me asking, so I have never experienced “messing things up” - just a config that didn’t end up working because of, say, graphics drivers not playing ball. It’s easy to just fall back to a non-nix package for those situations (or find the workaround).

The advice of starting small, and incrementally adding more things to it, is the best advice given in the documentation. If you go application-by-application, you’ll have full control over what’s happening, and be able to experiment as you wish.

And again, you don’t need to touch the configuration modules. For complex applications, just adding them to home.packages has zero risk as well.

@alper

there is no good documentation on it. (This isn’t it. )

This I’m less sure about. Sure, the whole nix ecosystem struggles with documentation - far too few of us are good at writing things targeted at various levels of experience, and there are still far too few of us to make up for that with numbers.

But the home-manager docs are pretty ok IMO. I suspect you’re not using nix-darwin, so you want the standalone setup, maybe that’s tripping you up?

[woile]

I think I’m gonna try to make npt add the deps to a flake, somewhere in the user’s config folder. And then install that flake. Maybe even later the flake can support even a shell

[alper]

@NobbZ

Personally I think adding import and export capabilities to a nix profile would massively benefit the ecosystem.

I’m just installing tools globally using nix profile install right now (this has somehow broken nix-env). It’s working great!

Not sure where the flake.lock is now? I’ll figure out how to upgrade these tools later at some point.

[NobbZ]

@alper

I’m just installing tools globally using nix profile install right now (this has somehow broken nix-env ). It’s working great!

It didn’t break it, its just that nix-env and nix profile are icompatible to each other. You can not manage a single profile using both tools, you can have one with nix-env and one with nix profile though.

@alper

Not sure where the flake.lock is now?

Which one? Usually its in your flake…

[alper]

@TLATER

So, if you run nix profile install . for the first time, nix will download the latest version of the nixos-22.11 branch from GitHub - NixOS/nixpkgs: Nix Packages collection, and write its hash into another file it creates called flake.lock . Then it will use the definitions from the repository to compute what to download from hydra (or build if not available yet) and finally install for you.

Where is this flake.lock? I should probably never touch it?

[NobbZ]

@alper

Where is this flake.lock?

In the flake.

In the case of nix profile install ., I’d assume to find it in whatever folder you are in when you run that command. Unless the flake you install from doesn’t have any inputs, then it doesn’t need a flake.lock, this is true for nixpkgs.

@alper

I should probably never touch it?

At least not manually. The primary means of interacting with a flakes lock file would be nix flake update and nix flake lock.

[dcecile]

Thanks for the discussion & instructions! I was looking for a way to set up reproducible profile packages, and this seems perfect.

I tried to use the flake definition above (using Nix 2.15.0), but got…

error: undefined variable 'nixpkgs'

       at /nix/store/bvn7x0wzpg3hzfyfvj7rclhsmqx98wjb-source/flake.nix:7:12:

            6|   outputs.packages."x86_64-linux".default = let
            7|     pkgs = nixpkgs.legacyPackages."x86_64-linux";

So here’s the flake.nix I’m using instead:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
  };


  outputs = { self, nixpkgs }: {
    packages."x86_64-linux".default = let
      pkgs = nixpkgs.legacyPackages."x86_64-linux";
    in pkgs.buildEnv {
      name = "home-packages";
      paths = with pkgs; [
        gping
      ];
    };
  };

}

To install the flake to my profile, I used nix profile install . (as suggested, no issues).

After editing flake.nix, all I need to do to apply the changes is nix profile upgrade 0 (this is the 0 entry in my nix profile list).

Is it okay that the flake is using legacyPackages, or would be recommended to use a different API?