Maybe this post should be an RFC, but I just have to collect my thoughts on this first. Feedback is appreciated. Does this idea have merit, or am I missing something big that makes this infeasible?
So I’ve been reading a lot through the issue tracker, particularly the issues on nix profile
UX. There were some interesting ideas in there, but my overall conclusion was that most of them were mostly bandaids on system that just wasn’t that well designed to solve issues that were more basic.
I touched upon some issues around moving to declarative profiles myself, and I think there might be a solution here that was already started by @bobvanderlinden in `nix profile` pins nixpkgs per package · Issue #7964 · NixOS/nix · GitHub, but I’d like some broader discussion on it as it’s a big change.
Moving from manifest.json to a flake
Just to get this out the way, manifest.json
will still be there for backwards compatibility.
The basic idea is that there is a basic flake that represents an empty profile:
{
inputs = {
nixpkgs = "xx-replace-nixpkgs-xx"
};
outputs = { self, nixpkgs }:
let
profilePackages =
{ }
// (with nixpkgs; { })
// ({ /*more-inputs*/ });
in {
packages."system-tuple" =
profilePackages // {
default = nixpkgs.buildEnv {
name = "user-profile";
paths = builtins.attrValues profilePackages;
};
};
};
}
For this discussion, the system tuple is inserted during the creation of the flake, but is not an input to the flake.
When the user runs nix profile install
for the first time (i.e. there is no profile-link
yet), this will be the basis to create the 0th profile generation as follows.
The algorithm
- Create a temporary folder and insert the basic flake from above into it
- Replace “system-tuple” with “aarch64-darwin”, “x86_64-linux” etc. depending on the system
- Replace “xx-replace-nixpkgs-xx” with whatever
flake:nixpkgs
resolves to (“github:NixOS/nixpkgs/nixpkgs-unstable” in the default global registry) - Lock the flake as with
nix flake lock --update-input nixpkgs
- Add the locked flake to the store with
nix store add-path
- Let
profile-0-flake-path
be the resulting store path
- Let
- Install the profile as you would with
nix profile install path:$profile-0-flake-path
, but instead of linking it asprofile-1-link
, link it asprofile-0-link
.
This gives us a basic profile that has no packages installed to it, with a locked nixpkgs input and a manifest.json
that enables full backwards compatibility with the current tools.
Now, we proceed with the regular installation of the package. This will be the exact same for all future calls to nix profile install
as well (except for the id of the generation);
- Create a temporary folder again and copy everything from
profile-flake-path
into it - Separate installable like so: For
nixpkgs#git
,flake
isnixpkgs
,attrPath
isgit
. - If
flake
is already in the inputs, go to step 14. Otherwise: - Add whatever
flake
resolves to to the inputs - Modify the flake by replacing
// ({ /*more-inputs*/ });
with
where// (with $input-name; { }) // ({ /*more-inputs*/ });
$input-name
is replaced with the name that was chosen for the input flake - Lock the flake as with
nix flake lock --update-input $flake
; - Modify the flake by adding the new attribute path to the correct set, inheriting from
legacyPackages
orpackages
. Fornixpkgs#git
on an M1 Mac, that is a string replacement of// (with nixpkgs; { })
with
or of// (with nixpkgs; { inherit (legacyPackages."aarch64-darwin") git ;})
// (with nixpkgs; { inherit (legacyPackages."aarch64-darwin")
with// (with nixpkgs; { inherit (legacyPackages."aarch64-darwin") git
- Add the locked flake to the store with
nix store add-path
- Let
profile-new-flake-path
be the resulting store path
- Let
- Let the current profiles flake path be
profile-old-flake-path
- Remove the flake
path:$profile-old-flake-path
and installpath:$profile-new-flake-path
in a single operation
We now have a profile that is mostly backwards compatible with the current system, except that every input is only linked once. The generations are there as symlinks like before, the manifest.json
is there (though it would contain less information), so rollbacks, list
, history
, all of that works like before. You wouldn’t be able to upgrade
as the installed flake never changes, but that’s the only limitation in backwards compatibilitiy I can think of.
What this allows
We can now solve a number of UX issues with nix profile
:
#7964, nix profile
pins nixpkgs per package
As nixpkgs is now locked, additional calls like nix profile install nixpkgs#just
will only add an attribute to the flake. There’s no implicit update and no re-downloading of the whole of nixpkgs. Explicit upgrading can still be made possible, but the algorithm for that is out-of-scope of this post. There’s also the question whether inputs of flakes should automatically follow existing inputs. This might be desirable, but I’m not sure about potential downsides.
#1807, First nix-env invocation is not reversible
We now have a 0th profile generation with an empty profile that you can always go back to and start anew from without having to manually create a new profile, installing a package into it and then manually creating symlinks in your old profile.
#7965, profiles are hard to reproduce
You now have access to the flake that built the profile and can just pull it out from the nix-store. A simple nix profile import
command that skips most of the steps of install
and just executes the installation step directly would be a potential import mechanism.
#5587, nix profile
allows installing duplicate packages
Through the semantics inherit
, trying to install a duplicate package will fail. The semantics of //
ensure there is always one clear candidate for a package, even if there are two attributes with identical names in different flakes. Which one is chosen depends on the order in which the flakes were added to the input, which should be fine in most cases, but using a union function that fails on duplicate keys would be a simple solution. Maybe there is one in builtins
or lib
already?
#7967, nix profile
entries need stable identifiers
The identifiers are the names of all the flake outputs, minus default
. These are guaranteed to be unique. This in turn solves #7960 and #7961, and also allows solving the next issue:
#7962, nix profile
counter-intuitively uses regexp to match packages
Instead of matching against the full attribute path of an installable, match against the identifier. In most cases, this will be sufficient, and the logic doesn’t need an extra case for regex.
nix profile list
is not helpful when installing from a flake with pkgs.buildEnv
I didn’t create an issue for this yet, but during my experiments with a declarative profile, I found that while using a flake as the basis for your profile is a great strategy, it’s not supported well by the current tools. Making this the default approach for profiles would allow us to change nix profile list
to recurse into the profile flake and list all the installed packages.
We can create a proper transition flow from imperative to declarative profiles
This came up in the discussion below. Right now, most of us agree that just using nix-env
for managing your packages is suboptimal, but in user profiles, there’s no good alternative. With the flake-based profiles, we can implement a nix profile edit
command, that basically copies the current flake from the store to a temporary directory, runs nix edit
on it, allows you to edit the flake, and then proceeds like install usually would.
This makes it very easy to try out declarative package management, and once you’re comfortable with it, you can copy the flake from the nix store and version-control it yourself.
What next
I feel like this could be implemented as a basic prototype with string replacement and calling existing nix commands in a basic bash or python script. I’ll try to do that soon, maybe I’ll find time on the weekend, but for now I’d just like to hear your thoughts on this. It seems like an awesome idea, but also a little too good to be true. What did I miss?