Distinguishing NixOS packagement: nixpkgs vs. Flakes / Home-Manager

[enoren5]

My main question: How does NixOS track different builds of generations?

To qualify that question at length, I’ve read the NixOS official doc on the Nix Language twice and I re-read Ryan’s incredible book on Flakes a few times already. But I have some outstanding lingering questions I can’t answer based on my reading so far.

Here goes - -

If the NixOS package manager builds based on references to commit hashes instead of versioning tags, then that means there could theoretically be as many different packages as there are thousands of commits in a given application’s source code repository. I can’t possibly expect NixOS package maintainers to build, store, host, and distribute all those unique binaries on package servers, right?

I see in the searchable package repositories, versions are consistent overtime and only change with major releases that correspond to tags on GitHub.

• Does this mean that the fluid package versioning according to commit hashes is only for Home-Manager which is used more so by developers who are testing recent changes on a beta release for a project that they are writing / contributing to?
• Does this also mean that with Flakes and Home-Manager, individual packages are assembled and compiled from source like Gentoo? Does this mean that the more a NixOS admin uses Flakes and Home-Manager, maintenance more closely resembles a source based distro?

To use an example to better illustrate my point, the latest release of Hyprland is v0.37.1. It came out like last week I believe. But the current NixOS stable package channel is “23.11” and the official searchable nixpkgs repo for this channel shows Hyprland v0.32.3. It’s obsolete. If I wanted a more recent version, I could switch my NixOS package channel to ‘unstable’ which would then download and install Hyprland v0.36.0, which is still not the latest. If I really wanted bleeding edge, I would need to compile v0.37.1 from source. The best way to do that would be to use a Flakes script for Home Manager. Is this correct?

So if I wanted to generate a local repository with a comprehensive history of the last 5 major Hyprland releases, I wouldn’t be able to do that with nixpkgs. I’d have to compile all that source code for each commit hash of my choice using Flakes and Home-Manager. Then I could flip the symlinks from one generation to the next, back and forth, up and down. Of course I don’t actually have a need or purpose to handle different versions of Hyprland as such, as a novice I am merely asking to better distinguish and wrap my mind around nixpkgs, Flakes, and how NixOS package management works in general.

[Atemu]

@enoren5

If the NixOS package manager builds based on references to commit hashes instead of versioning tags

It doesn’t.

The actual version of a “package” is the hash of a canonical representation of the entire dependency tree of the package all the way back to a single root. If anything about that dependency tree changes, so does the hash but if nothing changes, neither does the hash. If you just rename an a variable for instance, the Nix code still evaluates to the same data which means that the dependency tree does not change.

If a commit changes some other package which the package in question does not depend on (neither directly nor transitively) then the hash of the package in question will not change.

@enoren5

that means there could theoretically be as many different packages as there are thousands of commits in a given application’s source code repository

The reason is wrong but this effect still exists. Nixpkgs is no mere list of packages but (theoretically) an infinite data structure with an infinite amount of “different” packages that you could build. Only a subset of Nixpkgs is actually built and available in the binary cache.

Additionally, changes that affect a subset or even all packages do happen quite frequently and those cause rebuilds. There is indeed a lot of churn. This is held manageable through staging cycles where many changes each causing large amounts of rebuilds are bundled into one big rebuild every few weeks.

@enoren5

I can’t possibly expect NixOs package maintainers to build, store, and distribute all those unique binaries on package servers, right

No but that’s what’s happening

Though we maintainers barely do any of that by hand; that happens quite passively through automation. All I do is manage the Nix code in Nixpkgs. The CI system (hydra.nixos.org) picks up those changes every day or so, builds all packages it’s configured to discover and pushes any built derivations to cache.nixos.org where users can substitute from.

@enoren5

I see in the searchable package repositories, versions are consistent overtime and only change with major releases that correspond to tags on GitHub.

You might be confusing release channels with versions here. Stable channels (i.e. 23.11) are a snapshots of Nixpkgs master from a certain time (i.e. November 2023) with important fixes backported.

@enoren5

Does this mean that the fluid package versioning according to commit hashes is only for Home-Manager which is used more so by developers who are testing recent changes on a beta release for a project that they are writing / contributing to?

Nix’ “fluid package versioning” (as in the hash of the dependency tree) is used for everything. It’s fundamental to how Nix works.

@enoren5

Does this also mean that with Flakes and Home-Manager, individual packages are assembled and compiled from source like Gentoo? Does this mean that the more a NixOS admin uses Flakes and Home-Manager, maintenance more closely resembles a source based distro?

All Nix packages are assembled and compiled from source just like Gentoo. Nixpkgs is a source-based distro.

Because of Nix’ purity guarantees however, we can say that the same derivation will always produce the same result. It’s like a pure function. That’s where the “functional” package manager comes from.

Pure functions always have the same result when called with the same parameters and can therefore be memoized (~=cached). That’s what Nix does with derivations.

When you run nix-build locally, it instantiates a derivation from your local Nix code (you have at least one Nixpkgs revision on your machine) and then tries to build it. Before doing so however it asks cache.nixos.org whether it has built the same derivation before. If it has, it just downloads the result of that build rather than building it again.

@enoren5

‘unstable’ which would then download and install Hyprland v0.36.0, which is still not the latest. If I really wanted bleeding edge, I would need to compile v0.37.1 from source. The best way to do that would be to use a Flakes script for Home Manager. Is this correct?

You could do it in a flake, NixOS or home-manager but those are tangential to the issue you’re trying to solve.

What you actually need is the hyprland derivation but with slightly different parameters (a different source).

Nixpkgs’ override mechanism exists for this purpose: hyprland.overrideAttrs (prev: { src = pkgs.fetchFromGitHub { ... }; }) will return a new derivation that is the same as the regular hyprland derivation but with a different source from which the package will attempt to build. You’d plug that into wherever the hyprland package to be used is defined (i.e. programs.hyprland.package) and NixOS will deploy your definition instead of the regular hyprland.

This only works if hyprland’s build script’s assumptions w.r.t. patches, configureFlags etc. still hold. Such assumptions usually do hold but also quite frequently not; requiring further changes to the derivation.

Another (and IMHO better) way is to contribute this update to upstream Nixpkgs or add your review to an existing PR which already implemented it. That way everyone gets to have a newer version of Hyprland.

@enoren5

if I wanted to generate a local repository with a comprehensive history of the last 5 major Hyprland releases, I wouldn’t be able to do that with nixpkgs.

You sure would. It probably doesn’t package every version of hyprland but at least the major versions should exist. Though it’s probably better to just copy the definitions or use the whole revision of Nixpkgs if you’re doing local experiments.

[mars-monkey]

To use an example to better illustrate my point, the latest release of Hyprland is v0.37.1. It came out like last week I believe. But the current NixOS stable package channel is “23.11” and the official searchable nixpkgs repo for this channel shows Hyprland v0.32.3. It’s obsolete. If I wanted a more recent version, I could switch my NixOS package channel to ‘unstable’ which would then download and install Hyprland v0.36.0, which is still not the latest. If I really wanted bleeding edge, I would need to compile v0.37.1 from source. The best way to do that would be to use a Flakes script for Home Manager. Is this correct?

There is an official Hyprland flake provided by the developer that bundles the absolute latest release. It even has a Cachix binary cache so you don’t have to compile it locally. Here’s the docs on how to use it:

Hyprland on NixOS

Specifically, you have to use the Flake Package option (not Nixpkgs).

This doesn’t address your main question, but might be helpful for that specific use case.

[enoren5]

Thank you @Atemu for investing your time in answering all my questions. I really appreciate this. I have a lot to munch on, to think about, and to take away here.

@mars-monkey

There is an official Hyprland flake provided by the developer that bundles the absolute latest release. It even has a Cachix binary cache so you don’t have to compile it locally. Here’s the docs on how to use it:

Hyprland on NixOS – Hyprland Wiki

Specifically, you have to use the Flake Package option (not Nixpkgs).

This doesn’t address your main question, but might be helpful for that specific use case.

Noted with thanks. You are right that you didn’t answer my question, but having you reinforce the fact that Hyprland in their official docs already provides a Flake script to work with will help me install Hyprland on NixOS.