Flake.lock inadequate?

Development
1

I get it, nix doesn’t really care about versions but it’s all about the hashes of the input. And I love a lot of things about nixos - but sometimes it feels awkwardly disconnected from CVEs and package versions.
At least for me it’s not as easy as it should be to know which versions I have available.

The flake.lock holds the current state of the inputs - but wouldn’t it make more sense to have a lock file that locks everything that goes into the nix store and additionally shows a human readable version (if available for the sha)?

Honestly it would be nice to have a similar UX as e.g. homebrew.

$ nix flake outdated
ffmpeg c00170e9790f2c928844b5a6c7fe9022879a222174d0401e6c2203c851f16390 (6.1.1_1) < 258dcf3e6e4ff78b6e8b7ed7d19e729d592de01f8a1606490c9538493f9aecd5 (6.1.1_2)
podman bb815101287d83f94f6871a512e43e3a0d46723c6ce8d30985dfeb93c21143ad (4.8.2) < 181b9364100fba118b89a1e9efa38bd3fd6922fb3dfa83ef099941866bd06c9c (4.8.3)

$ nix flake update podman
locked podman bb815101287d83f94f6871a512e43e3a0d46723c6ce8d30985dfeb93c21143ad (4.8.2)

Given that the nix store can multiple versions of the same software the UX probably needs to change a bit - but I am wondering what the vision is here.

Please forgive the possible heresy. Just some random thoughts from a newbie :slight_smile:

1 Like
2

You can use nix flake show to get the versions of packages in your flake.

I don’t think versions should be part of your flake.lock data, given that your inputs don’t necessarily even refer to source code, let alone a single project that would have a useful version tag, nor is flake.lock intended to be human-readable, but package versions can be and absolutely are part of the metadata recorded in derivations.

I don’t use the nix profile commands much, perhaps this is more about UX surrounding feedback from installing/updating packages using it than it is about the contents of flake.lock?

3

Maybe it’s just an UX issue as there does not seem to be an easy way to say:

“show me the SHA (and if possible version) of all installed podman packages”

https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake-show.html

Just nix flake show looks like this for me

nix flake show 
git+file:///root/nixcfg?ref=refs/heads/main&rev=b231d9895c86a666510a3f831d3f5a1f16334109
└───nixosConfigurations
    ├───hetzner: NixOS configuration
    └───utm: NixOS configuration

Which isn’t that particular helpful to answer the above question.

I don’t think the version need (or even should) part of the flake.lock either. But the metadata for the SHA should be available somewhere.

Why isn’t the flake.lock a reflection of the entire nix store?

Again - a newbie here. But it feels like there is a lot of indirection to get to important information about the current system.

This might just be just a knowledge problem on my end and/or a tooling and UX problem.

4

Because you can have more than one flake on a system, which all write to the same nix store. Flakes are kind of like a Cargo.toml in rust land - that question is like asking why your Cargo.lock doesn’t contain a listing of the entire download cache in ~/.cache/cargo.

Flakes are not the tool to use if you want info about the nix store or the current profile or the currently active system generation.

Right, so your flake doesn’t contain any packages, it contains NixOS configurations, and what you want is not the versions of all packages in a flake, but the versions of all packages on your NixOS system, and you want to compare them between upgrades.

This can also be done, but you need to use nix store subcommands for this kind of thing. nix store diff-closures can be used if you want to compare system closures. I don’t know the incantations to just list the packages in a single closure off the top of my head, but I know they exist (otherwise diff-closures would not be possible).

Similarly, if you wanted to list packages in a nix profile, you’d use nix profile list for that.

You’re conflating a lot of things with flake metadata for some reason. Nix has a huge ecosystem surrounding it; the data you’re looking for is recorded, if you ask for help with specific use cases instead of starting by insisting that flakes don’t solve a problem they’re not designed to solve I think it’ll be easier to help you find what you need.

5

Let me rephrase that:

Why isn’t the flake.lock listing the packages that the flake brings to the nix store.

But that’s kind of clear now, too. It just lists the root of dependency tree.

I guess that’s how I ended asking that question as it does list the dependencies for the project.

and I was surprised to find flake.lock does not do the same way.
But it is locking them transitively via inputs. Which makes sense now.

True, but these configurations also install packages.
And I am after these packages - including their versions.

Sorry if the title was too heretical - or if it should have gone into #help instead.

My goal was:

I want to list all packages on a system/build/flake and see

  • the transitive dependencies and which caused them to be installed
  • I want to see the version number of packages (if they have one)
  • I want to see if there is a newer version

I’ll move back to #help and see if there tools for that.
Still I think it would nice if that something that is native to the official cli.

6

First problem: a notion of a package is fuzzy in general. Nix store can contains package data for previous generations or package data that is not in used anymore. So you have to be precise if you consider unused package data to be a package, etc.

nix path-info -r $a_nix_derivation gives the transitive closure.
nix why-depends $package $dependency explains why they have been installed.

Usually, a version number of a package can be obtained by reading the Nix store path directly, if it’s not possible, you should have the attribute path and perform the following evaluation attr.version, if there’s a version, you will get an output.

Evaluation can be done via nix-instantiate --eval usually or you favorite way to do so.

If you don’t know the attribute path, neither the store path, well, you don’t have the package, so there’s no version to query.

This question is hard and non-trivial, Nix doesn’t know what is a version, version are just metadata.

Now, we have various heuristics that should cover almost all cases, we know that versions are usually lexicographically ordered, so it is sufficient to evaluate any latest channels on status.nixos.org for your relevant channel and determine their .version for the package you care about.

If it’s higher, newer versions are available.

Is this wired in any tooling? No. Should it be? Yes. Unfortunately, we all have finite free time. :slight_smile: — if you are interested into this feature, I worked quite on the design to make it available for much more than that, so you can ping me (Matrix/DM/email/IRC/whatever) about it.

3 Likes
7

Not what I meant at all, I was just trying to explain that you’re looking at this from the wrong angle!

I think ultimately your problem is that the definition of “package” as typically used by flakes that build NixOS systems is not very granular, because the big nixpkgs monorepo acts as a middleman.

Kind of how you can get a rust toolchain package on some distros with a single version, but if you peek inside it contains multiple separate projects with individual versions. There, too, you need to use the rust tooling itself to figure out what the toolchain actually consists of, much like how you need to use nixpkgs tooling (and not flakes) to find out what nixpkgs contains.

The flake.lock format does support more granular package-like inputs, and in some corners you will in fact find little ecosystems of more granular packages split between flakes (hyprland is an example).

It just doesn’t make much sense for the abstraction level of a full distro, or at least there has not yet been a reason to break apart the current abstraction. Maybe if flakes ever stabilize, but I don’t think it would actually be feasible at the nixpkgs scale without upstream buy-in.

8

I think ultimately your problem is that the definition of “package” as typically used by flakes that build NixOS systems is not very granular, because the big nixpkgs monorepo acts as a middleman.

You mean it’s because the one commit SHA of the nixpkgs repo that marks the input to the flake, right? While true - that commit still defines all the versions for a given commit on the monorepo though.

It would be nice to have an easy cli way of accessing that info.

I see now why there is not much reason to have this explicit in a flake.lock.

And updating a single package is not as simple (as it’s part of the monorepo), It would instead be overriding a package that comes from the nixpkgs commit.

I hope I am getting closer :slight_smile:

9

First problem: a notion of a package is fuzzy in general. Nix store can contains package data for previous generations or package data that is not in used anymore. So you have to be precise if you consider unused package data to be a package, etc.

From my understanding a closure (that contains many derivations) is the root of all dependencies. For the moment of a clean install the nix store would reflect the same state as the closure. No?

And while I am mainly interested in the dependencies of the current closure it should also be possible to find the ones that are from a previous generation or not used anymore - I would assume.

These two commands raise questions:

nix path-info -r $a_nix_derivation
nix why-depends $package $dependency

While the path info seems to indicatate what gets included in the closure to have a working curl

nix path-info -r nixpkgs#curl

I am still not sure how to use why-depends to find all the reasons that caused curl to be part of the closure.

Usually, a version number of a package can be obtained by reading the Nix store path directly, if it’s not possible, you should have the attribute path and perform the following evaluation attr.version, if there’s a version, you will get an output.

You mean the easiest way to check for the version would be to just use

ls -la /nix/store

I guess that should work for the most part.

But a more standardized way via cli would still be nice. Be it just to have an explicit contract.

aluation can be done via nix-instantiate --eval usually or you favorite way to do so.

Sorry, there you lost the newbie on this side of the thread :slight_smile:

I want to see if there is a newer version
This question is hard and non-trivial, Nix doesn’t know what is a version, version are just metadata.

I figured. That said: as long it is a defined field in the metadata a lexical/semver comparision would be enough.

Now, we have various heuristics that should cover almost all cases, we know that versions are usually lexicographically ordered, so it is sufficient to evaluate any latest channels on status.nixos.org 1 for your relevant channel and determine their .version for the package you care about.

If it’s higher, newer versions are available.

That sounds good.

Is this wired in any tooling? No. Should it be? Yes.

Well, for me hearing that is the first step :slight_smile:

Unfortunately, we all have finite free time. :slight_smile: — if you are interested into this feature, I worked quite on the design to make it available for much more than that, so you can ping me (Matrix/DM/email/IRC/whatever) about it

I will.

My Haskell skills are still weak and I am just getting started with nix(os), so I am not sure how much I can contribute.

What I can contribute for sure are expectations of someone learning and give some inputs to the cli UX.

And I would love it to be more. But we’ll have to see how this goes.

1 Like
10

The Nix store is a forest, so it contains multiple roots and therefore multiple closures.

Current means a lot of things in general, it’s implementation defined, if you are using NixOS, current means /run/current-system.

In other Nix-based systems, it can mean other things.

On NixOS, you can nix why-depends /run/current-system $curl-path to understand why $curl is depended upon, you can use --precise to have even more details.

Mostly, yes.

This relies on assuming that all things have a version, but version is not a mandatory part of a derivation. A bunch of derivation does not even have version.

But such a CLI has yet to be designed, what does people care about? All versions of a given closure? etc.

What would be very helpful there is someone coordinating all the feedback on the ideal CLI, there was a CLI WG (I am supposedly part of) but it’s hard to get somewhere.

1 Like
11

Hm. Then I don’t have the correct grasp what a closure is yet. :-/

Ah, that was helpful.

nix why-depends /run/current-system `which curl` --precise
/nix/store/s0lyj84xgz7wk572g2iifanfq31qhd4m-nixos-system-nixos-24.05.20240102.bd645e8
└───sw -> /nix/store/cllcs92bkcqisp9x2xn2h8l5mgzw6gyl-system-path
    → /nix/store/cllcs92bkcqisp9x2xn2h8l5mgzw6gyl-system-path
    └───bin/curl -> /nix/store/llfg6h6cj2hv1zmnlg0xakqgdwirdi0c-curl-8.4.0-bin/bin/curl
        → /nix/store/llfg6h6cj2hv1zmnlg0xakqgdwirdi0c-curl-8.4.0-bin

But I require curl at different parts of the flake.
So I was hoping for to also have:

  1. environment.systemPackages
  2. users.users.tcurdt.packages

Not really.
It would mean there is contract for accessing optional version information.
The version information can still be optional.

How does that work in Nix land?

12

If I am reading you correctly, it seems like you might be suggested that a lock file essentially function as an SBOM. The real problem is that what you put in your flake inputs vs what is fetched elsewhere in code with fetcher functions is entirely arbitrary. Even if you maintain discipline in your own flakes, any reference to nixpkgs is gonna pull in an arbitrary amount of remote source fetches.

Part of this is just the timeline in which flakes came in. There were already millions of lines of Nix code fetching sources directly in the source before flakes existed as an idea. Perhaps a solution would involve a mode of Nix in which ad hoc fetchers are entirely disallowed, but I would want some of the short comings of flake inputs to be addressed before I actually use it.

Doing so would also be difficult since the some fetchers are impemented as FODs and others and builtin commands. Would we have to disallow FODs entirely to make this simple? Is that even feasible?

For this to be convenient flake inputs would have to be fetched lazily and in parallel to evaluation, which they currently are not. Also, essentially all of nixpkgs would have to be rewritten in this style to make use of standard functions like mkDerivation without breaking this new evaluation mode. Some user specified meta-data about the input would likely also be useful. Is this source code, or Nix lib code, or Nix packaging code, or configuration?

Just due to shere momentum, it may be more feasible to work in the opposite direction. Projects like sbomnix might be the answer long term.

Since you brought up CVE checking, it’s also worth mentioning that in the “future work” section of the original Nix thesis there was an idea to create some kind of “revocation list” of derivations which have known CVEs. Might be something worth actually implementing.

13

Indeed for an SBOM you would need to evaluate and track what is loaded.

Also in the current model (without “reuse input lock files”, Reuse input lock files · Issue #7730 · NixOS/nix · GitHub, as also linked by nrdxp in the preceding comment), you’d get a lot of false positives, because test dependencies and development dependencies (ie shells) will also be included in the lock file. (Unless all flake authors take very inconvenient measures against that - they won’t and shouldn’t)
While these are not completely irrelevant, these dependency relations cast far too wide a net to be useful.
That is, unless you really want to include the checks of the test tools that some developers use to develop the documentation tools for the static website of the test framework used by your development tools. Doing a survey of everything remotely relevant to your dependencies is cool, but probably not productive.