Concept: use any package version

[davidak]

@edolstra

The # character comes from URLs.

That’s an implementation detail the user don’t has to see if it don’t improve the user experience.

Like said earlier, i want a concept for a user-friendly package manager that has the features of Nix. I don’t care how we implement that later at this point. How parameters are used should be easy to change. That’s the perspective i want us to have when discussing this.

In the flake world, this command runs the default package from the “hello” flake.

That’s how it’s implemented now, but we can change that. See it as a completely different program.

If you want a command to “run the default package from the “hello” flake”, it would look like this translated from my concept above.

nix run --from hello

There is no default channel like nixpkgs (in fact there are no channels anymore).

I don’t know the term in your current implementation, but you probably understand what behavior i expect from the commands. Like it should be obvious with good UI design.

I could just write a wrapper:

if flake='':
  flake = 'nixpkgs'

You probably can do that in C++ too That’s not the hard part here.

The feedback i would like to get from the community is:

• Do you consider the UI concept intuitive and easy to use?
• do you see a way to improve it?
• Are there use-cases that i have missed (scope here is specifying packages, versions, “channels” (NixOS releases with updates), flakes)

The next step would be to implement a script to generate nixpkgs-all-versions-index data and put it into an ElasticSearch. And create a wrapper for Nix that has this UI. Then wen can try out how it feels. I could just go to people and ask “I have a package manager here, can i show you how it can install packages”. And see their reaction when we actually have 50 versions of ansible for example. And let them play around with it. See what questions people habe “how can i do X?” and where they get stuck. They should say “This is amazing, i want that!”

If this test is sucessfull and the community likes it, would you accept a PR to Nix that implements it, @edolstra?

[davidak]

For reference:

GitHub and GitLab also use the @ syntax when a commit in a different repo is linked.

This is the most elegant and intuitive way to reference a version of a project/package from all i have seen.

But maybe someone can think of a reason why this could lead to problems or limitations?

[gcv]

@edolstra

No, because doing dynamic lookups in an external database is inherently impure. firefox@55 could produce a different (and irreproducible) result every time.

If this is the main objection, then what about dropping implicit versioning and forcing a fully-specified version each time? firefox@55.0.3 or whatever. The proposal under discussion here is so incredibly useful that it should not be thrown away with the impure bathwater.

[davidak]

@gcv

If this is the main objection, then what about dropping implicit versioning and forcing a fully-specified version each time? firefox@55.0.3 or whatever.

When the version database link the version to the commit where the package was updated, it would be distinct. Since that should be quiet easy to implement, i will start with that.

Later it might be nice to dig deeper and find the latest commit with that version, so including security patches and other fixes. But that means, when a package is changed without changing the version, the version number is impure. A workaround is to use the commit id as explained above. But that is not that user friendly since you don’t see which version you will get.

Example:

nix run hello@9cc3372

environment.systemPackages = with pkgs; [ hello@9cc3372 ];

I would solve it by adding a comment in my code.

nix run hello@9cc3372 #2.10

environment.systemPackages = with pkgs; [
  hello@9cc3372 #2.10
];

Like i do when pinning nixpkgs now:

davidak/davidak.de/blob/aa6016e42f49ebd1e9466108e4ce608b00f689e2/nixpkgs.nix#L6

Another approach could be to pin the package version index. I don’t know if ElasticSearch can version the data and provide a specific version via the API.

nixpkgs.config = { useVersionsIndex = "2020-04-02"; }; (we can build it daily)

This can be set optionally and then versions are allowed in configuration.nix. When version is used and it is not set, show warning to either use commit id or set it.

I think when using imperative tooling, it is fine to get impure versions. This is how it is today when the channel is updated.

So nix run firefox@55 get’s the latest version in 55 release series (actually 55.0.3). But we can output the commit id, so when the user wants to use this exact version again later, they can use the commit id.

[user@nixos:~]$ nix run firefox@55
Using firefox 55.0.3 (9cc3372)

[user@nixos:~]$ firefox --version
Mozilla Firefox 55.0.3

@gcv

The proposal under discussion here is so incredibly useful that it should not be thrown away with the impure bathwater.

Yeah, i’m also very excited about this. But we have to do it correctly and solve upcoming issues. Pure packages are a very important part of Nix, so we shouldn’t compromise on this here.

[jonringer]

I think the lines of what is possible is being blurred here. The nix syntax (front-end) just needs to resolve to a derivation, which is used by nix to produce (instantiate) a build output. When you say nix run firefox, nix is saying “for my nixpkgs path, what does firefox mean?”, resolves it to a unique derivation, and then leverages that to the command being ran. if you were to do firefox@55, you would have to have knowledge about the point in time in which firefox derivation was on version 55 (so that the correct libraries, dependencies, and patches can be used; OR extend the the package set to include all the old versions, but that’s a massive maintenance burden)

nix-instantiate -A firefox
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/sd2i5gv7jl1d8mwqjiraxdpzcyrz94dd-firefox-74.0.drv
$ nix-build /nix/store/sd2i5gv7jl1d8mwqjiraxdpzcyrz94dd-firefox-74.0.drv
these paths will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/q8sc39abyh1ncrff6xmrflq2bx6c6d49-firefox-74.0
copying path '/nix/store/q8sc39abyh1ncrff6xmrflq2bx6c6d49-firefox-74.0' from 'https://cache.nixos.org'...
/nix/store/q8sc39abyh1ncrff6xmrflq2bx6c6d49-firefox-74.0

@davidak

When the version database link the version to the commit where the package was updated, it would be distinct. Since that should be quiet easy to implement, i will start with that.

This would also the undesirable effect including the entire dependency tree for each version.

if i were to do firefox@55 firefox@60 firefox@65 firefox@70 firefox@74, I would get a different set of xorg + stdenv. It would be massive.

$ nix path-info /nix/store/q8sc39abyh1ncrff6xmrflq2bx6c6d49-firefox-74.0 -Sh
/nix/store/q8sc39abyh1ncrff6xmrflq2bx6c6d49-firefox-74.0	 498.3M

to further complicate this, you could be pulling in from many release channels, as a system by have a stable and unstable branch defined.

[davidak]

@jonringer

if you were to do firefox@55, you would have to have knowledge about the point in time in which firefox derivation was on version 55 (so that the correct libraries, dependencies, and patches can be used

I just looked up how to search git commit messages. It’s quiet easy:

$ git log --oneline -1 --no-merges --grep='firefox:.* -> 55'
dec0929f197 firefox: 55.0 -> 55.0.3

@jonringer

OR extend the the package set to include all the old versions, but that’s a massive maintenance burden

And it was decided we don’t want to maintain more than needed. So this is not an option.

@jonringer

This would also the undesirable effect including the entire dependency tree for each version.

if i were to do firefox@55 firefox@60 firefox@65 firefox@70 firefox@74 , I would get a different set of xorg + stdenv. It would be massive.

Sure, but we want the dependencies known to work, right? That’s the price to pay for this feature! This has to be communicated clearly.

[davidak]

So it took me 10 minutes to get a list of all updated packages:

[davidak@X230:~/code/nixpkgs]$ git log --oneline --no-merges --grep='.*:.* -> ' | awk '{print $2 " " $5" ("$1")"}' | sort | less
0ad: 0.0.23 (d7679fb2b5a)
0ad: 0.19 (1e0434ebe0c)
0ad: -> (3455919cd01)
0ad data: (950a6984885)
* -> (10e8a801b69)
* -> (143ce1d1dd5)
18.09: Jellyfish (03cf538ef22)
1password: 0.4.1 (6abc2bf4063)
1password: 0.5.1 (1b6190e0050)
1password: 0.5.3 (96bf79c2a13)
1password: 0.5.4 (b065bb21333)
1password: 0.5.5 (b3d8aa2677e)
1password: 0.5.6-003 (4c127d4c68e)
1password: 0.5.6 (6233db03e7a)
1password: 0.5.7 (75dd222252a)
1password: 0.6.1 (2af98a46c14)
_1password: 0.6.2 (0d57e8a0e88)
1password: 0.7.0 (971bf59cd31)
[20.03] -> (e76d4a031af)
2048-in-terminal: 2017-11-29 (9058d33499d)
: 2.3.0 (7da8192959f)
_2bwm: 0.3 (e9b8203a389)

(49106 package versions)

The regex is not very robust, but you get the idea.

That includes 104 firefox versions from 27.0 to 73.0.1!

Older versions had different commit syntax, like:

b1ce15f77b1 firefox: Update to 15.0.1
d3ffbea9d64 Updating firefox 13.0 to 13.0.1. Maybe this improves the flash issue.
af1b08ac32e Updating firefox to 10.0.1. I'll let hydra build it.
362d1ba38fe update firefox to 3.6.6
0591ccedc0e * firefox -> firefox-2.
7cc6405859b firefox 2.0.0.13
c48c2aa23b4 firefox 1.5.0.6

You can specify the time range where git searches with --before=<date> and --after=<date>, so we can run it multiple times with different regexes that fit the commit style.

Now only the version where the package was initialized is missing.

[davidak@X230:~/code/nixpkgs]$ git log --oneline --no-merges --grep='init at' | awk '{print $2 " " $5" ("$1")"}' | sort
_0x0: 2018-06-24 (582374ea9ed)
1password: version (5d66a6bbdb7)
2048-in-terminal: 2015-01-15 (0b167040006)
20kly: 1.4 (0fd58ed0997)
2bwm: 0.2 (fd3a99633bb)
_3llo: 0.3.0 (0ad35681143)
3proxy: 0.8.13 (b571728081a)
3to2: 1.1.1 (1aaeb94855d)
6tunnel: 0.13 (f56cd1adcef)
90secondportraits: 1.01b (7281144f4b9)
abbaye-des-morts: 2.0.1 (ac6e484cf8a)
abcl: 1.4.0 (1d0fb40af00)
abcm2ps: 8.13.15 (107b1815239)
abcMIDI: 2017.06.10 (608045aed54)
abi-compliance-checker: 2.3 (8d42787c61e)
abi-dumper: 1.1 (cc07807b5bd)
abootimg: 0.6 (1b4027c6fca)

(8849 package versions)

Now i actually have a “database” (textfile) with 57955 package versions. It’s size is 1,9 MB. Searching it with grep takes 0,006 seconds.

[davidak@X230:~/code/nixpkgs]$ grep -E 'firefox: 55' nixpkgs-all-versions-index.txt
firefox: 55.0.1 (71d0a5a328f)
firefox: 55.0.2 (f175cdc4117)
firefox: 55.0 (39e6dfe2f60)
firefox: 55.0.3 (dec0929f197)

This is what i have after 38 minutes. What do you think?

Is it now clear what my goal is and how i want to achieve it?

[gcv]

@jonringer

if i were to do firefox@55 firefox@60 firefox@65 firefox@70 firefox@74 , I would get a different set of xorg + stdenv. It would be massive.

All mutual dependencies in the derivations of firefox versions 55 and 60 can be shared, correct? But if they have different dependencies, like different versions of libc, then a different set of X + stdenv is precisely the desired behavior. Firefox, or any other X app, is a pathological (albeit common) case here. Many situations, like C libraries used for development, will not have such large dependency graphs.

[edolstra]

@gcv

If this is the main objection, then what about dropping implicit versioning and forcing a fully-specified version each time? firefox@55.0.3 or whatever. The proposal under discussion here is so incredibly useful that it should not be thrown away with the impure bathwater.

firefox@55.0.3 is not any better. It’s still not reproducible, and derivations named firefox-55.0.3 can and do evaluate to a different store path in different Nixpkgs revisions.

The whole value proposition of Nix is reproducibility. The language will never support this kind of dynamic lookup in an external database.

For the CLI it’s different though. A command like nix search could do a query against ElasticSearch or whatever.

@davidak

environment.systemPackages = with pkgs; [ hello@9cc3372 #2.10 ];

It’s a nice syntax but it’s also underspecified (9cc3372 of what repository?). You can already do this:

(getFlake github:NixOS/nixpkgs/e727398a2dd45722fc606fa31c60ef5f87f4cdd5).legacyPackages.x86_64-linux.hello

(or use fetchTarball), but most of the time you don’t want this because it leads to closure bloat (since hello will have a completely different set of dependencies than the rest of the system).

[tgamblin]

@davidak

Being able to use any available version would be a killer feature for Nix! No other package manager would provide so many versions.

Since the Guix folks are on the record above, I’ll just say that this is exactly the goal of Spack (GitHub - spack/spack: A flexible package manager that supports multiple versions, configurations, platforms, and compilers.), but not just for versions. Spack is heavily inspired by Nix, but it’s more of a departure from Nix than Guix. Recipes are templated by version, compiler, build target, and arbitrary other parameters (“variants”) that can be exposed by the package.

$ spack install mpileaks                             # unconstrained
$ spack install mpileaks@3.3                         # @ custom version
$ spack install mpileaks@3.3 %gcc@4.7.3              # % custom compiler
$ spack install mpileaks@3.3 %gcc@4.7.3 +threads     # +/- build option
$ spack install mpileaks@3.3 cppflags="-O3 –g3"      # custom compiler flags
$ spack install mpileaks@3.3 target=skylake          # set target microarchitecture
$ spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3   # ^ dependency information

There’s no external database here, though – the packages expose options and constraints that define the space of possible builds, and we resolve a DAG from that. Spack allows you to have multiple environments with arbitrary configurations in them much like Nix, but it’s parameterized. We have to make some tradeoffs to allow that.

The builds are “reproducible” in the sense that the packages files + the concrete (resolved) DAG description with all the metadata will result in the same build. We don’t guarantee that spack install foo@version will always do the same thing, but we generate a lockfile that you can reproduce a particular configuration from. So it’s similar to what Nix in some ways, but you have to do dependency resolution to allow arbitrary parameters like this. If you are interested in this, see this FOSDEM talk.

@davidak

Like said earlier, i want a concept for a user-friendly package manager that has the features of Nix. I don’t care how we implement that later at this point. How parameters are used should be easy to change. That’s the perspective i want us to have when discussing this.

I think the Nix folks will have disagreements about some of the things we did in Spack to allow this flexibility, so I’ll be interested to see how this discussion plays out and how people might do this in Nix. I’d be interested to know if Spack fits your use case, as well as what from Nix is lacking that you’d miss (I’m sure there are plenty of things – Spack is not as mature as Nix yet!)

[shanesveller]

If this test is sucessfull and the community likes it, would you accept a PR to Nix that implements it, @edolstra?

I think there needs to be much more diligence around proving out that the idea presented is technically sound/feasible before trying to push aggressively to hear that such a PR would be welcomed upstream. There’s not yet a proper POC to look at yet, and worked examples are super important here.

One of the (probably casual, not ultra sincere) suggestions included on this thread entails an ElasticSearch dependency as part of resolving your Nix expressions, which I find fairly unappetizing personally. Given the depth and breadth of the nixpkgs Git history, it’s not super viable to do a walk dynamically at runtime, and the question of “how much of the frozen-in-time dependency tree gets pulled in” has not been answered at all, let alone thoroughly. (I’d wager that it’s almost impossible to answer generically and have safe outcomes a majority of the time.) It’s also very lossy/low-fidelity to just regex across commit messages as the source of truth for application version evolution, IMO. Other reservations were raised about not special-casing the syntax and having consistent usage between CLI and Nix code. These are all extremely impactful concerns to what a hypothetical implementation looks like, and we can’t neglect “implementation details” as unimportant when they contradict the narrative, as they represent the tips of icebergs waiting to sink the ship.

I get that this suggestion is, by the author’s own wording in the first post, something he hopes to grow into a more formal RFC and so doesn’t need to be held to particularly rigorous standards yet, but I can’t help but feel that over the course of this thread there were many valid concerns that have been raised and answered with a fairly cavalier/dismissive attitude. Meanwhile, several existing design decisions driven by technical motives were hand-waved as malleable without acknowledging or accounting for those technical motives. Someone somewhere does have to have an answer for them or this idea can’t hold water.

In the long run I bet the Flakes design would let OP solve this goal in userspace Nix code. If that’s still not enough, perhaps a custom frontend could be created if the subjective user-friendliness and approachability of the UX are more sacrosanct than the correctness of the outcome. To paraphrase Kent Beck, I think the goal should be “make it work, make it right, make it intuitive”, in that order, and designing for intuition is an entire field of study on its own.

[mjlbach]

This concept in some form would solve a huge problem in python package dependency management in nix, which forces all modules to be at the current snapshot of versions of the pinned nixpkgs and prevents locking/transferring dependencies across nix/non-nix systems. For some python packages, the dependencies are python only and would not have issues with pulling in large dependencies such as Xorg

[das-g]

Just to throw an idea out there: Instead of using ElasticSearch or another search database, maybe git bisect could be used to determine the latest NixPkgs revision to contain a wanted version or a specific package? An approached based on that could probably even handle version ranges or other looser version constraints (e.g. “Python 2.6, 2.7 or ≥ 3.3, but ≤ 3.6”).

It would require though (as input, configuration or hard-coded) some revisions between which to search.

[das-g]

@das-g

maybe git bisect could be used

It could probably even use the advantages and optimizations Nix-bisect -- Bisect Nix Builds brings to git bisect on the NixPkgs repo.

[davidak]

@das-g

maybe git bisect could be used

that would require to have nixpkgs git repo checked out which is on my system 1.6 GB! when the package index file is about 2 MB even ElasticSearch is not needed.

[danieldk]

@mjlbach

This concept in some form would solve a huge problem in python package dependency management in nix, which forces all modules to be at the current snapshot of versions of the pinned nixpkgs and prevents locking/transferring dependencies across nix/non-nix systems.

As far as I understand the limitations of Python packaging [1], this would actually make things worse. If someone would use versions of packages a and b that come from different nixpkgs revisions and both packages are dependent on c, then the closure would contain (potentially) conflicting versions of c since Python imports do not use fully qualified store paths.

[1] Strong disclaimer: I don’t use Python much except for the occasional 10-liner, so take this with a grain of salt.

[mjlbach]

Yes, I agree a naive implementation would have severe issues (which is why I added to caveat in some form). Pip gets around this by having min/max compatibility version bounds and doing dependency resolution which is very different than nix’s current approach to having an all-encompassing set of compatible packages.

[davidak]

FYI: Another implementation Nix Package Versions

[davidak]

And another search tool: https://4shells.com/nixdb/search

[jeff-hykin]

Alright I’ve made a tool that effectively does exactly what you want @davidak !

Nix Version Search

nvs --install firefox@55

Its not just for system-installing though.
nvs python@3 defaults to showing how to use the package inside of a shell.nix

NVS pings/combines the output of:

• nixhub.io
• flakehub.com
• history.nix-packages.com
• lazamar.co.uk

I’m probably going to put it in announcements after some more testing, but I’ve been following this thread for a while so I figured it’d be good to start here.