Concept: use any package version

[davidak]

We have a lot of package versions in git history of nixpkgs. Being able to use any available version would be a killer feature for Nix! No other package manager would provide so many versions.

This actually works right now when done manually: Use older package version with Nix · GitHub

I had this idea many years ago. Here is now my concept. It might become my first RFC.

Please provide feedback to the concept. Would you like to help implementing it?

@edolstra does this feature has a chance to be included in Nix?

Nix UI

I’m not a developer, so i don’t care so much about implementation details, but UI and UX design.

Install specific version of package

nix install firefox@55

You can specify only parts of the version. The latest available release of that version will be used. In this example 55.0.3.

Run specific version without permanent installation

nix run firefox@55

List all available versions of a package

nix list-versions firefox
74.0
73.0.1
72.0.2
71.0
70.0.1
69.0.3
...

For every minor release we have, list the latest patch version.

Show package details (including number of versions available)

nix show firefox
name: firefox
version: 74.0
description: A web browser built from Firefox source tree (with plugins: )
url: https://www.mozilla.org/en-US/firefox/
installed: no
...
available versions: 65 (use nix list-versions firefox)

Use specific version in package definition or module

propagatedBuildInputs = with python3Packages; [ paramiko@2.5.1 lxml@4.4 defusedxml ];

We might not want to allow this in nixpkgs itself, but it will be a killer feature for developers!

environment.systemPackages = with pkgs; [ firefox@55 ];

Users might need a specific version permanently and want to install it in a declarative way.

Implementation

For some packages, we have multiple versions in the current default channel. Search there if the requested version is available. For example ansible:

So nix install ansible@2.7 will install it from the default channel, whereas nix install ansible@2.5 will search the version from the history.

To get the available package versions, we have to search the complete git history of nixpkgs.

One approach would be to evaluate every commit. This probably takes too long!

We can search the git commit messages for common package update pattern.

Example: firefox: 54.0.1 -> 55.0 is package firefox@ version 55.0

This might miss later improvements and security patches of the package at this version. But it should work at the state of the update commit! (tested before PR is merged)

It is not efficient when every user downloads complete nixpkgs history and does heavy searching. My idea is to do it only once and build an index of all packages versions with the corresponding commit id. The result can be a CSV file.

This can be build by a standalone hydra job from master branch. As often as technically acceptable (every commit in master, every hour or only once a day).

By using master branch, it will list versions not even in unstable yet!

It would be good to use a commit where binaries exists for (channel generation), or at least mark the versions (in listing) where binaries exists in the binary cache.

I can try to implement this index generation script in python (my favorite language, already used in module tests and nixops), but the Nix (UI) side has to be done by someone else. For a prototype, i can just build a python wrapper for nix.

Remarks

I don’t think we should call this concept a nix channel, even tho we might use the channel mechanism to update it. The file name could be nixpkgs-all-versions-index.csv.

Would we want to export this data to repology to promote that Nix can provide all these versions? People might find Nix this way as a solution for their task (use old version).

The online package search can also show the number of available versions and by clicking on it, show all versions and by clicking a version, instructions to install and run.

We have to test if this old versions are actually usable on an up-to-date NixOS system and other distro.

Some packages might need a service (e.g. dconf?) at specific version to work. We might want to add a package metadata if the package is portable (not depending on a service). When false, the tooling can show a warning that it might not work. When not specified, assume true. We might find hints about this topic at flatpak portals.

For packages that write configurations or user data, state management is needed. For example, when i run firefox 50 the first time, a profile is created, when i run firefox 60 later, the profile is updated to version 60, when i run firefox 55 later, the profile of 60 is not compatible. I might want to use the older state of firefox 50. That could be solved with backups or filesystem snapshots. But since we don’t know where a program writes files, we can’t backup just that files. Maybe sandboxing/isolation can help. But that’s again a completely different topic and a big one on it’s own.

References

I got inspired for this implementation by All the versions with Nix

You know what'd be really cool? For either Nix or Guix to transparently support ... | Hacker News (here someone had the same idea + comments)

Other package managers UI for specifying version

apt

apt install package=version

brew

brew install postgresql@8.4.4

dnf

dnf install tito-0.5.6-1.fc22

chocolatey

choco install nodejs.install --version 0.10.35

[jonringer]

We do this to a limited capacity if it warrants having a split on a package. Similar to the openssl_1_0_2 vs openssl_1_1_2

Having deep pinning of packages would also blowup the package set a lot.

A big difference between those package repositories and nixpkgs is that they just unpack binaries into directories. Where as nixpkgs builds a derivation.

You can somewhat simulate this with overrideAttrs, but it’s not as ergonomic as the examples you presented.

[edolstra]

With flakes it’s already pretty easy to run/install something from an arbitrary version of Nixpkgs, e.g.

$ nix run nixpkgs/9077284a7ca32efd56e652286fdaa2ad851ce0c9#hello -c hello
Hello, world!

What is missing is a way to search historical versions of Nixpkgs for a particular version of a package. This cannot be done locally because it would be way too expensive and the database would be too large. However, @garbas is working on loading the packages.json files from each Nixpkgs release into ElasticSearch. The capability to search this database could be added to the nix CLI as a convenience.

However this would only be a CLI thing. This:

environment.systemPackages = with pkgs; [ firefox@55 ];

will never work because firefox@55 is very under-specified: Nix would have no idea what firefox@55 to use. Also, we’re moving away from a reliance on version numbers. The nix command, unlike nix-env, is based on attribute names rather than <pname>-<version>.

This can be build by a standalone hydra job from master branch.

That’s basically https://hydra.nixos.org/job/nixpkgs/trunk/tarball/latest/download-by-type/file/json-br.

[picnoir]

nix run firefox@55

For the record, Guix is able to do that, using this exact same syntax.

See GNU Guix Reference Manual

I guess a starting point would be to look at how this feature is
implemented.

[davidak]

Thanks for the feedback so far!

@jonringer

Having deep pinning of packages would also blowup the package set a lot.

Of course the feature can have disadvantages, but it solves a practical problem that otherwise cannot be solved easily and cleanly (install old .deb package over current one, build from source). If you need this, you would also accept the disadvantages.

Nix can be the rescue when people need a version their distro don’t provide (older, newer).

@jonringer

A big difference between those package repositories and nixpkgs is that they just unpack binaries into directories. Where as nixpkgs builds a derivation.

It’s the same when we have binary substitutes, so we should prefer nixpkgs commits that was build as channel somehow. But when someone needs a very specific version and we don’t have a binary package, they still can build it from source (unless the source is unavailable). That’s an advanntage, not a disadvantage! A bad solution is better than no solution.

@edolstra

This cannot be done locally because it would be way too expensive and the database would be too large. However, @garbas is working on loading the packages.json files from each Nixpkgs release into ElasticSearch. The capability to search this database could be added to the nix CLI as a convenience.

You are probably right. ElasticSearch is a good solution to speed up package search. We could add nixpkgs-all-versions-index.csv as one special “channel”.

@edolstra

However this would only be a CLI thing. This:

environment.systemPackages = with pkgs; [ firefox@55 ];

will never work because firefox@55 is very under-specified: Nix would have no idea what firefox@55 to use.

can’t we use the same logic as with CLI (if version is given, search in ElasticSearch for all matching ones and select the latest) or does this conflict with lazy evaluation?

@edolstra

Also, we’re moving away from a reliance on version numbers. The nix command, unlike nix-env , is based on attribute names rather than <pname>-<version> .

I think we can still support this @ syntax as an additional feature which searches versions in current channel and ElasticSearch as fallback.

@picnoir

For the record, Guix is able to do that, using this exact same syntax.

That’s amazing. So far I like everything I have seen of their UI and feel confirmed that this is the right way.

@picnoir

I guess a starting point would be to look at how this feature is
implemented.

I guess they just search the current channel or how it’s called there. They have a lot of guile versions for example.

[picnoir]

I guess they just search the current channel or how it’s called there. They have a lot of guile versions for example.

Indeed, looks like you’re right.

Couldn’t refrain myself to jump in this rabbit hole ><

The search is done through folding all the packages here:
http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages.scm#n281

If you look at the package definitions, several versions of the same
software needs to be defined for this to work.

For instance in the guile case:
http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/guile.scm

I’m not sure whether we can translate this in the Nix context though.

Nix folds are rather slow, such a fold would need to live somewhere in
the nix C++ codebase.

This would also probably mean to rething the whole all-packages.nix
structure since you would now bind several packages to the same
top-level attribute

I’m also unsure this would scale well for nixpkgs provided the fact the
repository is order of magnitude bigger from the guix one.

[edolstra]

@davidak

I think we can still support this @ syntax as an additional feature which searches versions in current channel and ElasticSearch as fallback.

No, because doing dynamic lookups in an external database is inherently impure. firefox@55 could produce a different (and irreproducible) result every time.

Unless you’re talking about keeping old versions in Nixpkgs. We don’t do that because it’s unmaintainable.

[SRGOM]

This All the versions with Nix is a way to almost achieve what op wants. firefox@55 is in fact ambiguous in that it’s not clear if this is 55.0.0 or 55.<max>.<max> or 55.0.<max>. In that sense, the solution on the link is very close to what you’re trying to achieve.

[davidak]

That article inspired me to the suggested implementation idea. It’s a good start, but my goal is to make all versions we have in the git history easily available to the user.

@SRGOM

firefox@55 is in fact ambiguous in that it’s not clear

As i said, i suggest to implement it as 55.<max>.<max>. Like GUIX actually does it.

[davidak]

@edolstra

No, because doing dynamic lookups in an external database is inherently impure. firefox@55 could produce a different (and irreproducible) result every time.

yeah, right.

What do you think about name@<commit id> (short or long commit id). That would be reproducible. name@channel would be also nice to more conveniently use packages from unstable (firefox@nixos-unstable). Needs logic that ‘nixos-*’ are channels and not versions or commits. A different channel could be explicitely specified by name@channel:nixpkgs-19.09-darwin.

That would make this solution a first class feature, that is convenient, obvious and easy too use. It’s probably the killer feature for desktop users. It fit’s into a tweet: “Just use firefox@unstable to get the latest version!”

https://nixos.wiki/wiki/FAQ#How_can_I_install_a_package_from_unstable_while_remaining_on_the_stable_channel.3F

That syntax can also be supported in CLI.

[timokau]

I think this is a great idea, though I’m not entirely sure this can be done reliably enough to be “blessed” as a core part of nix itself. It should definitely start out as a third-party tool.

@edolstra

@davidak

I think we can still support this @ syntax as an additional feature which searches versions in current channel and ElasticSearch as fallback.

No, because doing dynamic lookups in an external database is inherently impure. firefox@55 could produce a different (and irreproducible) result every time.

It could be done in a reproducible manner, if this database is versioned. The impurity really is not any different than the impurity introduced by channels. It can be fixed in the same way: Pinning the database to a particular version.

[FRidh]

@davidak

What do you think about name@<commit id> (short or long commit id). That would be reproducible. name@channel would be also nice to more conveniently use packages from unstable ( firefox@unstable ). Needs logic that ‘unstable’ and ‘nixos-*’ are channels and not versions or commits.

Already exists

nix run -f channel:nixos-20.03 hello

[jtojnar]

Using <attrName>@<channel> is inherently impure, unless you use pinning. Flakes solve that and you can use <flake>.<attrName> inside expressions, I think.

It might be possible to create implicit flakes using the @ syntax during evaluation but I think making them explicit in flakes.nix is more useful in all but the smallest expressions.

In console, nix build -f 'channel:nixos-unstable' <attrName> works. You can also use https://github.com/NixOS/nixpkgs/archive/<ref>.tar.gz as from, or the flake syntax Eelco mentioned above.

[davidak]

@FRidh

Already exists

nix run -f channel:nixos-20.03 hello

Yes. My goal is to make it simpler and more convenient to use.

nix run hello@nixos-20.03

is not only shorter, i think it’s symantically better.

Like you can read it as “Nix, please run the package hello with the version available in NixOS 20.03 release”.

It would share the syntax with specifying a specific version from history or at a specific commit.

nix run hello@2.8 (search in history index for commit id)
nix run hello@master (fetch git master)
nix run hello@9cc3372 (git commit)
nix run hello@nixos-20.03 (channel)

[edolstra]

Except for the first example (searching history) the nix-flakes branch supports these:

nix run nixpkgs/master#hello
nix run nixpkgs/9cc3372#hello
nix run nixpkgs/nixos-20.03#hello

[volth]

firefox@55 is a perfect example why it won’t work:

1. Moving old derivation into modern (20.03) nixpkgs won’t work because older firefoxes require patches to be built against glibc 2.30 (people are willing to avoid supporting such patches, that’s why old software disappears from nixpkgs so quickly)
2. Importing firefox@55 from older nixpkgs together with older glibc, older GTK, … won’t work because there are too many singletons in NixOS GUI (OpenGL, GTK env vars pointing to .so files, …) and here we would get into classical DLL Hell.

[davidak]

Yes, might be a bad example. As i said in the Remarks section, only packages work that have no additional dependency on some service or API (the kernel API is quiet stable, right?).

To 1.: glibc is a library, not a running service. why wouldn’t firefox@55 use an old glibc that works? is it not defined as dependency, so the package is impure?

We could add a meta.portable = false; (or pure?) to them. But that won’t get into old channels.

A practical exaple: we used a specific version of ansible at my former company, so our shared config works for everyone. It was easy to install it with pip on macOS, but was not straight forward on NixOS.

This feature could be huge in reproducible science. But pinning nixpkgs should be preferred.

We have to document it accordingly, what works, what might work and what is expected to break.

[edolstra]

@volth

Importing firefox@55 from older nixpkgs together with older glibc, older GTK, … won’t work because there are too many singletons in NixOS GUI (OpenGL, GTK env vars pointing to .so files, …) and here we would get into classical DLL Hell.

Well, that’s a separate issue that we really need to address. It used to be that you could fairly reliably use a binary closure between different versions of NixOS or on a different Linux distribution. Nowadays they just tend to segfault.

[davidak]

@edolstra

Except for the first example (searching history) the nix-flakes branch supports these:

nix run nixpkgs/master#hello
nix run nixpkgs/9cc3372#hello
nix run nixpkgs/nixos-20.03#hello

Great, so we talk just about the UI. I think that looks cluttered and # is a strange character to use in a parameter.

What do you think about the @ symantically? like ‘package at version x’ if we use that instead of #, it would look like

nix run nixpkgs/hello@nixos-20.03

It still feels cluttered because 3 elements are stitched together and is hard to read. A better symantic would be:

nix run --from nixpkgs hello@nixos-20.03

But what if we want a package from another flake in that shell?

nix run --from nixpkgs hello@nixos-20.03 --from dwarffs dwarffs

I suggest since years to set nixpkgs as default channel/flake, so we don’t have to set it explicitely in the tools.

nix run hello@nixos-20.03 --from dwarffs dwarffs

or just hello

nix run hello@nixos-20.03

or just hello from default channel

nix run hello

here we finally have a very short and simple command! and this is probably the most common use-case.

and the same syntax can be used for imperative install:

nix install hello

or

nix install hello@nixos-20.03 --from dwarffs dwarffs

i think that would be quiet intuitive.

[edolstra]

The # character comes from URLs. The flake references in the examples above are really abbreviations for a URL like git+https://github.com/NixOS/nixpkgs.git?ref=master#hello.

@davidak

nix run hello

In the flake world, this command runs the default package from the “hello” flake. There is no default channel like nixpkgs (in fact there are no channels anymore).