Clarification on Package Names and Versions

1. My Current Understanding:

There seem to be 4 parts for identifying a package:

  • Attr Name: the nix-shell name, such as:
    nixpkgs.ocamlPackages.pycaml

    1. I believe this is the most “official” name
    2. I don’t know much about how this name is generated. It seems to be related, but not quite, the path to the package inside nixPkgs. Ex: ocaml-modules/pycaml/default.nix.
      This is probably my greatest need of clarification.
    3. I believe this name can change what it refers to with new releases of nixpkgs. For example I think it is possible that nixpkgs.ruby can be ruby 2.1 on an old channel, but ruby 2.7 on a new channel. However, it seems that packages like python and llvm are frozen to keep pointing to their old versions (ex: python2 instead of python3). So I’m not sure.
    4. Often times part (and only part) of the name will include the version. Ex: nixpkgs.python38
    5. This refers to 1 and only one source (nixpkgs.python does not represent all sources offering python, or all sources offering python 2.7.18)
  • Version: version =

    1. I believe this field is not required
    2. No strict format. It seems that the version is more permissive than the package name, since it can start with letters or numbers. It generally seems any string matching:
      [.-+a-zA-Z0-9_]
      in any order or quantity is allowed. For example:
      version = "unstable-2020-08-31"
    3. The version can be set dynamically. Ex:
      version = "${builtins.substring 0 6 rev}"
    4. There is an official comparison method by nix builtins.parseDrvName to decide if one package is a more-up-to-date version of another package. This is detailed in the nix-env --help. However, many many packages seem to ignore this format entirely when setting their version.
  • PName: pname = , the “nix-env name”, the “pkg name”, the name of what is being installed, such as firefox, python, or zsh. This is what users use to install the latest version of a package without knowing the version.

    1. Not all packages have a pname, according to How to find the real name (once installed) of a package
    2. The pname is also dynamic, and seemingly more so than the “specific name” (talked about next). For example:
      pname = "dokuwiki-${hostName}"
    3. Multiple packages can have the same pname, but different Attr Name. Ex: python 3.8 can be found under many different Attr Names (which makes since due to different sources being able to offer the same tool & version).
    4. Some pnames include a variant name, such as
      pname = "python3-minimal"
      I’m not sure if this is an anti-pattern or not as I don’t know much about overrides and args.
    5. There seem to be naming constraints, which look like they need to match the following regex [a-zA-Z0-9_][.-+a-zA-Z0-9_]
      However, I do not know of official requirements.
    6. Some pnames have the anti-pattern of including the version, such as
      pname = "cryptodev-linux-1.11"
  • Name: name =, Very confusingly the package has a name that is different from the pname/pkgName. I will refer to this one as the “package + version name” or the “specific package name” or “specific version name”. This name is also used in nix-env.

    1. Unlike the first two, I believe all packages must have the package + version name.
    2. This name can still be programatic, sometimes very programatic, for example:
      name = "heapster-${lib.strings.substring 0 7 rev}"
    3. I believe this is often, but not necessarily, the pname + version joined by a -
      However, this is a problem. Both the version and the pname are allowed to contain -, letters, and numbers, meaning there is no way to know where the pname ends and the version begins. For example, this package 2048-in-terminal-2017-11-29, the 2017-11-29 is the version.


Does this sound about right so far @vcunat ?

Questions: (@ anyone)

  • Can nix-env -qa -f $commit on one machine can have different output than nix-env -qa -f $commit on another machine/OS?
  • Can the pname/specific name/version be the result of a curl request (aka impure)
  • Is it true that the Attr Name + nixpkg-commit-hash should uniquely be a specific-version of a specific-package? (even if that specific-version and specific-package can be a different depending on the OS)
  • How is the Attr Name determined?
    (including “sub” Attr Names like the “CGICompile” in perl530Packages.CGICompile )
  • If the nixpkg-commit-hash changes (forward or backwards in time) can an Attr Name change what is refers to? e.g.
    nixpkgs.python = python-2.7.18
    then at a later commit
    nixpkgs.python = python-3.8.1

I’ll try to keep this post up to date as new information comes in, and try to add links to relevant documentation and discourse posts as well.

4 Likes

The top-level attributes for packages are defined in:

https://github.com/NixOS/nixpkgs/blob/f0d05c42fbe21b6459260085e693a6c7f1d59fcf/pkgs/top-level/all-packages.nix

That’s (largely) because there are packages that depend on Python 2 or older LLVM versions. In general, we try to have only one version in nixpkgs.

Pretty much anything can be generated programmatically, including attribute names. Some package attributes are also generated programmatically.

This is because Nix generally doesn’t do version management. E.g., when you build a system with nixos-rebuild, you get a generation that contains the package versions that are provided by the nixpkgs revision that is used. So, there is no need to determine what packages should be upgraded, etc. Package versions are used as part of the output path. But other than that, they are largely metadata for the user.

If you use nix-env -i. However, it has to evaluate all pname attributes in nixpkgs to find matching packages. So it is generally better to use nix-env -iA and specify an attribute, because then Nix only has to evaluate the attribute (and any transitive dependencies). This saves time and memory. That said, it is even better to not use nix-env at all, because it’s a form of imperative package management. Specifying the packages in configuration.nix or using something like home-manager allows to to set up a system or environment declaratively.

This is a historical artifact. In the past the name attribute was used for the package name plus version. Now name is generated from pname and version. But not all derivations have been updated yet to use pname + version.

As mentioned above. This is generated from pname and version and should not be specified anymore in new derivations.

Yes, since the machine could have overlays or overrides defined. This is one one of the issues that flakes attempt to solve (avoiding that external state influences the evaluation).

Nix evaluation is pure (with some exceptions, such as built-in fetchers).

Yes, assuming there are no overrides or overlays.

Yes, though unlikely to happen with the python attribute (AFAIK). But other attributes get updated (e.g. gcc is gcc9 now, but will be bumped to gcc10 in the future).

6 Likes

Thanks for the quick response, its incredibly helpful.

For context, I’m working on this issue for building a better search for searching archived versions since that’s blocking many people from porting their project to nix.

This is the only comment where I didn’t mostly understand the answer. Assuming a fresh install, no configuration.nix or user code/config of any kind, is the answer still yes?
Like version = stdenv.isDarwin, would be allowed by the system? (even if not approved in a PR)

Yes, assuming there are no overrides or overlays.

So are pname and version required fields now?

Haha well yeah :laughing: those exceptions are kinda the whole part I’m interested in.
Is something like pname = stdenv.fetch ... allowed?

Good, for a second there you had me thinking that absolutely everything in nix was potentially an impure function of the user’s environment. If at least this is constant I can start working with something.

Oh I very much disagree. I mean you can’t achieve nix’s goal of reproducibility without meticulously managing versions. Rollbacks, pinning, hashes, they only exist to manage versions. Nix manages versions like Snowden keeping track of his laptop at Defcon. Nix just does a bad job at finding/switching between all the available versions.

Now apt and brew are not version managers. They don’t know exactly what is installed (packages are allowed to update themselves). You cannot always specify a version, and you sure as hell can’t ask it to install two versions of the same package.

1 Like

Generally we prefer to be “at the latest version” unless there’s compelling reasons not to, which why you see things like python3{6,7,8,9,10} available.

I think what @danieldk is trying to say that we don’t try to capture all versions. But as a user, you’re free to pin versions however you want in an overlay.

Not an exact list of all derivations using “name =” , doesn’t seem like a simple search and replace task…

search for the following in hound

^\s*name = "[^"]+";

https://search.nix.gsc.io/?q=name%20%3D%20"&i=nope&files=.nix&repos=NixOS-nixpkgs

Maybe I can come up with a better regex to get better matches if i have time. might be tricky if nix is not regular … :wink:

The compelling reason is that numerous projects require a specific version of a dependency (consider glibc) in order to function.

An example of that which has recently come up is glibc. If you have a binary which needs to run on ubu 18.04 or later, you need to link with glibc-2.27. Linking with any later version means your binary will not function on that platform.

nix with version 20.03 links glibc-2.32 by default. The latest distros of fedora and ubuntu run 2.31. So a binary built with nix will not run with any common distro even after patchelf.

Very nice. It would be good to be able to say

      buildInputs = [ glibc_2_27   ]   # etc

and have the program work.

Currently that is not possible.

FWIW, all of this is irrelevant of the projects goal is that nix builds packages specifically for nixos, and nixos only runs projects that have been designed to build on nixos.

If it is an explicit goal that nix function as a package manager for debian and redhat systems (not to mention darwin, arch etc) then the package versions are absolutely central to that function.

Well, that may not be a great idea anyway, since compiled binaries/libraries will have Nix store paths embedded. Even if you patch out the Nix store paths of dynamic libraries using patchelf, the binaries can contain further references to the Nix store.

I guess the only way to build somewhat compatible binaries is to use buildFHSUserEnv and then build the project in a more traditional manner.

A bit off-topic, but Fedora (33) currently uses 2.32 as well.

If you use Nix as a package manager on Debian and Red Hat systems, then there is no problem, since packages built using Nix will use glibc from the Nix store, not the system’s glibc.


At any rate, if you want to deploy on systems without Nix, then you should probably create a Docker image with dockerTools.buildLayeredImage and deploy that or build the program in an Ubuntu/Red Hat/whatever container using Docker.

1 Like

how did i find this?

# generate error message
nix-build -E 'with import <nixpkgs> { }; callPackage (stdenv.mkDerivation { name = ", invalid"; }) { }' 
error: store path 'xxxx-, invalid' contains illegal character ','

# get nix source
git clone https://github.com/NixOS/nix --depth 1

# find error message
grep -rHn 'contains illegal character' nix/
nix/src/libstore/path.cc:16:            throw BadStorePath("store path '%s' contains illegal character '%s'", path, c);
3 Likes