Understand nixpkgs structure / semantic

Help
1 
pkgs= import <nixpkgs>  { config= { allowUnfree = true; }; } 
pkgs.python38Packages.pytorch.pname 
pkgs.python38Full.pkgs.pytorch.pname # pkgs.python38.pkgs.pytorch.pname
pkgs.python3.pkgs.pytorch.pname      # pkgs.python3.pythonVersion # "3.8"

pkgs.python.pkgs.requests.pname

why is the semantic different if it is the same thing/set?

  • is it the same? is one referencing to the other? (is it somewhere documented?)

pkgs.rPackages.zoo.name

why is the semantic different for different languages?

  • no pname (only name)
  • no pkgs or R${version}Packages

why are there no R version variants (like in python[39] )?

3 Likes
2

python38Packages is short for python38Full.pkgs. Probably similar with pkgs and nixPkgs.

Can’t answer you anything about R but the simple answer is that no one did it.

3

Hello @Sandro,

is it somewhere documented?

If python38Packages it is a short hand for python38Full.pkgs

  • why is only python38 working in the overlay?
overlays = [ ( self: super: {
            python38Full = super.python38Full.override {
            #python38 = super.python38.override {
              packageOverrides = python-self: python-super: {
                pytorch = python-super.pytorch-bin.overrideAttrs (oldAttrs: {
                  pname = "pytorch_bin";
                });
              };
            };
          }) ]
pkgs= import <nixpkgs>  { inherit overlays; }
pkgs.python38Packages.pytorch.pname
# "pytorch"

pkgs= import <nixpkgs>  {}
pkgs.python38.pkgs.pytorch      ==  pkgs.python38Packages.pytorch
# true
pkgs.python38Full.pkgs.pytorch  ==  pkgs.python38Packages.pytorch
# false

pkgs= import <nixpkgs>  { } 
PyF    = pkgs.python38Full.pkgs
PyB    = pkgs.python38.pkgs
PyPkgs = pkgs.python38Packages

PyF == PyB    # false
PyF == PyPkgs # false
PyB == PyPkgs # false
PyF == PyF    # error: enum-0.4.7 not supported for interpreter python3.8

ps: pkgs and nixPkgs (was the same)

1 Like
4

I also need support for multiple versions of R. My plan is to have an overlay defining r35, r36, r40, etc. Another thing is I need to be able to create sets of R packages as of different MRAN dates, so I extended the code from nixpgks to have the date as an argument to the function that creates derivations for the packages (and allows to use my internal mirror of MRAN).

5

Like @Sandro, I don’t know the R ecosystem, but I guess there is only one package set because there was no need for it until now?

Anyway @alexv, If you can upstream your overlay so that we propose rXXPackages, I suppose it will help people also.

6

looks like there are some issues with python38Full

7

For the usual reasons:

  • developed by different people
  • developed at a different time
  • different needs
  • lack of time and coordination to unify it

Semantics is a part of best practices which is something we’re all still learning about.

8

How can some know that pkgs.python38Full.pkgs links to pkgs.python38Packages (if it is like that)?

  • and how could you trace back how python3xFull is created (the link in Nixos Pkgs only links to cpython)?
9

@Sandro

I tried to verify your statement
for now the test would tell me that

  • pkgs.python38Packages[.pandas] is NOT linked to/short for pkgs.python38Full.pkgs[.pandas]

Using python38 → pkgs.python38Full.pkgs don’t get recognized.
(I tested different constellations)

{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz) {} }:
#{ pkgs ? import <nixpkgs> {} }:
let
  #python3 = pkgs.python38.buildEnv.override {
  python3 = pkgs.python38Full.buildEnv.override {
    extraLibs = [ 
        pkgs.python38Packages.pandas 
        #pkgs.python38Full.pkgs.pandas 
        ];
  };
in let
envPy =  pkgs.mkShell {
    name = "shellPy";
    buildInputs = [ python3 ];
    shellHook = ''
                echo ""
                ${python3}/bin/python -c "import pandas as pd; print(pd.__version__) "
                '';
} ;
in 
envPy

results in:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'

I would like to investigate that situation a step deeper but don’t know what methods/tools there are in nixos/nix to do so.

  • please help
10

@freezeboy My plan is basically resurrect the derivations for older versions of R buried in the nixpgks git history. The main potential complication as I see it now is the recent transition to the pluggable implementations of BLAS and LAPACK which older derivations for R don’t support. BTW this is currently broken for R (R built with MKL computes incorrect results · Issue #104026 · NixOS/nixpkgs · GitHub). I haven’t been able to figure out what the problem is and there is no other activity in the github issue.

11

Yeah, it is something different but it evals for me in a nix-shell.

12

Hello @Sandro,
is there a way in nix to show up the differences in those derivatives?

What is the best method of getting down to the root case of those and other errors in nix/nixpkgs?

13

I don’t know one but there is probably one.

14

Maybe nix-diff: Explain why two Nix derivations differ could help - nix-shell -p haskellPackages.nix-diff

1 Like
15

Why do packages like

  • avahi
  • cntk # it get created from the application
  • mxnet # the application has a pname - the package not
    don’t have a pname 
nixpkgs =  import <nixpkgs>  { }
nixpkgs.pkgs.python38.pkgs.mxnet.pname 
nixpkgs.pkgs.mxnet.pname

why do packages like

  • opnecv4 or
  • pyside2-tools

don’t have an overridePythonAttrs (so based on the appication opencv / pyside2 )

Is it better to create an application from an python package or a package from an application?

16

Because no one converted them yet. If you require them to be converted feel free to send a PR.