pythonPackages39.netcdf4 doesn't build x86_64-linux

[Kiyoshi364]

Hi,

I’m new to nix.
I’m trying to run nix-shell --pure -p python39Packages.netcdf4
I expected to enter in a shell, but I got an error while building (at the end because it is big).

I have read this github issue, I’m not sure if it’s related:
https://github.com/NixOS/nixpkgs/issues/128717
I think that it’s the same package (maybe a different version) that also have tests problem.

Is there any way to fix this in my machine and/or upstream?
Can I try to build it in my machine without the tests?

I’m on Arch-linux with x86_64 architecture x86_64.

$ nix-shell --version
nix-shell (Nix) 2.13.2

I think that I do have enough disk space, because of:

$ btrfs filesystem df /
Data, single: total=18.01GiB, used=12.92GiB
System, DUP: total=8.00MiB, used=32.00KiB
Metadata, DUP: total=2.00GiB, used=748.81MiB
GlobalReserve, single: total=63.78MiB, used=0.00B

(This is here because note: build failure may have been caused by lack of free disk space)

This is the part that I think it is relevant of the output (also the tail of the output):

......................../build/netCDF4-1.6.2/test/tst_types.py:88: UserWarning: WARNING: missing_value not used since it
cannot be safely cast to variable data type
  assert_array_equal(v3[:],-1*np.ones(n2dim,v3.dtype))
........................................./build/netCDF4-1.6.2/test/tst_masked.py:93: UserWarning: WARNING: missing_value not used since it
cannot be safely cast to variable data type
  data = dataset['v'][:]
....................compressed lossy with shuffle and standard quantization =  187767  max err =  0.0002441403696464839
compressed lossy with shuffle and alternate quantization =  155389  max err =  0.00048828088386476676
compressed lossy with shuffle and alternate quantization =  163033  max err =  0.0002441403696464839
......EEEsE.
======================================================================
ERROR: runTest (tst_cdf5.test_cdf5)
testing NETCDF3_64BIT_DATA format (CDF-5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netCDF4-1.6.2/test/tst_cdf5.py", line 20, in setUp
    v[:ndim] = arrdata
  File "src/netCDF4/_netCDF4.pyx", line 5519, in netCDF4._netCDF4.Variable.__setitem__
  File "src/netCDF4/_netCDF4.pyx", line 5802, in netCDF4._netCDF4.Variable._put
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Operation not allowed in define mode

======================================================================
ERROR: runTest (tst_atts.VariablesTestCase)
testing attributes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netCDF4-1.6.2/test/tst_atts.py", line 43, in setUp
    with netCDF4.Dataset(self.file,'w') as f:
  File "src/netCDF4/_netCDF4.pyx", line 2463, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2026, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'tst_atts.nc'

======================================================================
ERROR: test_reports_alignment_capabilities (tst_alignment.AlignmentTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netCDF4-1.6.2/test/tst_alignment.py", line 36, in setUp
    f = Dataset(self.file, 'w')
  File "src/netCDF4/_netCDF4.pyx", line 2463, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2026, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/build/tmpbqvqkjfl.nc'

======================================================================
ERROR: test_version_settings (tst_alignment.AlignmentTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netCDF4-1.6.2/test/tst_alignment.py", line 36, in setUp
    f = Dataset(self.file, 'w')
  File "src/netCDF4/_netCDF4.pyx", line 2463, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2026, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/build/tmpbqvqkjfl.nc'

----------------------------------------------------------------------
Ran 97 tests in 50.968s

FAILED (errors=4, skipped=1)
error: builder for '/nix/store/rcz1clss9dlyrcxsdh3nq3xhnzd02zpl-python3.9-netCDF4-1.6.2.drv' failed with exit code 1;
       last 10 log lines:
       >   File "/build/netCDF4-1.6.2/test/tst_alignment.py", line 36, in setUp
       >     f = Dataset(self.file, 'w')
       >   File "src/netCDF4/_netCDF4.pyx", line 2463, in netCDF4._netCDF4.Dataset.__init__
       >   File "src/netCDF4/_netCDF4.pyx", line 2026, in netCDF4._netCDF4._ensure_nc_success
       > PermissionError: [Errno 13] Permission denied: b'/build/tmpbqvqkjfl.nc'
       >
       > ----------------------------------------------------------------------
       > Ran 97 tests in 50.968s
       >
       > FAILED (errors=4, skipped=1)
       For full logs, run 'nix log /nix/store/rcz1clss9dlyrcxsdh3nq3xhnzd02zpl-python3.9-netCDF4-1.6.2.drv'.
       note: build failure may have been caused by lack of free disk space

[nixinator]

What channel are you on

nix-channel --list or sudo nix-channel --list

nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'

If it’s pulling it from source , it means hydra may have failed to compile it or hydra has not got around to compiling it.

You may have to resort to turning the tests off, if they are just failing in the nix sandbox. I wouldn’t recommend it however.

give

nix-shell -p hydra-check

hydra-check --channel unstable python38Packages.netcdf4 

Build Status for nixpkgs.python38Packages.netcdf4.x86_64-linux on unstable

✔ python3.8-netCDF4-1.5.8 from 2022-01-05 - https://hydra.nixos.org/build/163232939 ```
`

[Kiyoshi364]

$ nix-channel --list

(yes, no output)

$ nix-instantiate --eval -E '(import {}).lib.version'
error: cannot coerce a set to a string

       at «string»:1:2:

            1| (import {}).lib.version
             |  ^
(use '--show-trace' to show detailed location information)

$ nix-shell -p hydra-check

[nix-shell:~]$ hydra-check --channel unstable python38.Packages.netcdf4
Build Status for nixpkgs.python38.Packages.netcdf4.x86_64-linux on unstable
⚠️ This job is not a member of the latest evaluation of its jobset. This means it was removed or had an evaluation error.

[nix-shell:~]$ hydra-check --channel unstable python39.Packages.netcdf4
Build Status for nixpkgs.python39.Packages.netcdf4.x86_64-linux on unstable
⚠️ This job is not a member of the latest evaluation of its jobset. This means it was removed or had an evaluation error.

(I ran twice, because I’m trying to run version 3.9)

I’m guessing that I should add a channel and then try again.
Should I run one of those commands?

$ nix-channel --add https://nixos.org/channels/nixos-22.11 nixos22-11

$ nix-channel --add https://nixos.org/channels/nixos-unstable nixos-unstable

[Kiyoshi364]

Also, how can I disable the tests?

[nixinator]

Ok, i quoted the commands now, discourse was mangling it. sorry! try them again, and correct you post .

you machine has a ‘per user’ channel, which usually trips people when they are new users.

So it’s quite possible and normal for different users to have different channels.

How do I build a nix-shell which depends on some unstable packages?

I have not found need to import a particular git revision, but that is one alternative. I usually reference nixpkgs paths defined in NIX_PATH: import <unstable> {} or something. Is there a reason that either of these would not work? Could you take a package straight out of a nixpkgs channel, packages having dependencies from different nixpkgs versions, or do you need a package from unstable with its dependencies changed to the stable versions, for example?

should help you grab it from stable/unstable .

[Kiyoshi364]

I’m making a new post because I don’t like deleting information.

$ sudo nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable

$ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
"23.05pre455185.8c66bd1b68f"

$ nix-shell -p hydra-check

[nix-shell:~]$ hydra-check --channel unstable python38Packages.netcdf4
Build Status for nixpkgs.python38Packages.netcdf4.x86_64-linux on unstable
✔️ python3.8-netCDF4-1.5.8 from 2022-01-05 - https://hydra.nixos.org/build/163232939

[nix-shell:~]$ hydra-check --channel unstable python39Packages.netcdf4
Build Status for nixpkgs.python39Packages.netcdf4.x86_64-linux on unstable
✔️ python3.9-netCDF4-1.6.2 from 2023-01-13 - https://hydra.nixos.org/build/205196220

(I ran twice, because I’m trying to run version 3.9)

It magically got green, so I’ll try again. And didn’t work.

Also, I’m using a shell.nix file, it starts with

{ pkgs ? import <nixpkgs> {} }:

I read the link, how do I choose a tarball to fetch?
(I’ll probably fix the entire shell.nix to the same tarball)

[nixinator]

Ok, i quoted the commands now, discourse was mangling it. sorry!

you machine has a ‘per user’ channel, which usually trips people when they are new users.

So it’s quite possible and normal for different users to have different channels.

if you channels look like this

nix-channel --list

unstable https://nixos.org/channels/nixos-unstable

you can run a shell.nix

{ pkgs ? import <unstable> {} }:
pkgs.mkShell {
  buildInputs = [
        pkgs.python39Packages.netcdf4
  ];
}

beware you’ll have to login and logout to get $NIX_PATH updated!

using channels it’s a little hard to express simply as the mapping of channels vs imports isn’t great IMHO.

Selecting a channel with nix-shell

I’d recommend to avoid using channels for the following reasons: https://github.com/nix-dot-dev/nix.dev/issues/16#issuecomment-713701415

Flakes make this a little easier , as you basically define everything with a single flake.nix and don’t have all these imperative steps like nix-channels.

however, hydra-check is showing that the package has been built successfully by hydra!

I’m sorry that this is so confusing, this why imperative channels need to go!

[Kiyoshi364]

Looks like I’ve got it solved.

This worked for me:

{ pkgs ? import (fetchTarball "channel:nixos-22.11") {} }:

(Now I’m fixing the tarball, it’s here to help others and maybe future me)

Then I went https://status.nixos.org/ and got a commit hash that I liked (nixos-22.11).
The following “headers” worked

let
    nixpkgs = import <nixpkgs> {};
    tarball = nixpkgs.fetchFromGithub {
            owner = "NixOS";
            repo = "nixpkgs";
            rev = "put the commit hash here";
            sha256 = "the first time it will fail because of sha missmatch, then copy the right one";
        };
in
{ pkgs ? import tarball {} }:

This is a bit smaller:

let
    commit = "put the commit hash here";
    tarball = fetchTarball "https://github.com/NixOS/nixpkgs/archive/${commit}.zip";
in
{ pkgs ? import tarball {} }:

Is there a better way to do that?

Thanks for the help. Links helped a lot.

Should I do something to close this?

[nixinator]

you can tell discourse this has a solution and thats it!

as you can see , you can write a nix-shell or any piece of nix code in a completely un-reproducable way.

Channels are confusing, they are imperative… the tools that manage them could be so much better

however, i hope your starting to see why the new flake tooling is so important. it makes you declare all inputs, and it also locks all inputs.

So they are much more shareable as they don’t rely on some magic channel state , which differs from my machine to yours.

I hope that makes sense, and you can try flakes out soon.

The flake registry has a danger of recreating the niggles with channels, as my flake registry could be different to yours … thus causing all sort or reproduciblity problems.

[Kiyoshi364]

It makes sense, I’ll try flakes soon.