2024-04-03 Nix team meeting minutes #135

thufschmitt · April 8, 2024, 3:29pm

nix-prefetch-url seems to uncompress archive if `content-encode: gzip` is set · Issue #10254 · NixOS/nix · GitHub

github.com/NixOS/nix

nix-prefetch-url seems to uncompress archive if `content-encode: gzip` is set

opened 03:32AM - 16 Mar 24 UTC

closed 07:06PM - 03 Apr 24 UTC

ghost

bug

**Describe the bug** nix-prefetch-url is downloading and uncompresing the fil…e sometimes. this appears to happen if the download is uses gzip content encoding. when `content-encode: gzip` is not set (and thus the download not doubly compressed) nix-prefetch-url works as expected. **Steps To Reproduce** ```sh nix-prefetch-url --print-path https://releases.warp.dev/stable/v0.2024.03.12.08.02.stable_01/warp-terminal-v0.2024.03.12.08.02.stable_01-1-x86_64.pkg.tar.zst ``` ```console 0gwji3934p4c0qmfyirjwyg7dy1rhdlyrvj2x2c2xwllpxd89pw9 /nix/store/biisn805rah1s25smdzpcd4hxqm0vpw3-warp-terminal-v0.2024.03.12.08.02.stable_01-1-x86_64.pkg.tar.zst $ file /nix/store/biisn805rah1s25smdzpcd4hxqm0vpw3-warp-terminal-v0.2024.03.12.08.02.stable_01-1-x86_64.pkg.tar.zst /nix/store/biisn805rah1s25smdzpcd4hxqm0vpw3-warp-terminal-v0.2024.03.12.08.02.stable_01-1-x86_64.pkg.tar.zst: POSIX tar archive ``` expect that the file is zstd compressed. **Expected behavior** expect that the downloaded file is compressed and the hash is calculated against the compressed file. **Additional context** https://github.com/NixOS/nixpkgs/issues/295553#issuecomment-1999896284 tried versions `nix-prefetch-url (Nix) 2.19.3` and `nix-prefetch-url (Nix) 2.17.1` and get same results. also happens on the setup used by r-ryantm. **Priorities** Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).

Cannot invalidate cache of files fetched in pure evaluation mode · Issue #9814 · NixOS/nix · GitHub

Same as Cannot invalidate cache of files fetched in pure evaluation mode · Issue #9814 · NixOS/nix · GitHub

`nix build -f. pkg1 ... pkg100` is 10x slower than a build with single expression · Issue #9724 · NixOS/nix · GitHub

github.com/NixOS/nix

`nix build -f. pkg1 ... pkg100` is 10x slower than a build with single expression

opened 10:53PM - 08 Jan 24 UTC

trofi

bug new-cli performance

**Describe the bug** I noticed the `nix build -f.` slowdown when did mass reb…uilds locally worth of 3000 packages. I expected speedy build startup and instead had to wait minutes for a startup. **Steps To Reproduce** Prerequisite: build a list of first 100 packages in `nixkgs: ``` $ nix-instantiate --strict --eval --expr 'with import ./. {}; map (p: p.name) (lib.take 100 (lib.filter (p: let e = builtins.tryEval (lib.isDerivation p.value); in e.success && e.value) (lib.attrsToList pkgs)))' [ "AMB-plugins" "ArchiSteamFarm" "AusweisApp2" "BeatSaberModManager" "CHOWTapeModel" "ChowCentaur" "ChowKick" "ChowPhaser" "CoinMP" "DisnixWebService" "EBTKS" "EmptyEpsilon" "FIL-plugins" "Fabric" "HentaiAtHome" "LAStools" "LASzip" "LASzip2" "LibreArp" "LibreArp-lv2" "Literate" "MACS2" "MIDIVisualizer" "MMA" "NSPlist" "OSCAR" "OVMF" "OVMFFull" "PageEdit" "PlistCpp" "QuadProgpp" "R" "SDL" "SDL2" "SDL2_gfx" "SDL2_image" "SDL2_image_2_0_5" "SDL2_mixer" "SDL2_mixer_2_0" "SDL2_net" "SDL2_sound" "SDL2_ttf" "SDL_Pango" "SDL_audiolib" "SDL_classic" "SDL_compat" "SDL_gfx" "SDL_gpu" "SDL_image" "SDL_mixer" "SDL_net" "SDL_sixel" "SDL_sound" "SDL_stretch" "SDL_ttf" "SP800-90B_EntropyAssessment" "SPAdes" "SkypeExport" "StormLib" "Sylk" "ViennaRNA" "Xaw3d" "_0verkill" "_0x" "_1oom" "_1password" "_1password-gui" "_1password-gui-beta" "_2048-cli" "_2048-cli-curses" "_2048-cli-terminal" "_2048-in-terminal" "_20kly" "_2bwm" "_3270font" "_389-ds-base" "_3llo" "_3mux" "_3proxy" "_4th" "_4ti2" "_5etools" "_6tunnel" "_7kaa" "_7zz" "_86Box" "_90secondportraits" "_915resolution" "_9base" "_9menu" "_9pfs" "_9ptls" "a2jmidid" "a2ps" "a4" "a4term" "a52dec" "aaa" "aaaaxy" "aacgain" ] ``` Good (a single expression build): ``` $ time NIXPKGS_ALLOW_UNFREE=1 nix build --impure --expr 'with import ./. {}; runCommand "foo" { buildInputs = [ AMB-plugins ArchiSteamFarm AusweisApp2 BeatSaberModManager CHOWTapeModel ChowCentaur ChowKick ChowPhaser CoinMP DisnixWebService EBTKS EmptyEpsilon FIL-plugins Fabric HentaiAtHome LAStools LASzip LASzip2 LibreArp LibreArp-lv2 Literate MACS2 MIDIVisualizer MMA NSPlist OSCAR OVMF OVMFFull PageEdit PlistCpp QuadProgpp R SDL SDL2 SDL2_gfx SDL2_image SDL2_image_2_0_5 SDL2_mixer SDL2_mixer_2_0 SDL2_net SDL2_sound SDL2_ttf SDL_Pango SDL_audiolib SDL_classic SDL_compat SDL_gfx SDL_gpu SDL_image SDL_mixer SDL_net SDL_sixel SDL_sound SDL_stretch SDL_ttf SP800-90B_EntropyAssessment SPAdes SkypeExport StormLib Sylk ViennaRNA Xaw3d _0verkill _0x _1oom _1password _1password-gui _1password-gui-beta _2048-cli _2048-cli-curses _2048-cli-terminal _2048-in-terminal _20kly _2bwm _3270font _389-ds-base _3llo _3mux _3proxy _4th _4ti2 _5etools _6tunnel _7kaa _7zz _86Box _90secondportraits _915resolution _9base _9menu _9pfs _9ptls a2jmidid a2ps a4 a4term a52dec aaa aaaaxy aacgain ]; } "touch $out"' --dry-run ... real 0m3,481s user 0m2,216s sys 0m0,466s ``` Bad (`nix build -f.`): ``` $ time NIXPKGS_ALLOW_UNFREE=1 nix build -f. --no-link AMB-plugins ArchiSteamFarm AusweisApp2 BeatSaberModManager CHOWTapeModel ChowCentaur ChowKick ChowPhaser CoinMP DisnixWebService EBTKS EmptyEpsilon FIL-plugins Fabric HentaiAtHome LAStools LASzip LASzip2 LibreArp LibreArp-lv2 Literate MACS2 MIDIVisualizer MMA NSPlist OSCAR OVMF OVMFFull PageEdit PlistCpp QuadProgpp R SDL SDL2 SDL2_gfx SDL2_image SDL2_image_2_0_5 SDL2_mixer SDL2_mixer_2_0 SDL2_net SDL2_sound SDL2_ttf SDL_Pango SDL_audiolib SDL_classic SDL_compat SDL_gfx SDL_gpu SDL_image SDL_mixer SDL_net SDL_sixel SDL_sound SDL_stretch SDL_ttf SP800-90B_EntropyAssessment SPAdes SkypeExport StormLib Sylk ViennaRNA Xaw3d _0verkill _0x _1oom _1password _1password-gui _1password-gui-beta _2048-cli _2048-cli-curses _2048-cli-terminal _2048-in-terminal _20kly _2bwm _3270font _389-ds-base _3llo _3mux _3proxy _4th _4ti2 _5etools _6tunnel _7kaa _7zz _86Box _90secondportraits _915resolution _9base _9menu _9pfs _9ptls a2jmidid a2ps a4 a4term a52dec aaa aaaaxy aacgain --dry-run ... real 0m31,570s user 0m21,679s sys 0m3,392s ``` 30s vs 3s is a 10x slowdown. **Expected behavior** Ideally `$ nix build -f.` should work as fast as a single expression. **`nix-env --version` output**: nix-env (Nix) 2.19.2 **Priorities** Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).

Nix fails to remount the store with all the required flags · Issue #9705 · NixOS/nix · GitHub

github.com/NixOS/nix

Nix fails to remount the store with all the required flags

opened 01:01AM - 07 Jan 24 UTC

gdamjan

bug idea approved store

Normally `/nix/store` is bind-mounted as read-only. That's to avoid accidental c…hanges to the Nix store even by root. `nix`/`nix-daemon` though try to remount the nix-store to read-write in its own namespace, but the implementation is simplistic[1] and will fail in different scenarios. [1] https://github.com/NixOS/nix/blob/master/src/libstore/local-store.cc#L581-L582 I'm running nixos as a `systemd-nspawn` container, where the whole nixos filesystem lives on the host system (in `/var/lib/machines/nixos`), and the container is then run with it as a root filesystem ("booting" `/sbin/init` which execs `/nix/store/d842nhr0ckaw7ijlf54q9dnm37dnn00p-nixos-system-nix-container-24.05.20240102.bd645e8/init`). In that case, the root and the nix-store are mounted with some additional flags compared to "normal" nixos (nodev and idmapped): ``` $ findmnt / TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/root[/var/lib/machines/nixos] ext4 rw,nodev,relatime,idmapped $ findmnt /nix/store/ TARGET SOURCE FSTYPE OPTIONS /nix/store /dev/mapper/root[/var/lib/machines/nixos/nix/store] ext4 ro,nodev,relatime,idmapped ``` The problem is nix(nix-daemon) fails to remount the store to read-write: ``` $ strace -f -e mount nix-daemon strace: Process 206 attached accepted connection from pid 230, user root (trusted) strace: Process 239 attached [pid 239] mount(NULL, "/nix/store", 0x7fa5ce409bd5, MS_REMOUNT|MS_BIND, NULL) = -1 EPERM (Operation not permitted) unexpected Nix daemon error: error: remounting /nix/store writable: Operation not permitted ``` The most simple hack to solve the issue was to add the MS_NODEV flag in the remount call: ```diff diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 0f3c37c8a..ce6dd13ae 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -579,7 +579,7 @@ void LocalStore::makeStoreWritable() throw SysError("getting info about the Nix store mount point"); if (stat.f_flag & ST_RDONLY) { - if (mount(0, realStoreDir.get().c_str(), "none", MS_REMOUNT | MS_BIND, 0) == -1) + if (mount(0, realStoreDir.get().c_str(), "none", MS_REMOUNT | MS_BIND | MS_NODEV, 0) == -1) throw SysError("remounting %1% writable", realStoreDir); } #endif ``` after this change, nix/nix-daemon successfully can remount the /nix/store. Now a more robust solution would probably be to first introspect all the mount flags and then only flip the read-only flag - other `mount` cli tools seem to do that (busybox or util-linux mount for ex). I can try to pursue that fix, if it's acceptable.

nix creates processes in cgroup inner nodes · Issue #9675 · NixOS/nix · GitHub

Approved the idea, open for contributions

Linked Create per-derivation cgroup for better resource control · Issue #10374 · NixOS/nix · GitHub to this as that feature request is somewhat related.

possible regression: "stack overflow (possible infinite recursion)" · Issue #9672 · NixOS/nix · GitHub

github.com/NixOS/nix

possible regression: "stack overflow (possible infinite recursion)"

opened 12:41PM - 28 Dec 23 UTC

NobbZ

bug regression fetching

**Describe the bug** When trying to `nix build gitlab:nobodyinperson/annextim…elog` with a recent build from `master`, an error is printed: ```text $ nix build gilab:nobodyinperson/annextimelog copying '/nix/store/p418a043fs5smlx6cgj9z4q57kxlmym8-source' to the storeerror: stack overflow (possible infinite recursion) ``` 2.18.1 worked fine and produces a proper `result` link. I am currently trying to bisect the issue. Which is not quite easy, as the error only occurs, when parts of the result are not available in the store, and I have not yet figured out what exactly needs to get removed, so I have to do a full GC each cycle (and building nix takes another huge amount of time) **Steps To Reproduce** 1. `nix build gilab:nobodyinperson/annextimelog` with a nix "nightly" 2. See error **Expected behavior** A `result` links is created, like it got created during the 2.18.1 **`nix-env --version` output** ``` $ nix --version nix (Nix) 2.20.0pre20231227_a21c762 ``` **Additional context** So far this one flake is the only that made me have the problem. **Priorities** Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).

Release schedule

Agreement on only supporting the latest release and Nixpkgs version
Release candidates?
- @thufschmitt: would have that to get cleaner final releases
- @Ericson2314: would rather improve the testing story, would like to have the foundation spend money on this
- @edolstra: Not convinced by the usefulness because people can already try master

Port Nix-Perl to Meson by p01arst0rm · Pull Request #10378 · NixOS/nix · GitHub

@Ericson2314: Reviewing it, fully onboard modulo some small things, but would like approval from another member
@thufschmitt: Let’s go!
- The perl bindings are hardly changed, so easy to revert
@edolstra: Not convinced, knows the old build system but not Meson

Aleksanaa · April 9, 2024, 6:07pm

But a look into the calendar says it is already 2024

fricklerhandwerk · April 9, 2024, 6:42pm

Thanks, fixed the typo.

2024-04-03 Nix team meeting minutes #135

nix-prefetch-url seems to uncompress archive if `content-encode: gzip` is set · Issue #10254 · NixOS/nix · GitHub

Cannot invalidate cache of files fetched in pure evaluation mode · Issue #9814 · NixOS/nix · GitHub

`nix build -f. pkg1 ... pkg100` is 10x slower than a build with single expression · Issue #9724 · NixOS/nix · GitHub

Nix fails to remount the store with all the required flags · Issue #9705 · NixOS/nix · GitHub

nix creates processes in cgroup inner nodes · Issue #9675 · NixOS/nix · GitHub

possible regression: "stack overflow (possible infinite recursion)" · Issue #9672 · NixOS/nix · GitHub

Nix aborts updating with an assertion failure. · Issue #10360 · NixOS/nix · GitHub

'nix build' builds the wrong package when you run it from a store path · Issue #10372 · NixOS/nix · GitHub

nix shell fails if a derivation output is a symlink · Issue #10375 · NixOS/nix · GitHub

Release schedule

Port Nix-Perl to Meson by p01arst0rm · Pull Request #10378 · NixOS/nix · GitHub