Questions after a massive MASSIVE channel update

Help
1

I have those channel points:

… previous channel points

lrwxrwxrwx 1 root root 60 2 avril 14:08 channels-165-link → /nix/store/7c51c3kxjs539c59qbfl5s3q809d68m7-user-environment
lrwxrwxrwx 1 root root 60 6 avril 13:12 channels-166-link → /nix/store/whqjzg9zxa7iyw2pcblrvnkli6zhv8b1-user-environment

There was a few days before the 6th april, where there was no channel update points available for channels

After trying to nixos-rebuild with channel point 166: at first my system exploded in size. Not the first time it happens… but this time it is really huge

When I look at the store now. (to get a idea where the space is consumed)… I clearly notice that almost ALL ALL ALL my packages, although the same version, have a different hash…

I have not looked at each and every packages, but I had the idea of making a script so as to have the ratio of duplicated same-version but a different nix-store hash. Not written the script yet, but I may do it.

Saying it is 100% of the packages is false. I do not know if it is wrong to say that it is > 95% of the packages that have been sort of dupplicated.

I do not have knowledge about nixos enough.

What would you do so as to ensure those are legit new derivations? For example nixos sort of dispatched a correction that impacted almost all of its packages for a reason or another

Honestly I thought about a man-in-the-middle attack where all my system was hacked onto a dubious altered version. I guess my ignorance is the source of my fears… but it is hard to imagine that a correction on the nixos side may have impacted so many packages all at once.

If you have pointers/references for reading about it. Other post or texts dealing by those kind of events. I often have such problem when switching landmark versions, e.g. from 25.05 to 25.11: but then it makes sense that most of the packages change, or may have slight modification while targeting the same code…

Here it feels super strange. So I want to know more.

What would you do, if you were at my place, in my situation: smoke tests, rapid checks, stuffs to look at in nixpkgs, etc…

I am not yet hands in the earth & mud regarding nixos and nixpkgs development. So drop me advices, they are all welcome.

REMARK: I thought it was a nixos development question - may need reconsideration by moderators

1 Like
2

As a rule of thumb:

The store is correct. This is how things look after a staging merge, which happens about every 2 weeks.

1 Like
3

I prefer rules of compilers over rules of thumb :slight_smile: I am clearly wanting to KNOW what is changing the hashes. I know my thumbs very well :wink:

For me a hash reflects the over string-content of the files. If a hash is different, it means something in the code has change: something in the code of derivations.

And it has impacted my packages at > 95%.

For example if all packages *.nix files were changed, I could accept that the resulting derivations are all changed. That would totally make sense.

4

Then use nvd, nix-diff or similar to identify changes between old and new system closures, probably some compiler or a library close to the graphs roots changed and caused an effective rebuild of everything.

But keep in mind that the store contains more than just what is in the system. There’s usually a lot of stuff from devshells or ephemeral uses in the store.

Nothing is broken here, and unless there is no problem, there really is no problem.

1 Like
5

Sorry I had edited my answer, with more content

6

Nix is input addressed, so if the hash changed, something leading to what you get out changed.

7

You got it. That’s why I asked. For more than 95% of my packages something has changed. I will write my script for computing the ratio. I think I will learn in the process.

And I will look at nvd, nix-diff

They may reveal what I need to understand

8

Also. Do not interpret this anwser: I am just factual
I do not share at all your naive acceptation of the state of the nix store.

Understand: NixOS is my defacto mono-choice and I do not plan to change for another distro in many years to come. (probably Guix for culture and other reasons, I will look at it, but probably not adopt it).
I will keep Nixos as my fundation. For this reason I am particular “unforgiving” when there are issues. Because I love the program/ecosystem.

When you tell me “there is no issue with the store/ if there is no problem then there is no problem.”

Look, I have a surface m4 rewritting system that manages my configuration. I can add more notes, I feel more at home, I write comments.
All of this packages were in the nix store, working well at some moment/point in time
So I have in my own configuration history many many quirks that can happen

->rg borken
parsing.nix.m4
122:# haskellpkgs               . parsec-parsers   # borken in 2305
dev-lisp.nix.m4
52:# RELEAS(lisp, s7)                         # meant to be embeded # borken in 2505
database.nix.m4
21:# haskellpkgs.table                    # table is a tool to format CSV files into ASCII tables # borken in 2305
83:# RELEAS(database, dragonflydb                                   # modern replacement for redis and memcached  # borken 2305)
114:SUBPKG(python, sqlitedict)                                       # persistent threadsafe dict using sqlite # borken for now
285:# SUBPKG(haskell, postgresql-orm)             # borken in 2505
290:# SUBPKG(haskell, pg-entity)                  # haskell binding for postgresql entity # borken because pg-transact
292:# SUBPKG(haskell, pg-transact)                # monad for postgresql transactions # borken in 2505
biology.nix.m4
10:# SUBPKG(haskell, seqloc-datafiles) # currently borken
multimedia-audio.nix.m4
154:# RELEAS(multimedia)   zam-plugins                 # compilation is borken for now
198:# RELEAS(multimedia, fmsynth)                                             # borken in 2411)
dev-python.nix.m4
22:# RELEAS(python, codon)              # LLVM compiler for python          # borken in 2511
104:# SUBPKG(python, cexprtk)             # compilation borken in new python version
dev-javascript.nix.m4
10:# SUBPKG(node, jsonlint)                 # broken in 2511
dev-lisp.nix.m4
5:# SUBPKG(sbcl, cl-termbox)                 # readline replacement for sbcl # broken in 2505
67:# RELEAS(lisp, chickenPackages_5.chickenEggs.sq)                  # broken in 2505
dev-sml.nix.m4
8:# RELEAS(sml, manticore  # a parallel pure variant of Standard ML # broken in 2411)
nlp.nix.m4
40:# SUBPKG(python, fastdiff)        # fastdiff is a Python library for computing the difference between two text files # currently broken in 2511
hash-crypto.nix.m4
19:# haskellpkgs.blakesum   # broken ? check again
34:# haskellPackages.skein      # broken ? check again
solver-others.nix.m4
51:# sbclPackages                             . odepack            # broken in 2305
53:# SUBPKG(r,sundialr)                     # R interface to sundials, a suite of libraries for solving ODEs and DAEs (Differential-Algebraic Equations). currently broken. check again.
56:# rpkgs                                    . r2sundials         # broken in 2305
76:# RELEAS(solver-others, openmodelica . omparser)                 # brokenin 2311)
database.nix.m4
13:# haskellpkgs                          . tables        # in memory database # broken in 2305
53:#haskellpkgs.BerkeleyDBXML             # haskell binding for berkely db xml   # broken # probably fixable since it is based on a former BerkeleyDBXML package
66:RELEAS(database, lmdb)                                           # broken 2305)
73:RELEAS(database, leveldb)                                        # fast and lightweight key value store # broken 2305)
75:# SUBPKG(sbcl, leveldb)                                          # broken in 2505
126:# SUBPKG(lua, sqlite)                                            # lua binding for sqlite , seems broken
129:# SUBPKG(haskell, sqlite-easy)                                   # primitive haskell binding for sqlite # broken
133:# SUBPKG(haskell, sqlite)                                        # seems broken
165:# RELEAS(database, neo4j-desktop)                                # gui for neo4j) # suddenly broken in 2511
167:# haskellpkgs                            . haskell-neo4j-client   # broken in 2411
188:# # haskellpkgs                          . redis-hs                 # redis client for haskell   # broken 2305
218:# RELEAS(database, migra)                     # diff for postgresql schema  # broken in 2505
223:# SUBPKG(python,psycopg2cffi)                 # broken in latest python version
254:# SUBPKG(postgre, timescaledb_toolkit)        # time series database for postgresql toolkit # broken in 2505
267:SUBPKG(postgre, age)                          # graph database for postgresql   # ok in postgre14 but broken after
275:# SUBPKG(postgre, cstore_fdw)                 # foreign data wrapper for postgresql to access columnar storage # broken
283:# SUBPKG(haskell,PostgreSQL)                  # broken in 2505
284:# SUBPKG(haskell, postgresql-pure)            # broken in 2511
288:# SUBPKG(haskell, psql)                       # postgresql client  # broken in 2505
293:# SUBPKG(haskell, pg-store)                   # broken
biology.nix.m4
9:# SUBPKG(haskell, a50)           # currently broken
multimedia-video.nix.m4
29:# SUBPKG(haskell, ffmpeg-light)              # broken in 25.05
dev-r.nix.m4
11:# SUBPKG(r, dataset)                        # datasets for R  # broken in 2505
42:# SUBPKG(r, shinydashboard)                 # dashboard framework for R                   # broken in nixos 2505 update 81
56:# rpkgs                           . SQDA  # maybe for qualitative data analysis # broken in 2311
solver-sat.nix.m4
22:# haskellpkgs                      . funsat         # modern DPLL SAT solver    # broken because of data-trim and bitset
dev-wayland.nix.m4
24:# RELEAS(wayland, vkdt-wayland)                      # seemsbroken in2311
p2p.nix.m4
19:PN(2411, jesec-rtorrent)          # broken in 2505
dev-wasm.nix.m4
29:# PN(2405, lunatic)               # erlang runtime for wasm # seems broken in 2411, because of a little upgrade in time library in cargo ; to check)
graphics.nix.m4
17:# SUBPKG(haskell, Michelangelo)                         # haskell interface for openGL) # broken
68:# SUBPKG(gimp3, resynthesizer)                          # broken in 2305
solver-cas.nix.m4
14:# RELEAS(solver-cas,         mathematica-cuda)       # broken in 2311
machine-learning.nix.m4
8:# SUBPKG(python, shap)                # broken in 2305
51:# SUBPKG(haskell, eigen)              # C++ linear algebra library    # broken 2305
62:# sbclPackages  . blas                # broken 2305 ?
63:# haskellpkgs   . blas                # broken 2305
93:# SUBPKG(r, treedata_table)            # treedata.table is an R package for fast manipulation of tree-structured data  # broken in update 81 nixos 2505
127:# SUBPKG(haskell, xgboost-haskell)  # broken in 2505
138:# vowpal-wabbit online learning # broken suddenly in 2405 , should be ok soon or in another version => unstable
142:# SUBPKG(haskell, vowpal-utils)                                 # vowpal-utils: Haskell bindings for Vowpal Wabbit # broken
144:# SUBPKG(r, RVowpalWabbit)                                      # currently broken
167:# SUBPKG(r,  prob)                # prob: Elementary probability on finite sample spaces # broken 2305
184:                                                                # seems to be a good friamework, but broken in 2405
188:# pythonpkgs                             . theano-pymc          # broken 2305
196:# OpenBUGS seems temporarilly broken in 2311
232:# RELEAS(LLM, lmstudio)             # can load models and interact with   # seems a bit broken / too new for now
dev-xml.nix.m4
82:# RELEAS(xml, yamlpath)            # xpath for yaml/json compatible data # broken 2311)
120:# SUBPKG(haskell, json2)             # broken
121:# SUBPKG(haskell, json-b)            # broken
122:# SUBPKG(haskell, JSONb)             # broken
123:# SUBPKG(haskell, xml2json)          # broken
document-markdown.nix.m4
33:# RELEAS(document-markdown, obsidian)                 # markdown editor for notes) # suddenlily broken in 2511
dev-erlang.nix.m4
13:# SUBPKG(haskell, erlang-ffi)                              # speak to erlang from Haskell Node  # broken in 2511
15:# SUBPKG(haskell, codec-beam)                            # erlang VM bytecode assembler # broken in 2511
document-convertion.nix.m4
3:# RELEAS(document-convertion, wkhtmltopdf   # convert html to pdf   # broken in 2405)
dev-shell.nix.m4
73:# haskellpkgs                 . procex         # shell scripting in Haskell # broken in 2505
terminal.nix.m4
9:# RELEAS(terminal,hyper)                                      # broken
org.nix.m4
10:# PN(2311, khoj)                  # for querying md and org notes and more  # broken
dev-prolog.nix.m4
17:# SUBPKG(haskell, prolog-graph)         # graph library for prolog # broken in 2305 cuz of openssl1 and swiProlog
20:# SUBPKG(haskell, prolog)               # prolog interpreter written in Haskell # broken in 2305
21:# SUBPKG(haskell, NanoProlog)           # broken since 1605 → 2305
30:# SUBPKG(haskell, datalog)              # broken in  current version ? # missing in 2003
33:# RELEAS(n2003, haskellPackages . souffle-dsl # broken since 2009 and missing in 2003)
web.nix.m4
68:# RELEAS(web, curl-impersonate-ff)           # curl able to impersonate chrome and ff # broken in 2511
80:# RELEAS(web, ladybird)                          # broken,but I was not really using it, so retry later)
104:PN(2411, slrn)                         # pinned because of broken c++ compilation in last version, check later for newer if needed
163:RELEAS(web, filezilla                ) # broken 2311
168:# pythonpkgs                        . flask-autoindex          # autoindex for flask, for big directories # broken in 2305
hardware.nix.m4
4:# RELEAS(hardware,            openrgb)                   # no longer know what it was doing => broken in 2505
multimedia-audio.nix.m4
44:# haskellpkgs                       . temporal-csound              # broken in 2305
45:# haskellpkgs                       . hCsound                      # broken in 2305
74:# haskellpkgs                       . SFML                       # broken in 2305
178:# RELEAS(multimedia, tunefish                                              # broken in 2411)
202:# RELEAS(multimedia, ams-lv2)                                             # broken in 2411)
hash-others.nix.m4
12:SUBPKG(haskell, hashes)                     # broken ?
15:# SUBPKG(haskell, crc32c) 		            # refuses to evaluate in2311 # broken
25:# SUBPKG(haskell, farmhash)                 # broken ? check again    # The farmhash package provides the FarmHash library for NixOS.
27:# SUBPKG(sbcl,  cl-libfarmhash)             # broken in 2305
30:# SUBPKG(haskell, murmurhash3)                # broken
document-latex.nix.m4
21:# RELEAS(latex, svg2tikz)                                  # issue in latest version , marked as broken
dev-dhall.nix.m4
19:# haskellpkgs                 . shake-dhall                         # 2405 broken
dev-forth.nix.m4
8:# haskellpkgs                 . miniforth     # broken in 2405
dev-haskell.nix.m4
34:#RELEAS(haskell, clean)                                         # broken
37:# SUBPKG(haskell, Frank)                # broken because of pkgs she-06
45:#SUBPKG(haskell, language-lua)                # haskell library for parsing and manipulating Lua code     #broken in 2511
62:# SUBPKG(haskell, language-python)                # haskell library for parsing and manipulating Python code # broken in 2505
model-graph.nix.m4
52:# PN(1709, haskellPackages . penrose)                     # still broken
76:# haskellpkgs                 . rdf4h                  # 2405 broken
dev-all.nix.m4
60:# SUBPKG(haskell, protobuf-native)          # via c++  # broken compilation
83:# SUBPKG(haskell, flatbuffers)          # serialization library for haskell  # broken
106:# RELEAS(all, nats-top)       # high performance NATS server   # suddenly broken in 2511
dev-asm.nix.m4
66:# haskellpkgs                   . tal          # typed assembly language # broken in 2305 → 2505
69:# RELEAS(dev-asm, retdec)                # Retargetable machine-code decompiler based on LLVM        # broken in 2511
70:# RELEAS(dev-asm, klee)                  # Symbolic virtual machine based on LLVM                    # broken in 2511
parsing.nix.m4
18:# SUBPKG(haskell, clexer)             # parses c++ code in simple token # currently marked as broken
31:# SUBPKG(haskell, antlr-haskell)  # broken in 2305
32:# SUBPKG(haskell, antlrc)         # broken
43:SUBPKG(python, lark)                         # broken in 2311 ?
66:# SUBPKG(haskell, peg)           # Parsing Expression Grammars in Haskell # broken in 2305
67:# SUBPKG(haskell, peggy)        # Parsing Expression Grammars in Haskell # broken in 2305
68:# SUBPKG(haskell, ponder)        # Parsing Expression Grammars in Haskell # broken in 2305
101:# haskellpkgs               . papillon         # PEG parser generator for Haskell # broken in 2305
102:# haskellpkgs               . pappy            # Packrat parsing; linear-time parsers for grammars in TDPL # broken in 2305
108:# haskellpkgs               . parco-parsec     # broken in 2305
111:# haskellpkgs               . parsec3          # Monadic parser combinators # broken in 2305
112:# haskellpkgs               . parsec2          # Monadic parser combinators # broken in 2305
118:# haskellpkgs                 . parsec-extra   # 2405 broken
125:# haskellpkgs               . xml-parsec       # XML parser combinator # broken in 2305
1 Like
9

Most likely the change is a libc or gcc update, which results in every C-based package being updated, which in turn makes every other language under the sun update.

Unlike traditional package managers, nix assumes that a dependency change can cause compile-time changes, and rebuilds reverse dependencies (in practice, nix even enforces that dependency changes cause compile-time changes, because of how it handles linking).

So whenever a low-level library has any change, every hash of all its reverse dependencies updates, even if only a few packages had version changes.

4 Likes
10

Here’s another tool for you research: nix-tree. Run it on a NixOS without arguments or just with the --derivation flag and enjoy!

3 Likes
11

Please use triple backticks for code blocks.

1 Like
12

Superb tool. I already tried it, but forgot about it. Put a link to it directly in other nix-dedicated scripts for troubleshoot.

13

And exactly this is what most poeple do not understand correctly.

The content in the store is not only mandated by what you have in your configuration, it is also mandated by an occasional nix shell or nix build (or the v2 counterparts) or a projects devShells output you work on regularly (or v2 counterpart).

Really, there is nothing wrong in having something twice, thrice or ten times even. This is just how a NixOS system will end after an hour of actually using nix.

If you are concerned about the diskspace used by all the various means of pulling stuff into the store, then learn about the various ways how GC-roots might get created, and how you can deal with them.

Also you might want to look into “store optimisation” which will add an hardlinking layer to the store that often reduces actual disk usage by 10 to 30%.

3 Likes
14

Yup, probably GCC or bash or something close to the root like that, as noted, store paths are (typically) input addressed, meaning that they are addressed not by their content but by the instructions that build them. In practice this means the following could (and does) happen:

  1. You use some package for an app which seemingly has nothing to do with C
  2. Glibc or GCC has a patch
  3. That package, and 95% of your other packages, actually do depend on C somehow. Maybe the package uses a python script in the build process.
    1. Well, the python interpreter is written in C.
    2. If GCC had a security patch for some edge case, it would be important to rebuild cpython with the patched GCC.
    3. Turns out the security patch didn’t apply when compiling cpython, the cpython binary is exactly the same as the old one!
    4. Doesn’t matter, the inputs (GCC) have changed, so the identical cpython binary goes to a new store path.
  4. Now that cpython has “changed”, your build script has different inputs. It doesn’t matter that the old cpython version and even binary contents are exactly the same as the old one. It doesn’t matter that the build script produced the same output. What matters is somewhere a dependency changed, so the package gets a new store path, even if the version and contents are the same. Hence giving a package seemingly unrelated to C a new store path.

Slightly relevant minute from nixcon: https://www.youtube.com/live/OXRfLgtKDz8?si=M-zaDIRCmNtnS_pJ&t=20850 (timestamp 5:47:29). A simple change to something like bash can have far and wide impacts across the whole store.

You’d have to actually inspect the store to know what specifically changed. nix-tree and nix-du are helpful. If you use flakes, flint can help too. nix-info -m can be a good sanity check tool to make sure nothing weird is happening.

3 Likes
15

It’s not so occasional in my case. For example, I have safe default python anchor a bit in the past, but for most daily and fast stuff I will use the python that pkgs offers me… Those occasional/regular use of the default python always result, when needed, in new entries in the store… I just looked at nix.settings.auto-optimise-store I am tempted… does that mean that only new derivation built will make the optimisation on dupplicated content… or it may impact the filesystem in a way that could interact badly with previous generation? I use only symbolic links… my understaning of hardlinks is super shallow. That makes me a bit hesitating. For GC-Roots, I have not looked into details between the profiles symlinks and how they act as root over nix-store entries… again stuffs to dig better for me.

16

I’m not quite understanding this, you are putting python in environment.systemPackages/home.packages? that should only result in store paths when you rebuild, not when you run python like how i am interpreting this.

Store optimization is fine, it works with previous generations. You can get optimization on existing derivations by running nix-store --optimise.

Regarding gcroots, if you use devShells often with direnv, you may want to look into GitHub - linyinfeng/angrr: Auto Nix GC Root Retention · GitHub, which is great for keeping them down. Even if you don’t use direnv, you can configure a profile retention policy for your system profile and enable nix.gc.automatic, which angrr automatically integrates with, to clean up old generations.

1 Like
17

Q1. No. I use channels extensively. And I have a railroad-redirection system that makes stuffs being indexes on those channels. In the code below, the channelling is done by predef.nMyDedicatedChannel.

octavepkgs          = predef.nOctave.octavePackages;
octaveversion       = predef.nOctave.octaveFull;

pythonpkgs          = predef.nPython.python313Packages;
pythonversion       = predef.nPython.python313Packages.python;
pythonchannel       = predef.nPython;

pythonnewestpkgs    = predef.nPythonNewest.python314Packages;
pythonnewestversion = predef.nPythonNewest.python314Packages.python;

rpkgs               = predef.nR.rPackages;
rversion            = predef.nR.R;

Currently I have moved my standard python to 3.13… so the dupplication is probably not happening for now but I am not always up-to-date.

For octave it is not mandatory at all, the project is not updated frequently, but it’s just because I have done it for most language where I have dependencies on my projects, and I sort of want stability, like a bit independently from the natural nixpkgs update routine.

If I understand my system correctly its a poor man flake-like system (but I use flake only occasionnally), for pinning stuffs. I may use flake in the future, but only when I feel totally at ease with nixos internals… at least more than I currently do.

What makes me not deciding to use flakes also, is to have locking symlinks spread all over the place… I need to understand the nixos system better so as to allow this happening on my discs

Q2. Optimisation wise… I think I really should put it on… But I just need to spend moment to find the culprit in my current “issue”/”normal disc space expansion”. I need to understand this depency on input for the store hash, explaining how gcc or bash can impact the rest. I get in conceptually, I understand the things. Now I have to see it in real code :wink: I think this issue is a good moment to climb this step.

Q3. My size problem happening regularily (at minimum every landmark version change), so currently I need to manage the profiles manually. But I will look at nix.gc.automatic…

18

I have some aliases on my system that help me check differences to previous rebuilds. (Here: https://codeberg.org/BlastboomStrice/dotfiles/src/.config/nixos-config/modules/nixos/aliases.nix)

If you want to check the differences across all the existing rebuilds on your system, try this

nix profile diff-closures --profile /nix/var/nix/profiles/system
1 Like
19

Is there an experimental feature to toggle so as this works? When I launch this same command, I just have nothing at all… blank… “nix profile diff-closures --profile /nix/var/nix/profiles/system” … not even a 10 second pause for parsing the the store one way or another… instantly back to the CLI

20

You need nix-command.

And this seems to be the root cause of your problem: Using lots of nixpkgs revisions at once will result in many glibc/GCC/other dependency revisions in your store. I often see people using both nixos-unstable and nixos-25.11, but that’s as many nixpkgs revs as I see.

For dependencies used in development, I recommend making a shell.nix for each project. using an inputs pinning tool with it (Nixtamal seems to be the current one people like, I use npins for stuff in my flakes that does not need to be fetched before evaluating anything), and using Direnv with nix-direnv, all instead of installing dev packages globally, which is discouraged in Nix.

Here’s an example (sorry if any of the instructions here are wrong, I dont know much nix2 cli):

{ pkgs ? import <nixpkgs> {} }:
pkgs.stdenv.mkDerivation {
  name = "example1";
  dontUnpack = true;
  installPhase = ''
    cat ${
      pkgs.stdenv.mkDerivation {
        name = "example2";
        dontUnpack = true;
        installPhase = ''
          echo "Hello, World!" > $out
        '';
      }
    } > $out
  '';
};

Put that Nix file somewhere, add the pkgs instant. Run nix-instantiate --eval $FILE_PATH, and then run nix --extra-experimental-features 'nix-command' derivation show $DRV_PATH, where $DRV_PATH is the output of the nix-instantiate command. Under inputDrvs, you should see an example2.drv, nix derivation show that if you want, then, inside the installPhase of example2, add # Comment above the echo command. Now instantiate again, and you will see that the drv path is different. You can then show the new drv path and new example2 path, and hopefully see that the store is input-addressed.

1 Like