tl;dr seenix.dev lays every byte of a Nix closure out on a map, one pixel per byte, and lets you zoom from a whole NixOS system down to the hex of libc.so.6. Try hello, firefox or a GNOME desktop. Nothing runs on a server.
Disclaimer: AI used for the software not the writing
16 Likes
Cool visualization!
Looking at the GNOME desktop example…
Surely mesa-26.2.2 can’t depend on ALL of llvm-21.1.8-lib? Does it need lib/libLLVMPowerPCCodeGen.a? Could there be any ways we could split LLVM up?
Also, it’s hard to inspect specific subpaths of the store path (for example, I had to type that path out by hand as there was no way to copy).
This is absolutely brilliant!!! What a fun project and write up!!! Love it.
1 Like
Found some relevant issues:
opened 05:32PM - 14 Mar 22 UTC
0.kind: bug
6.topic: closure size
2.status: stale
### Describe the bug
Looking at space used by nix packages, it seems there's a … lot of packages that used Nx space due to having static libraries in out/lib outputs. The first such package I found is llvm.lib used by mesa: it consumes ~300 MB while most of the space is used by static libraries that are helpful only while building something, shared libraries in the package are only around ~100 MB, so it's grown around 3x by that issue. Then, when I started looking at all static libraries in /nix/store with `find`, rav1e stands out from all other packages. Entire consumed space by rav1e is 89M, librav1e.a is 81M while librav1e.so.0.5.1 is only 3,2M. The package is around 10X bigger than if static libraries were separated and 25X bigger than if there was a lib output with only shared library.
### Steps To Reproduce
Steps to reproduce the behavior:
1. `nix-store -qR /run/current-system | xargs -I [] find [] -name '*.a' -exec du -h {} \; | sort -h`
### Expected behavior
Static libraries are in dev output
### Additional context
My /nix/store (and `nix-store -qR /run/current-system | xargs du -csh | sort -h`) is 12GB after `nix-collect-garbage -d` and I'm trying to understand why it's so big while a similar configuration with other distros would consume a way less space...
### Notify maintainers
@lovek323 @7c6f434c @dtzWill @primeos for LLVM, it seems rav1e has no maintainers and I don't know who to mention for a generic solution across all the packages...
### Metadata
```console
- system: `"x86_64-linux"`
- host os: `Linux 5.16.11-zen1, NixOS, 22.05 (Quokka)`
- multi-user?: `yes`
- sandbox: `no`
- version: `nix-env (Nix) 2.7.0pre20220221_caf5172`
- channels(root): `"nixos-22.05pre338578.a2e281f5770"`
- nixpkgs: `/etc/nixpkgs`
```
opened 05:50AM - 08 May 23 UTC
0.kind: enhancement
0.kind: question
6.topic: closure size
2.status: stale
6.topic: llvm/clang
### Describe the bug
Recently I'm working on a project depends on llvmPackage… s, and want to package it into a standalone container (to release). However, looks like `llvmPackages_*.llvm.libs` is coupled with dynamic **and** static libraries, which significantly increases the closure size :( . Is there anyway to declare that the derivation only depends on llvm dynamic libraries only? e.g. `libLLVM-15.so`.
### Steps To Reproduce
Steps to reproduce the behavior:
```console
[user@system:~]$ ls `nix eval --raw nixpkgs#llvmPackages_15.llvm.lib`/lib
LLVMPolly.so libLLVMAsmParser.a libLLVMExegesisMips.a libLLVMMCA.a libLLVMPowerPCInfo.a libLLVMVEDisassembler.a
LLVMgold.so libLLVMAsmPrinter.a libLLVMExegesisPowerPC.a libLLVMMCDisassembler.a libLLVMProfileData.a libLLVMVEInfo.a
libLLVM-15.0.7.so libLLVMBPFAsmParser.a libLLVMExegesisX86.a libLLVMMCJIT.a libLLVMRISCVAsmParser.a libLLVMVectorize.a
libLLVM-15.so libLLVMBPFCodeGen.a libLLVMExtensions.a libLLVMMCParser.a libLLVMRISCVCodeGen.a libLLVMWebAssemblyAsmParser.a
libLLVM.so libLLVMBPFDesc.a libLLVMFileCheck.a libLLVMMIRParser.a libLLVMRISCVDesc.a libLLVMWebAssemblyCodeGen.a
libLLVMAArch64AsmParser.a libLLVMBPFDisassembler.a libLLVMFrontendOpenACC.a libLLVMMSP430AsmParser.a libLLVMRISCVDisassembler.a libLLVMWebAssemblyDesc.a
libLLVMAArch64CodeGen.a libLLVMBPFInfo.a libLLVMFrontendOpenMP.a libLLVMMSP430CodeGen.a libLLVMRISCVInfo.a libLLVMWebAssemblyDisassembler.a
libLLVMAArch64Desc.a libLLVMBinaryFormat.a libLLVMFuzzMutate.a libLLVMMSP430Desc.a libLLVMRemarks.a libLLVMWebAssemblyInfo.a
libLLVMAArch64Disassembler.a libLLVMBitReader.a libLLVMFuzzerCLI.a libLLVMMSP430Disassembler.a libLLVMRuntimeDyld.a libLLVMWebAssemblyUtils.a
libLLVMAArch64Info.a libLLVMBitWriter.a libLLVMGlobalISel.a libLLVMMSP430Info.a libLLVMScalarOpts.a libLLVMWindowsDriver.a
libLLVMAArch64Utils.a libLLVMBitstreamReader.a libLLVMHexagonAsmParser.a libLLVMMipsAsmParser.a libLLVMSelectionDAG.a libLLVMWindowsManifest.a
libLLVMAMDGPUAsmParser.a libLLVMCFGuard.a libLLVMHexagonCodeGen.a libLLVMMipsCodeGen.a libLLVMSparcAsmParser.a libLLVMX86AsmParser.a
libLLVMAMDGPUCodeGen.a libLLVMCFIVerify.a libLLVMHexagonDesc.a libLLVMMipsDesc.a libLLVMSparcCodeGen.a libLLVMX86CodeGen.a
libLLVMAMDGPUDesc.a libLLVMCodeGen.a libLLVMHexagonDisassembler.a libLLVMMipsDisassembler.a libLLVMSparcDesc.a libLLVMX86Desc.a
libLLVMAMDGPUDisassembler.a libLLVMCore.a libLLVMHexagonInfo.a libLLVMMipsInfo.a libLLVMSparcDisassembler.a libLLVMX86Disassembler.a
libLLVMAMDGPUInfo.a libLLVMCoroutines.a libLLVMIRReader.a libLLVMNVPTXCodeGen.a libLLVMSparcInfo.a libLLVMX86Info.a
libLLVMAMDGPUTargetMCA.a libLLVMCoverage.a libLLVMInstCombine.a libLLVMNVPTXDesc.a libLLVMSupport.a libLLVMX86TargetMCA.a
libLLVMAMDGPUUtils.a libLLVMDWARFLinker.a libLLVMInstrumentation.a libLLVMNVPTXInfo.a libLLVMSymbolize.a libLLVMXCoreCodeGen.a
libLLVMARMAsmParser.a libLLVMDWP.a libLLVMInterfaceStub.a libLLVMObjCARCOpts.a libLLVMSystemZAsmParser.a libLLVMXCoreDesc.a
libLLVMARMCodeGen.a libLLVMDebugInfoCodeView.a libLLVMInterpreter.a libLLVMObjCopy.a libLLVMSystemZCodeGen.a libLLVMXCoreDisassembler.a
libLLVMARMDesc.a libLLVMDebugInfoDWARF.a libLLVMJITLink.a libLLVMObject.a libLLVMSystemZDesc.a libLLVMXCoreInfo.a
libLLVMARMDisassembler.a libLLVMDebugInfoGSYM.a libLLVMLTO.a libLLVMObjectYAML.a libLLVMSystemZDisassembler.a libLLVMXRay.a
libLLVMARMInfo.a libLLVMDebugInfoMSF.a libLLVMLanaiAsmParser.a libLLVMOption.a libLLVMSystemZInfo.a libLLVMipo.a
libLLVMARMUtils.a libLLVMDebugInfoPDB.a libLLVMLanaiCodeGen.a libLLVMOrcJIT.a libLLVMTableGen.a libLTO.so
libLLVMAVRAsmParser.a libLLVMDebuginfod.a libLLVMLanaiDesc.a libLLVMOrcShared.a libLLVMTableGenGlobalISel.a libLTO.so.15
libLLVMAVRCodeGen.a libLLVMDemangle.a libLLVMLanaiDisassembler.a libLLVMOrcTargetProcess.a libLLVMTarget.a libPolly.a
libLLVMAVRDesc.a libLLVMDiff.a libLLVMLanaiInfo.a libLLVMPasses.a libLLVMTextAPI.a libPollyISL.a
libLLVMAVRDisassembler.a libLLVMDlltoolDriver.a libLLVMLibDriver.a libLLVMPowerPCAsmParser.a libLLVMTransformUtils.a libRemarks.so
libLLVMAVRInfo.a libLLVMExecutionEngine.a libLLVMLineEditor.a libLLVMPowerPCCodeGen.a libLLVMVEAsmParser.a libRemarks.so.15
libLLVMAggressiveInstCombine.a libLLVMExegesis.a libLLVMLinker.a libLLVMPowerPCDesc.a libLLVMVECodeGen.a
libLLVMAnalysis.a libLLVMExegesisAArch64.a libLLVMMC.a libLLVMPowerPCDisassembler.a libLLVMVEDesc.a
```
### Expected behavior
Anyway to include dynamic libraries only?
### Notify maintainers
@fpletz @Ericson2314 @sternenseemann @rrbutani
### Metadata
```console
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-linux"`
- host os: `Linux 6.2.13, NixOS, 23.05 (Stoat), 23.05.20230502.1a411f2`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.15.0`
- channels(root): `"nixos"`
- channels(lyc): `"nixpkgs"`
- nixpkgs: `/home/lyc/.nix-defexpr/channels/nixpkgs`
```
Seems tricky due to what was mentioned here:
staging ← revert-162607-libllvm-drop-static
opened 10:42AM - 17 Jun 22 UTC
Reverts #162607 / 1748887ff2353a16806a46dc27fb603981a047d5.
Reason for revert… : This change caused llvm-config{,-native} to be unable
to find static archives bundled with LLVM, as has been [reported]. Ever
since #152944 using moveToOutput in LLVM is _evil_ because llvm-config
obtains it knowledge about the installation locations from the CMake
configure step.
Consequently a change like #162607 will need to be implemented by making
LLVM itself install the static archives to the correct location or by
adding yet another patch which updates llvm-config's knowledge of the
location. The latter is not desireable in my opinion, though, since it
is just asking for this sort of trouble: Before #152944 we had an
outputs.patch that did this sort of things which broke spectacularly in
edge cases.
Fixes #148117.
[reported]: https://github.com/NixOS/nixpkgs/issues/148117#issuecomment-1158245576
---
Tested with
```console
$ ./result-dev/bin/llvm-config --link-static --libs
$ ./result-dev/bin/llvm-config --link-static --libnames
```
for the default LLVM version.
Note that this PR *will* increase the closure size of LLVM, but reduced closure size is not worth sacrifizing correctness for. There is no trivial way to fix @waldheinz's original patch, so reverting until a better solution is found is the better option.
###### Things done
- Built on platform(s)
- [x] x86_64-linux
- [ ] aarch64-linux
- [ ] x86_64-darwin
- [ ] aarch64-darwin
- [ ] For non-Linux: Is `sandbox = true` set in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html))
- [ ] Tested, as applicable:
- [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests))
- and/or [package tests](https://nixos.org/manual/nixpkgs/unstable/#sec-package-tests)
- or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test)
- made sure NixOS tests are [linked](https://nixos.org/manual/nixpkgs/unstable/#ssec-nixos-tests-linking) to the relevant packages
- [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage)
- [ ] Tested basic functionality of all binary files (usually in `./result/bin/`)
- [22.11 Release Notes (or backporting 22.05 Release notes)](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#generating-2211-release-notes)
- [ ] (Package updates) Added a release notes entry if the change is major or breaking
- [ ] (Module updates) Added a release notes entry if the change is significant
- [ ] (Module addition) Added a release notes entry if adding a new NixOS module
- [ ] (Release notes changes) Ran `nixos/doc/manual/md-to-db.sh` to update generated release notes
- [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).
1 Like