Offline build "source closure"

Help
1

Hi there,

if i run the following command on a nix(os) sytem with an empty nix store:

nix-build '<nixpkgs>' -A pkgs.hello --option substituters ""

then it will basically build everything from scratch, like gentoo systems in the old days.

Of course, it will have to fetch a long list of source code folders in order to do that.
Is there a way to print the list of nix store paths that result from downloading these sources?

I hope to result in a scenario where i can store this list of sources in some file that could be nix-store --imported on an offline machine which can then build it all from scratch.

4 Likes
2

this may help you with your quest, are you expecting the internet to ‘vanish’ soon :wink: .

interesting concepts here https://collapseos.org/why.html

1 Like
3

@nixinator awesome, that pointer is what i needed!

i ended up copying and reducing the copy-tarballs.pl script to simply print the list of nix store paths of all downloaded sourcecode tarballs. this list is easily nix-store --exportable, and thus is what i needed.

Thank you very much!

I try to set up a “vendoring”-like demo for project partners who have some interesting requirements in their own projects.

1 Like
4

great!! if you can drive nix/OS adoption… it’s all good!

5

ok i ran into another problem…

when i build pkgs.hello on an offline system, it still tries to download at least busybox from nixos.org. i think that the busybox that is used by my own system and the offline one is not for sure the same one. is there a way to capture what is necessary to download and build those sources? it would be easy to add some more extra paths to the source closure, but i don’t know how to express them.

it seems like i can obtain the bootstrap tools from release.nix like this:

nix-build -E '(import "${(import ./nix/nixpkgs.nix).path}/pkgs/top-level/release.nix" {}).stdenvBootstrapTools.x86_64-linux.dist'

but (regardless if its x86-64 or i686) that does not match the same hash.

EDIT:

ok, if i add nix-store -qR --include-outputs $(nix-instantiate ./nix/nixpkgs.nix -A pkgs.bash) then it’s enough to build anything else from scratch.

2 Likes
6

ok so this works great for individual packages.

if i have some pkgs.hello as part of a whole nixos system, then the <nixpkgs/maintainers/scripts/find-tarballs.nix> expression does not contain the hello.tar.gz source code. i also tried it on the .config.system.build.toplevel attribute, but to no avail.

what is the right expression to run <nixpkgs/maintainers/scripts/find-tarballs.nix> on if i want to obtain all source code tarballs of also all directly or indirectly installed packages of a nixos config?

7

you may have more luck with this…

for idea!

8

Another update:

i was now finally able to build some packages completely from source including all bootstrapping on an offline machine. In order to generate all the source code paths and bootstrap tools content that is needed, the following commands proved sufficient:

# the nixpkgs etc. that are needed at evaluation time, all stored in the NIV json doc
evaluationPaths=$(nix-instantiate --eval --strict -E "builtins.map builtins.toString (builtins.attrValues (import ./nix/sources.nix {}))" | jq -r .[])

# all source code tarballs that result from the set of derivations that is emitted by default.nix
sourceClosurePaths=$(nix show-derivation -r $(nix-instantiate default.nix) | jq -r 'to_entries[] | select(.value.outputs.out.hash != null) | .key' | xargs nix-store -r)

nix-store --export $evaluationPaths $sourceClosurePaths > all-source-tarballs.closure

this a little bit more elaborate jq line gets all derivations that are somehow part of the final derivations, and filters out all derivations that have fixed output, like tarballs usually have.

8 Likes
9

well that’s brilliant.

would you consider making this into wiki page for wiki.nixos.org

1 Like