Contract, Setup Nix to build packages in a different store and support moving packages between machines

Hello,

I have a small project that I need help with.

I want nix to build software on a non-standard store directory, not on /nix.
Also, the software that is build should be relocatable in different machines, of course all with the same base storage path.

I tried myself and I failed over and over.

Fundamental packages like go don’t build (don’t pass tests).
Packages that are built, are not relocatable.

Can somebody help?

2 Likes

I want nix to build software on a non-standard store directory, not on /nix.
Also, the software that is build should be relocatable in different machines, of course all with the same base storage path.

Fundamental packages like go don’t build (don’t pass tests).
Packages that are built, are not relocatable.

I think it could be a good idea to post which subset of Nixpkgs you need most; I would expect people to be more interested if the packages in question are the ones they understand and use.

(I am a bit surprised that building Nix from source with a different prefix and bootstrapping from scratch did not work out, to be honest, but when I did such things I did not need any Go)

1 Like

I am building a software stack.
So I will need compilers, fundamental libraries, and pretty much whatever is needed to run a production and a dev environment.
To be honest, pretty much everything.

But all the tools should already be in nixpkg, they just don’t always build right, or if they build are not relocatable in a new machine…

1 Like

There are indeed places where ${builtins.storeDir} is not used but /nix/store is hardcoded. Those would need to be fixed.

Hummm, I believe we are talking about much more basic issues.
The one you mentioned was never a problem.

1 Like

A bit of a hack, but one I’ve had success with in the past: GitHub - timjrd/nixrewrite: Prototype implementation of Nix store path rewriting

Hi,

Depending if you compile with sandboxed build or not, you may have issues.
libuv, for example, compiles within sandboxed build with a custom store but fails when the sandbox is disabled.

For Go, my overlay for a host with a custom nix store looks like this:

final: prev: {
  # ...
  go_bootstrap = tryUpstream prev.go_bootstrap (attrs: {
    prePatch = attrs.prePatch + ''
      sed -i '/TestChown/aif true \{ return\; \}' src/os/os_unix_test.go
    '';
  });
  go_1_15 = tryUpstream prev.go_1_15 (attrs: {
    prePatch = attrs.prePatch + ''
      sed -i '/TestChown/aif true \{ return\; \}' src/os/os_unix_test.go
      sed -i '/TestFileChown/aif true \{ return\; \}' src/os/os_unix_test.go
      sed -i '/TestLchown/aif true \{ return\; \}' src/os/os_unix_test.go
    '';
  });
}

On my desktop, the filesystem is ZFS. On the hosts the store is mostly in NFS (it may sometimes be in a Lustre FS (a parallel filesystem posix compliant)).

1 Like

does nfs /nix/store actuall work very well? does nfs actually work well for anything?
i’d be interested to hear what results you are getting, and how you have it setup.

I used to successfully run concurrent nix-builds on the same nfs store with this patch: Nix Installation Guide - NixOS Wiki

Does this actually work how i’m thinking?

How does stuff like GCROOTS work, as your going to have multiple machines/users writing these roots to prevent garbage collection?

The patch just lets sqllite use nfs dotfile locking semantics…

Or is the fact the each machine has it own NFS mount, with it’s own copy of the /nix ?

doesn’t /nix/store have to be available and mounted before the system will boot? Doesn’t Initrd come from the /nix/store?

A sample configuration would be really interesting if you have time to upload it here , i’d like to see how it works and what you can do with it.

How are you trying to use a non-standard store directory?

The last time I tried was ~3 years ago, building Nix from source on Debian. I don’t remember the exact commands I used though. The main issue I had was that rebuilding everything from source is slow, and Nix rebuilds things a lot more often than other package managers that rely on semver. I didn’t try copying anything to another machine though.

I used that as a remote builder only. Indeed sharing gc roots may be problematic, and I would be surprised if you could run NixOS with a NFS store.

Thanks everybody for your comments!

Eventually I manage to solve all the problems myself, and at the moment I don’t think I need much more help.

The goal of the project was to create a software stack for packages.redbeardlab.com a public CernVM-FS, or CVMFS for short, repository.

CVMFS is a software distribution mechanism developed at CERN to distribute scientific software in the datacenters.

2 Likes