tl;dr I have closed-source but natively build nix packages I’d like to share, but I’m not sure how.
In more detail: I have a Rust crate in a private repository that I’m building with crane. The repository is private, but I have a functional flake output that can build the package successfully. However, I would like to share the package - not the source - as a third-party package with others. Essentially, I’d like to write a module I can hand to users but need something to put into { services.foo.package = privatePackage; }.
Potential solutions:
I could try and statically compile my crate, which would alleviate the need to express any dependencies for the compiled binary and I could just fetchUrl a publicly-available URL.
This would work, but I’m hitting a lot of annoying speedbumps with musl+openssl-sys, so I’m exploring other options (and would probably make a fat binary, too)
I could build the package outside of the context of nix entirely and then try patching the ELF’s dynamically linked libraries (with patchelf or otherwise).
This would probably also work but 1) I’m a little nervous about rewriting library references that may introduce incompatibilities and 2) it feels like a hopefully(?) unnecessary workaround given that I have fully-functional derivation for my package already.
My ideal hope is that I can somehow use my packages that I already have building with nix but without providing direct access to the source, so providing a package derivation complete with an accessible { src = ... } is the problem.
So far the closest I’ve come is potentially abusing import from derivation (IFD) for this purpose. I’m totally happy to upload the built/realized derivations to a shared cache, and I can happily nix build /nix/store/path-to-built-derivation and my clients that already have access to the cache will fetch the dependencies, but I don’t have anything to write into a configuration.nix or flake.nix that I can express declaratively rather than imperatively with a nix build. I thought I could write out the package’s .drv store path into an import /nix/store/<path>.drv expression but that doesn’t seem to work either.
Any ideas? Are there other possibilities that would let me provide built nix packages directly in a form that’s able to be referenced in a NixOS configuration (not just from a shared cache with nix build or nix copy)?
Does your package need to reference its exact output path, or can it be unpacked into no matter what path and still work? If the latter, I would tarball the output, and write a script to generate an expression that formally depends on all the dependencies and simply unpacks the tarball. Possibly the tarball needs to be handled via requireFile.
Interesting - so the idea would be to just tar up the package’s output, fetch it statically for the parent package’s src, and then inside of the parent package, specify all of the same buildInputs? That’s a solution I hadn’t considered. I guess the main thing would be to ensure that the dependencies are pinned to ensure that the tarballed executable can always reach its paths and avoid doing things like let the parent package’s instance of pkgs be changed and potentially shift the dependencies underneath the static tarball. I could potentially do something like use the exact same derivation in the “parent” or source-unavailable package and strategically override the src and instruct it not to perform any build steps.
Yes, you are right, that’s probably way better than generation.
Yes another option is like that: you give them an expression with impossible-to-realise src, and a binary cache that only has the runtime dependencies of the package. So evaluation-level all is honest, output paths are literally the same, and the download goes via substitution.
They have to trust your cache and you would probably have glibc in it, but for whatever they want from mainline they probably have the mainline cache with higher priority, and they literally run opaque binaries from you so they have to trust you don’t ship keyloggers anyway.
Wow, thank you so much @Infinisil - that may be exactly what I’m looking for Based upon the function’s documentation, it fits my use case almost exactly.
If you can spare some time for additional questions:
In the interest of supporting users as best as possible, my initial thought is to use storePath (which I found in the docs for fetchClosure) if the experimental feature flag is not enabled - is there a canonical way of determining that, or is it just builtin ? fetchClosure? It does seem like if I can use fetchClosure, I probably should, since it alleviates the need to trust and configure a remote store.
Similarly, I assume that I can intelligently fall back on storePath by including another assertion to check whether I’m in pure evaluation mode (since the docs indicate that the function isn’t available outside inside of it - I assume there’s some way to determine that) if fetchClosure isn’t available with the user caveat that they’ll need to 1) configure the remote store and trusted signing key and 2) are in outside pure evaluation mode.
Edit: I mis-read the non-pure requirement for storePath
Based upon the function’s documentation, it fits my use case almost exactly.