How is this any good? Import from derivation is good because derivations are reproducibly built during the evaluation/build process. This, on the other hand just allows using random binary blobs during Nix evaluation.
Not to mention that DetSys has established that Dix is a fork they are ready to append anything they or their customers need regardless of the upstream or community opinion on the matter.
So updating the YAML parser dependency could cause differences in evaluation results across Nix versions
This is no different than updating the library the binary blob is built with though, just that the output will differ across the dependency versions instead of Nix versions. You know what a real solution is? Using IFD with a pinned version of the library.
Instead of trying to circumvent the need for IFDs I suggest DetSys focus on improving the evaluator in a way that there’s no need to do so. Snix/Tvix have already shown that it’s possible
The proper solution to using wasm with nix is a lot more complex than what it sounds like they did here.
You need:
- A way to run some kind of cross-platform wasm-only derivation.
- A bootstrap chain to get self-hosted wasm compilers built entirely in such derivations.
- Nix optimized for reasonably-efficient IFD.
- A separate nix config control for allowing wasm-IFD even when regular IFD is disallowed.
- This
builtins.wasmfunction. (Assuming its execution is actually sufficiently sandboxed… wasn’t clear to me)
Congratulations for 1 step in the path, I guess, but this isn’t ready for wide use in the nix language. There’s a long way to go.
If you’re fine with IFD you can enable allow-unsafe-native-code-during-evaluation and use
let
cliProg = pkgs.ioutputnixexpressions;
in
builtins.exec [ (lib.getExe cliProg) "args" "here" ];
To achieve something “similar”. (You probably know this, adding context for other readers)
I think having a WASM VM in Nix is pretty cool, the code is available so when someone cares enough to “complete” it they have something to build upon at least.
builtins.exec is also completely not sandboxed. Just in case anyone didn’t catch that part. Hence the word “unsafe” in the option name to enable it.
The point here is to greatly expand what you can achieve inside of pure Nix expressions. It’s unclear to me how an improved evaluator would address the use cases mentioned in the post.
Is it only me, or is the fib example is a very poor choice? If one writes a multiple recursion in nix and rust and then benchmarks the multiply recursive nix function against an ahead-of-time compiled and optimised plain recursive function (Compiler Explorer) then obviously the results are going to be dramatic.
Just wanted to mention a non-cherry picked fib. Comparing that against builtins.wasm is left as an exercise to the reader.
let memo = builtins.genList (x: if x < 2 then 1 else ((builtins.elemAt memo (x - 2)) + (builtins.elemAt memo (x - 1)))) 100; in builtins.elemAt memo 50
That is not to say that having the entire LLVM ecosystem at your disposal for compiling existing libraries that do non-trivial parsing/string manipulation isn’t great. That is actually the whole selling point of this feature - reusing as much stuff as possible.
I definitely need to let it sink in what exactly this would mean if upstreamed. Checking in random binary blobs is definitely undesireable, that’s how you get trojans in your supply chain. IFD does solve this but you’re not gaining much IFD can’t already do.
I suppose there’s performance improvements and room for caching improvements over pure IFD if you reuse a WASM module many times? I’m not familiar enough with IFD to say for sure.
I think, more than anything, this shows a real need to move away from NixEL, and WASM is an excellent choice for a replacement. If this is the start of something bigger (a piecemeal replacement of the Nix frontend), I’m very excited to see where it could go.
This feature sounds useful, but…
Wasm modules are often small enough that you can commit them into your Git repositories directly. For example, the compiled Wasm module for parsing and generating YAML is 180 KiB—probably still an acceptable size for adding to a repository like Nixpkgs.
Alternatively, you can fetch the Wasm module at evaluation time like this:
builtins.wasm { path = builtins.fetchurl https://.../nix_wasm_plugin_fib.wasm; function = "fib"; } 33If you’re using flakes, you can use the
fileflake input type to fetch a single Wasm module via HTTP. This allows you to update the Wasm dependency automatically usingnix flake update.Finally, you could use import-from-derivation to declaratively build the Wasm module from source. But then you’re back to using import-from-derivation, which somewhat defeats the purpose!
I really can’t see myself checking prebuilt binaries into version control, or adding code that will fetch a prebuilt file. I guess that for my purposes I’ll just stick with using Import From Derivation.
Yeah, soft-launching a backdoor is crazy talk. That should never be acceptable in nixpkgs.
Fully agree this shouldn’t be allowed in nixpkgs if upstreamed. However, I think there’s usecases in private build chains where there’s more trust. Or maybe some use even with IFD that IFD alone doesn’t handle gracefully (perhaps as a an extended lib?). I think there are opportunities if used responsibly.
If you’re using a preprocessor to generate wasm blobs you can also use a proprocessor to spit out json or some other format that’s easy for nix to consume, that’s isn’t a binary blob.
Sounds like we could use some sort of WASM hex0 for Nix, and use it to produce other WASM blobs. Would be cool to see.
Probably also need caching across evals so we don’t have to rebuild every wasm blob from scratch.
it is basically away of introducing side-effects into pure Nix evaluation.
We’re also working on WASI support, which would enable side effects like writing to the file system. What you’re seeing in this blog post, however, doesn’t introduce side effects. You pass in a Nix language value, you get a computed Nix language value out.
Sorry if this is a naive question, but what’s the benefit of WASI-enabled side-effects vs. just turning the sandbox off?
It would enable you to create actual derivation logic in Rust, C++, etc.
Seems a bit unfair, with wasm you generate the blobs once, which could be from IFD so you still have the source, then reuse it many times for evaluation. While doing the preprocessing using IFD means you get it everywhere you need input, plus you still depend on binary blobs after all, which output files instead of nix values.
IFD and preprocessing are different. And JSON files are not binary blobs. You can even use preprocessing to generate nix code, if you like, which is again not binary blobs.
We can already do that btw, that’s what 2nix is.
IMO the only reason we even have this discussion is that nix’s evaluator is so slow and single-threaded that IFD is actually a problem. And so we need all these other hacks for a semblance of performance.
What happened to dynamic drvs, anyway?
Can you expand on what the vision is here? WASI doesn’t seem relevant for wasm consumed at eval time by Nix expressions. Those should need to be semantically pure, and WASI is largely about IO and side effects, as you mentioned, which doesn’t seem compatible. If it’s just about access to the derivation and toFile style primitives, then I don’t understand where WASI comes into play.
If it’s about derivation { system = "wasm32-wasi"; } style derivations, then I don’t see how that enables you to do derivation logic in wasm, short of things like IFD or dynamic-derivations which already work without wasm.