Protocol for Nix to query lazily-evaluated expressions from external daemons?

Development
1

I’m exploring ideas for improving Nix’s interoperability with other languages and wanted to check if something like this already exists (or has been proposed).

The Problem

Nix natively supports JSON parsing (builtins.fromJSON), but not YAML. The common workaround is:

  1. Define a derivation that converts YAML to JSON (e.g., using yq)

  2. Use Import-From-Derivation (IFD)

  3. Call builtins.readJSON on the result

This works, but has some drawbacks:

  • The converted JSON ends up in the Nix store unnecessarily

  • For large YAML files, the entire file must be processed even if you only need a small part of it

  • IFD blocks evaluation and doesn’t parallelize well

The Idea

What if Nix could connect to a sandboxed external daemon and query it for lazily-evaluated Nix expression values?

The daemon would be provided by an external process (written in any language), and Nix would send requests like:

  • builtins.typeOf on a value

  • Attribute access like a.b.c

  • Other operations that force evaluation

The daemon would then return the result as a Nix value, evaluating only the requested parts lazily.

For the YAML example: instead of converting the entire file to JSON via IFD, Nix could query a YAML-parsing daemon for just config.services.nginx.enable, and the daemon would parse only what’s needed and return the Nix value directly.

This would enable:

  • Parsing arbitrary formats (YAML, TOML, XML, etc.) lazily without IFD or polluting the Nix store

  • Better interop between Nix and other languages sharing the same Nix store

  • External processes providing “virtual” Nix values that are computed on-demand

Questions

  1. Does something like this already exist in Nix?

  2. Has this been proposed before in an RFC or GitHub issue?

  3. Would this kind of extension mechanism be feasible within Nix’s architecture?

Any pointers would be appreciated!

3 Likes
2

What if Nix could parse YAML?

2 Likes
3

I hope we have a more general solution to this kind of problems

1 Like
4

It is possible to use the Nix library to build a frontend that consumes values in non-Nix formats, converts them to Nix values, and instantiates a derivation which can be processed by the standard tooling. There exists tools that already do this.

5

This idea still suffers from one of the most surprisingly annoying aspects of IFD: system dependence of the eval itself.

You either can’t do the eval except on a particular system, or you need to system-space non-derivations. Either way it’s not pretty.

6

Solution using existing tools:

  1. run nix with –allow-unsafe-native-code-during-evaluation
  2. use builtins.exec to run yq converting the YAML to JSON
  3. use builtins.fromJSON on the resulting string
7

Something I’ve seen discussed before (from people at Shopify IIRC) would be for nix to provide a WASM evaluator. Then you could compile your YAML parser into WASM and use that. That would avoid the safety problem as well as the system dependency problem. It’s still a very significant change though.

8

FWIW, @edolstra and @Mic92 have been playing around with adding WASM support for eval (as in builtins.wasm) and WASM derivations.

WASM support by edolstra · Pull Request #309 · DeterminateSystems/nix-src · GitHub

Such an approach should in principle allow for implementing more performant parsers that do not have to be a part of nix itself.

5 Likes