Nickel: the Nix (language) spin-off

[yannham]

@nrj

but I’m afraid that doesn’t clarify much for me. The goal of CUE is configuration, so I don’t see how it’s less adapted to doing so.

Fair enough. What CUE seems to offer compared to a static data language like JSON:

• Modularity. Thanks to the merge operation, you can write separate, modular blocks that are then combined into a configuration.
• Validation. CUE has a particularly well-behaved type system that allows to conveniently specify, combine and verify data schemas. In that category, CUE’s approach really stands out.

However, CUE is very restricted computationally. It is on purpose: first, to follow the philosophy that configuration is not general purpose programming and that it ought to stay close to just filling out options with values. The second reason is that it allows the type and merge system of CUE to be particularly well-behaved in return: types and values live in the same world, they can be freely combined in an associative and commutative way. CUE can generate inhabitants (i.e. examples of well-typed values) of a type, can eliminate redundant conditions, and more.

The consequence of this restricted computational power is that it doesn’t seem adapted to generating data from other data, and at reducing boiler-plate in configurations in general. Concretely, you can’t define nor use functions in the general sense (though you can encode simple ones to some extent). Something as simple as mapping over a list to produce a new value is not possible, for example. This is specific to CUE: Dhall (although not Turing-complete), Jsonnet, Nix and Skylark all allow some form of computations to take place. CUE has something called the scripting layer, that can do computations and perform side-effects (like creating files, querying the network, or whatever), but it is a different language and run in a strictly separated phase.

The approach of Nickel is to try to offer sane and simple defaults so that you have to only leverage complexity when you need it. Writing a simple configuration is like writing JSON. Doing a bit of validation using pre-defined contracts (think data validators) can almost feel like CUE (although not as well-behaved). Most recursion should be achieved with standard combinators like map, fold, etc. without resorting to recursive functions explicitly, like you can do in Dhall for example. But when you need it, you have the full power of a programming language at your disposal. For Nix, build systems in general, and probably other applications, you may have to perform parsing, to generate build graphs dynamically, etc.: in these cases, you may reach the limits of something like CUE, leading to dirty hacks at best, or to changing the whole technology stack at worst.

In conclusion, Nickel and CUE make different trade-offs with respect to the computational expressiveness: CUE is validation-oriented and very restricted computationally, while Nickel (and Nix, and others) offers more power to generate and transform data, in exchange of more complexity.

[gregwebs]

Cue has list comprehensions, perhaps it didn’t when this comment was written, but mapping over values is straightforward.

With the example of build systems and dynamic graphs mentioned here, that is exactly what dagger.io does with Cue.

Certainly though, defining functions is first class in Nickel and not in Cue where functions are either written in Golang or out of band in the scripting layer or some other interpreter of Cue output.

[asymmetric]

@yannham are you looking for outside help? If so, what’s the best way a new contributor could get their feet wet? Are you looking for more contributions in the style of full translation of nixpkgs/lib/lists.nix by francois-caddet · Pull Request #722 · tweag/nickel · GitHub?

[yannham]

@gregwebs Thanks for the clarification. I learned list comprehensions exist since, it was maybe already a thing but at least I didn’t know them at the time of writing. Sorry for the misinformation. I imagine it’s still not always possible to translate a pipeline that, say, maps a few functions, then filterMap, then fold, to CUE, for example. But then maybe you don’t need it that often.

[yannham]

@asymmetric Sure, contributions are welcome we try to maintain a list of issues (including the tag “good first issue”) to track what we work on, and what needs to be worked on. If you see something that you feel up to, feel free to manifest yourself there. I think there aren’t currently good first issues that are open, unfortunately.

In particular the LSP could get some love (one example: record field completion is a good argument for Nickel’s type and contract system, it would be very nice and isn’t too hard to pull up - although not trivial either -. But right now nobody has the bandwidth to do so in the team.)

Beside programming, you can contribute to the design by chiming in bikeshedding issues or RFC pull request (like [RFC003] Nix-nickel (Draft, WIP) by yannham · Pull Request #693 · tweag/nickel · GitHub, which is work in progress interleaved with concrete experimentation). Opinions, feedbacks and experiences are welcome. There is also documentation: as a newcomer, if you found some documentation badly written or obscure, or you missed a specific kind (tutorial, how-to, specification, etc.), please report.

At last, one useful thing is to, well, write Nickel! As your example, translating and typing Nix libraries is useful to make the code available to Nickel, to stress test the language and the type system on actual code. It can also be incorporated into benchmarks, etc. You can propose additions to the standard library, which is pretty limited right now, or write helpers as stand-alone libraries. In particular I think a huge part of the practical value proposition will lie in contracts, which encode knowledge as a executable validators, but those need to be written (be it for Terraform resources, Kubernetes pods, Nix flakes, etc.).

Indeed, people knowing about Nickel are mostly from the Nix community, but to be usable for Nix requires much more than just a working language. We have to think about Nixpkgs interop, about evaluating derivations, about string contexts, about having decent performances, at least in par with Nix, etc… We are progressing toward this milestone, but in the meantime, user feedback is scarce and would be very valuable.

[blaggacao]

Instead of writing validators from scratch a interop shim for json schemata is likely to produce more leverage at lower maintenance.

I know we’ve been discussing this in the past, but I don’t remember the reasons against favoring a shim over re-implementations.

Or has this just been falling through the priority/attention sieve?

[yannham]

A JSON Schema to Nickel tool shouldn’t be very hard to implement indeed, and would be of great value. Somebody at Tweag started a side work on a nickel-kubernetes project some time ago, including a simple PoC (but not far from working) of an openAPI2Nickel contract generator, but the author has left since. At the time, he was also blocked by two issues, the first has been closed long ago and the second one has a recent open PR that should close it (Optional fields).

To answer your question, it’s probably just the latter option: falling through priority sieve. I don’t see a good reason against shims, and I don’t see obvious hardblockers on the way either.

[ysndr]

@asymmetric If you are interested in continuing my work on the LSP, feel free to contact me.
I wrote my master thesis on it and gladly help onboarding.

[peterhoeg]

@yannham

Do you have specific optimizations in mind?

Performance isn’t mentioned in the rationale document, but what are your expectations in terms of performance relative to nix? Eval’ing nixpkgs is really quite slow (although I have nothing remotely intelligent to say about where the slowness exactly comes from) and it needs to happen again and again, so are your plans to address that in any way?

[yannham]

@peterhoeg

Performance isn’t mentioned in the rationale document, but what are your expectations in terms of performance relative to nix? Eval’ing nixpkgs is really quite slow (although I have nothing remotely intelligent to say about where the slowness exactly comes from) and it needs to happen again and again, so are your plans to address that in any way?

Until now, the implementation was naive on many aspects, making any form of comparison with Nix probably meaningless (Nix has performance problems, but it has been here for almost 20 years, so it still had had a bit more engineering time put in it). But as of today, the main obvious bottlenecks have been taken care of, and I think a first goal “be as performant as Nix, or at least not dramatically worse”, can be approachable. Contracts can be challenging with this respect, though, as they incur more run-time checks than in bare Nix expressions.

In general, we’ve discussed the performance of Nix several times with Eelco and others, and while there’s always room for improvement in the implementation of the Nix interpreter, there is also a number of external factors - e.g. the architecture of Nixpkgs - which can’t all be addressed only by the language side (although the language can be the enabler of a better and more efficient architecture, for example). One design constraint in Nickel is to avoid any feature or implementation choice that can hurt parallel evaluation (the interpreter is not currently evaluating in parallel, but the rationale is that this is a good performance boost, so the day we will implement it, there won’t be any obvious blocker).

In a (somehow distant) future, we want to explore the idea of self-adjusting computations combined with “cached call-by-name” (well, that’s our current name for it, at least). A full exposition is out of scope for this answer, but from a high-level perspective, the goals are:

1. Caching, both during evaluation and persistently between evaluations, with custom granularity and caching strategy. The evaluation model of Nickel should include a native notion of such a cache. Currently in Nix, laziness is in some way one very rigid and specific evaluation caching strategy (you cache each and every binding, attributes, and list elements, and only them, basically), and is tightly coupled with the evaluation model. I remember from pairing sessions with @thufschmitt that the flake evaluation cache, which obviously requires a different granularity (probably attribute-level, and only for the top-level flake?) needed some hacky patching of the interpreter to work. Our goal is to have a clean and decoupled notion of evaluation cache, so that we are able to evolve the caching strategy or even have different competing ones at the same time (live evaluation cache vs persistent cache, like flakes) without having to butcher the interpreter or depart from the original specification.
2. Incremental evaluation (the self adjusting computation bit). You change one option in your configuration, and the interpreter + cache is able to determine (or at least approximate) which parts need to be re-evaluated, and only re-evaluate those parts.

These ideas need are still to be fully developed, experimented, and tested on Nixpkgs specifically, but I do think that they open exciting perspectives