2023-09-18 Nix team meeting minutes #87

Development Nix
1

Attendees: @roberth @infinisil @edolstra @thufschmitt @tomberek @fricklerhandwerk
Notes: @infinisil @fricklerhandwerk

Agenda

  • NIX_PATH vs nix-path bug
  • Planning flakes stabilisation
    • Write down components for working on separately
      • fetchTree
      • lockfile
      • pure evaluation
      • evaluation caching
      • package schema
      • reference resolution
        • reference syntax
        • registry
        • version resolution

NIX_PATH vs nix-path bug

Flakes stabilisation

(recap of the NixCon discussions between @fricklerhandwerk, @infinisil, @roberth, @tomberek and others: Proposal to split flakes into their different aspects to stabilize them independently)

  • @edolstra: Be careful of the 2nd system effect. Now is the time to stabilize them, not to start proposing substantial changes
    • @fricklerhandwerk: Not suggesting incompatible changes, just attacking stabilisation of major components independently
  • @roberth: Flakes have been here for a long time, but people didn’t really have the opportunity to change them
    • @edolstra: Not true, barely anyone came up with proposals
  • @infinisil: Breaking up and rearchitecturing flakes is as important as the stabilisation itself
  • @edolstra: Doing that is research, it might take an unbounded amount of time
  • @fricklerhandwerk: We have some partial solutions that can be used independently, like the lockfile format. Cf. @lheckemann’ talk on flakes at NixCon.
  • @roberth: fetchTree is already a standalone component that could be stabilized independently
  • @edolstra: What would stabilizing lockfiles mean? There’s no way of consuming them from Nix
  • @roberth: stabilizing the lock file would also involve stabilizing the dependency injection, besides the lock file format itself
  • @tomberek: Plan towards getting closer to the finish line. But Flakes are made by strongly interconnected things that will be hard to really pull apart.
  • @fricklerhandwerk:
    • We didn’t have the setup for people to both seriously discuss proposals and implement them timely. This is different now that there is this team.
    • Breaking things apart should help parallelizing the work
  • @infinisil: What does it mean to finalize something? We should consider Flakes as an experiment that’s widely used but not properly designed. However we could fix those issues in a backwards compatible way after stabilizing.
  • @fricklerhandwerk: Concrete proposal: get funding for that split work. There is just a lot of work that we agreed has to be done but we collectively don’t have time for.
    • @edolstra: Disagree this would be much work. Barely any implementation changes needed.
    • @fricklerhandwerk: There may or may not be much to change in the code, but to solidify and if necessary refine the design we’d need RFC-style (not necessarily full RFC process) decision making, weighing the alternatives and considering long-term costs
  • @tomberek: What would be the minimal requirements for declaring Flakes stable?
    • Defining stable as “no backwards-incompatible changes”
    • Can we support it indefinitely without regreting it?
      • For instance, if we have an impending change to the lockfile format, should we wait for it?
  • (some discussion on whether we should uphold the value that Nix expressions shall be reproducible indefinitely for stable features)
    • (the team silently agreed, @infinisil raised some objections and counter-proposals)
  • @fricklerhandwerk: Doesn’t prevent us from working on uncontroversial things at the same time. We all seem to agree that we want the limbo state resolved as quickly as possible, the question now is how exactly.
  • Possible blockers for stabilisation:
    • flake-schemas extension: needs decision on blocker or not
      • extension can be added later, does not need to break flakes
      • decision: not a blocker
    • “CLI” flake schema (standardized output attributes): finalize
      • once made stable, schema conventions must persist
      • clarify what is under “undefined behavior” and what is “defined behavior” for CLI and official tooling
    • flake.nix format is not written in Nix language but looks very similar
      • a restriction that could be lifted
      • needs to be clarified for new users
      • conflicting requirements: consistency of language vs. restricted power for tracking metadata
        • flake.inputs.toml, flake.inputs.json (human) or in flake.lock (auto)
      • Needs discussion
    • registry
      • proposal: not used implicitly, consider a registry as an extension
      • Needs discussion, but there is likely a way forward
      • possibly in-scope: no official support for implicit registry usage inside flake.nix
      • proposal: retain CLI conveniences
    • lockfile format/semantics: final rework
      • proposal: blocking, this is a large change
      • proposal: not-blocking, existing usage
      • decision: based on implementation timeline
        • @edolstra proposes giving it a month from now
        • @fricklerhandwerk: strongly disagree with timelines or artificial time pressure
          • also there are still many open questions, so it’s to some extent design work that may or may not advance predictably
          • a month here is simply unrealistic. we’re already busy with CLI stabilisation and only make slow progress with that. right now we’re adding at least 4-5 new items that will require a lot more than one session to deal with.
          • (added in writing) we have no evidence that timelines ever worked here, we just do too many things concurrently and without schedule
            • for timelines we’d need a known amount of developer time available and an estimate of the effort; we have neither
    • locking mechanism: (clarify?)
      • What’s the behavior of nix flake lock? Are transitive lockfiles taken into account? Should it be pluggable like @lheckemann suggested?
      • likely part of CLI+flakes review, and part of the lockfile semantics issue above
    • call-flake.nix interface: add overrides/follows to API
      • (clarify? @roberth? @tomberek?)
        • @roberth: (in writing only) an overrides function is something I expect to get “for free” from the reuse input lockfiles change, as it needs such a function in call-flake.nix anyway
      • proposal: blocker
        • @roberth: (in writing only) adding a new pseudo-input meta turned out to be a breaking change! something similar would be needed for this
      • proposal: not a blocker, we have to support this anyway
    • CLI experience: review in a similar way as nix-command
      • agreement?
    • breakout pure eval
      • proposal: not a blocker
        • Should be a blocker if pure-eval is the default behavior for flake evaluation and there are questions about the semantics
    • breakout eval cache
      • proposal: not a blocker. Doesn’t impact the semantics
    • fetchTree
      • proposal: blocker, defines requirements for lockfiles, input declarations
  • will continue next Monday
10 Likes
2

So exciting to see a concerted effort in this regard. Love you all!

Do you have links to learn more about each of these “possible blockers for stabilization”?

3

I would really like to see lock files start looking more or less as such

{
  "__version": 1,
  "name": "something",
  "someMetadata": "whatever",
  "license": "MIT"
}
### 1
hash, name, etc
sha256-AAA, nixpkgs, etc
sha256-AAA, nixpkgs2, etc

This kind of lockfile is composed out of 3 parts:

  • Unstructured JSON for metadata. Everything that does not fit under dependencies goes there
  • ### 1 - split between 2 data structures. Number indicates which key of dependencies CSV format is a primary key
  • CSV structure - defines dependencies of the package (of flake?)

In case of nix this should get parsed to something like this:

{
  meta = {
    "__version" = 1;
    "name" = "something";
    "someMetadata" = "whatever";
    "license" = "MIT";
  };
  deps = {
    nixpkgs = {
      hash = "sha256-AAA";
      name = "nixpkgs";
      etc = "etc";
    };
    nixpkgs2 = {
      hash = "sha256-AAA";
      name = "nixpkgs2";
      etc = "etc";
    };
  };
}

Goals of such structure:

  • Reduce size
  • Increase readability
  • Simplify git conflicts
  • Being flexible enough to be a great fit for most cases (nix, npm, cargo, etc)

I have not yet written a spec/parser for this but it sounds like its about time

CC: @fricklerhandwerk, @roberth, @Infinisil, @edolstra, @thufschmitt, @tomberek, I would appreciate it if you guys could discuss this or a similar approach in the next Nix team meeting. Feel free to DM me if you guys need some better examples or POC

4

Mixing two formats in a file like this is an extremely weird idea. Especially since this is not human readable data, and any format can encompass the data of the other. You even show how this could look like in plain JSON in the second code snippet, so why require writing a parser instead of using that directly?

2 Likes
5

I totally agree, it looks weird but I feel like that’s the only sane option.
most data formats are relatively slow to parse and are very verbose. JSON is pretty fast but at least in the case of lock files - it is not very readable and solving git conflicts is a massive pain

Also, may I add that we are having an issue in nixpkgs with npm lock files due to their size and the fact that they are constantly changing. While this does not entirely solve the problem - it heavily alleviates it

6

Not that I’m aware of, that’s why I’m posting all these notes to have something at all, somewhere. We’ll expand on the tracking issue as we go: Start a Flakes milestone list · Issue #8870 · NixOS/nix · GitHub

We haven’t exactly agreed yet about how to proceed overall, but I’m sure there will be room to go into these details. There are enough people interested in that and you can start working on it already. Maybe just open a repo following @infinisil’s pattern to discuss larger issues where a design document is the desired outcome: [RFC 0138] Developing RFCs in repositories by infinisil · Pull Request #138 · NixOS/rfcs · GitHub

1 Like
7

These are also covered by Reuse input lock files · Issue #7730 · NixOS/nix · GitHub.

  • Most transitive dependencies do not need to be recorded (reduce size, fewer conflicts)
  • No more duplicate node names that need to be numbered arbitrarily (fewer conflicts)
  • More readable because the lock file more closely reflects the inputs declaration (readability)
  • Being flexible enough to be a great fit for most cases (nix, npm, cargo, etc)

Not sure where you want to take this idea. Something like the locking infra of dream2nix? They’re onto something and I’d like locking to be less ad hoc…

8

These are also covered by Reuse input lock files · Issue #7730 · NixOS/nix · GitHub.

:+1:

Not sure where you want to take this idea. Something like the locking infra of dream2nix? They’re onto something and I’d like locking to be less ad hoc…

I’d like to have a simple flexible lock file format that works well with git. I actually created an issue on dream2nix repo to see their thoughts New lockfile format? · Issue #694 · nix-community/dream2nix · GitHub. Also, I pinged bun developers on Discord since they are using binary lockfiles which I’d say is even more bothersome

9

I suppose git might work better if each entry is on a single line, which is feasible within JSON - not resorting to a custom format and parser.

wow…

10

wow…

Let’s be honest here, if nix had support for bison - we would do the same for npm packages in nixpkgs :sweat_smile:

11

I have to mention that I nerdsnipped @tomberek on building a Packrat parser (Packrat parser - Wikipedia) builtin in Nix and he reported interesting findings here: You're invited to talk on Matrix

I still believe it would be awesome to have an “all you can eBNF-ize” buffet for parsers in Nix via such a builtin and having it performant would enable some interesting stuff IMHO for lockfiles too.