Nix team report 2022-10 — 2023-03

Announcements
1

This is the first report of the Nix team of maintainers. The team was created in October 2022. The report covers the first six months of our work, with the purpose to reflect on what we did, how it went, and what to do next.

Authors: @edolstra @thufschmitt @fricklerhandwerk @tomberek @roberth @Ericson2314

Key activities

  • Drafted a vision statement without conclusion
  • Set up a team handbook, establishing a work routine, and a review and decision-making process
  • Discussed and prioritised bug reports and issues
  • Evaluated the merits and implementation strategies of pull requests
  • Discussed open design questions that block work on issues or getting pull requests merged
  • Discussed overarching or longer-term technical and procedural issues in the team, the project, and the ecosystem
  • To a lesser extent, collaboratively reviewed pull requests

Progress on stated objectives

In the following we evaluate how well we fulfilled our responsibilities as originally stated in the team announcement. In the meantime we clarified and narrowed down our objectives, and will use the new version in the next report.

Establish, communicate, and maintain a technical roadmap

  • We drafted a vision statement and refrained from proceeding, as there were too many open questions without resolution in sight.
  • We did not make any significant progress on a technical roadmap. We began working out a plan for stabilising the new CLI, but will now wait for the outcome of RFC 136.

Enable contributors

We discussed, reviewed, and merged multiple interesting PRs in collaboration with their authors, including:

  • Support for XDG directories (@balsoft)
  • Reading installable paths from stdin (@nrdxp)
  • More context in Nix language error messages (@layus)
  • Improved error messages on conflicting profiles (@bobvanderlinden)
  • Improved error messages related to trusted users and substituters (@solene)
  • Self-contained derivation outputs (@ncfavier)
  • Displaying the command documentation on separate pages (@balsoft)

We encouraged, reviewed, and merged PRs updating the hacking guide (@yorickvp) and adding a CONTRIBUTING.md (@bobvanderlinden).

Ensure capacity for reviewing PRs and addressing issues

The team’s workflow is by now quite well established, and we dedicated a significant amount of total time to the team’s stated goals. We have a rather predictable workflow, formalised in our team handbook, that proved suitable for reviewing and shepherding small to medium changes and making narrowly-scoped design decisions.

Most of the work was still done by team members themselves though. While we helped multiple authors to make progress on their PRs by answering questions, making design decisions, or helping with more involved technical details, many items got stuck in our pipeline because we still tend to lose track of them due to new topics taking our attention.

Overall, our meeting routine proved a solid basis for steering the project, and our current focus on fixing bugs and improving testing and documentation resulted in notable progress in these areas. At the same time, the synchronous process is a bottleneck, given team members can so far only exercise limited autonomy. This is due to very few explicit values and guidelines expressing team consensus. It is compounded by the lack of a more far-reaching vision that would further ease prioritisation.

John Ericson (@Ericson2314), after productively participating in team meetings since the beginning, officially joined the team in March 2023 and now has merge access to the Nix repository.

Define and assert quality criteria for contributions

We set up a PR template that prompts authors to state motivation and provide context, or add more information depending on the complexity or scope of the change. We have also established a checklist for PRs, but we have yet to follow it routinely. We added link checking to documentation. We helped with merging incremental improvements to the testing situation.

Keep code healthy, documentation up to date

We merged a series of refactorings of Nix store code and added multiple tests.
The unit tests of the store layer core data types have been greatly expanded, and included property tests.

We made many significant improvements to the Nix reference manual:

  • Documentation rendering code refactored for improved readability
  • Link checking was added
  • Multiple additions to the glossary
  • Reworked and expanded documentation on operators and paths
  • Added many details on various settings
  • Provided more details on installation and deinstallation

Manage the release lifecycle

We have established the goal of improving the release process. To this end we have documented the release process. Releases are not signed by any specific person anymore.

Unfortunately we have had a regression on each release during the team’s operation, highlighting the need for improvements to the test suite.

Regularly publish reports on work done

We publish all meeting notes, complete with links to discussions on every issue and pull request, usually within a week. This is the first report, concerning the past 6 months.

Engage with third parties in the interest of the project

We did not engage with third parties as a team except for responding to a direct – literal – request for comments about a proposal to change the Nix language, and collecting feedback for this report.

Team members individually discussed related issues with various people

Notable pull requests merged

Since only the most significant user-facing changes appear in the release notes, here is a more complete list of notable pull requests that we merged in this period. The list is grouped by topic and sorted by time of completion, and gives an impression of how we distributed our efforts.

Installer

  • #7500 Updated docs to delete build users and group

  • #7607 Expand installation instructions

  • #7418 Add GitHub template for installer issues

  • #7547 Fix Nix installation on older versions of fish

  • #7551 Include macOS sandbox files in the Nix binary

  • #7411 Remove GPG-signing of releases

  • #7132 Fix fish shell MANPATH creation

Bug fixes

  • #7283 Fix requires non-existent output (a non-repeatable failure)

  • #7725 Disable GC during coroutine execution (Mitigate evaluator GC crash on Darwin)

  • #7796 Flakes: Ensure that self.outPath == ./.

Documentation

  • #7006 Manual: generalize anchor redirects

  • #7541 Check links in the manual

  • #7913 Add information on the build-hook setting

  • #7754 Documentation: builtins.fetchGit when used on a local path

  • #7318 Add “instantiate” to glossary

  • #7714 doc: add __structuredAttrs, outputChecks, unsafeDiscardReferences

  • #7490 Define the terms “realise” and “valid” for store paths in the glossary

  • #7628 doc: Update binary-cache-substituter.md

  • #7498 Refactor documentation of operators, document + for strings and paths

  • #7278 antiquotation → string interpolation

  • #7066 Manual: architecture overview

  • #7485 Define “store derivation” in the glossary

  • #7300 Manual: Add links to GitHub source, set title to “Nix Reference Manual”

  • #7191 Explain how Nix handles antiquotation of paths

User Experience

  • #7594 read installable paths from stdin

  • #5588 Follow XDG Base Directory standard

  • #7754 Scope down --derivation to just the commands that use it

  • #7474 Use a pipe for all install commands

  • #7734 nix-shell: Colour the prompt red if the user is root

  • #5226 Move the default profiles to the user’s home

  • #6988 nix flake show: don’t evaluate derivations for foreign systems by default

  • #7087 Self-contained derivation outputs

  • #7606 flake check: Recognize well-known community attributes (lib, darwinConfigurations, …)

  • #7504 nix develop: Set Linux personality

  • #7423 Support flake references in the classic CLI

  • #3600 Automatic UID allocation

Error messages

  • #7788 Improve error on conflict for nix profile install

  • #7461 Improve warning when an untrusted user is using an untrusted substituter

  • #6204 Add context to better locate runtime coercions

  • #7473 Improve SQLite error messages

  • #7462 Inform user instead of warning them when using a trusted substituter

Testing

  • #7775 Add a basic daemon authorization test

  • #7713 Add more property tests

Developer experience

  • #7901 Add ./configure --disable-tests option

  • #7883 Add JSON design guideline

  • #7748 Split libcmd headers and files

  • #6538 Simplify flake for this repository

  • #7744 Factor out InstallableStorePath to its own file, deduplicate

  • #6815 OutputSpec for DerivationGoal and DerivedPath, today’s OutputSpec → ExtendedOutputSpec

  • #7577 .github: Add pull request template

  • #7412 Document the release process

  • #7433 Improve hacking.md and add clangd+bear to devshell

  • #7427 State priorities in triaging and discussion process

  • #7329 Add maintainers’ handbook

Goals for the next period

The team’s main responsibility is to set a direction for the development of Nix and ensure that the code is in good shape.

We aim to achieve this by improving the contributor experience and attracting more maintainers – that is, by helping other people contributing to Nix and eventually taking responsibility – in order to scale the development process to match users’ needs.

Objectives

For the next 6 months period, we aim to orient our activities around achieving the following objectives, against which we will measure our success:

  • It is obvious what is worthwhile to work on.
  • It is easy to find the right place in the code to make a change.
  • It is clear what is expected of a pull request.
  • There is a predictable process for getting a change merged and released.

Tasks

From the above objectives we derive the following tasks to focus on:

  • Establish, communicate, and maintain a technical roadmap
  • Improve documentation targeted at contributors
    • Record architecture and design decisions
    • Elaborate contribution guides and abide to them
    • Define and assert quality criteria for contributions
  • Maintain the issue tracker and triage pull requests
  • Help contributors succeed with pull requests that address roadmap milestones
  • Manage the release lifecycle
  • Regularly publish reports on work done
  • Engage with third parties in the interest of the project
  • Ensure the required maintainer capacity for all of the above
19 Likes
2

I just wanna say I’m happy to see this level of transparency.

5 Likes
3

Ditto. Its effects are compounding.

One minor procedural suggestion:

I’ve cought myself more than once actively seeking to open discourse looking for the Nix Team’s or the Nixpkgs Team’s reports as the point of reference that they are.

Sometimes I wanted to quickly navigate them to find a data point that I only faintly remember.

A help to that end for the consumer would be a separate discourse category with an appropriate name comprising the entire collection of the Community’s Accountability reports, such as Meeting Minutes and Status Reports.

To make in particular your (@fricklerhandwerk) easier with contextual report snippets (i.e
cross-posting), I wonder if you have scripts in place to ease your administrative burden.

This practice of yours is extremely useful and appreciated and I wonder of scalability considerations have already been made around it.

4

We have a search query to past meeting notes in the Nix documentation team handbook and should definitely add one for the Nix team. (Would be happy to merge a PR!) Having one for the docs team‘s monthly updates would be nice, too. In any case that information should be easily accessible from obvious entry points such as the teams’s info pages, but not have to be maintained by hand. There shouldn’t be an extra discourse category for this, in my opinion.

There is no automation in place, and ideally all of that would be just queries for particular GitHub activities. Right now we do quite some copy-pasting and manual linking during and after the meetings because we take notes collaboratively and neither GitHub nor Discourse support that kind of workflow.

I’d also strongly support a global view on community and team activities, and from recent discussions I take that @garbas, @idabzo, @btjh, @pxc, @jeremiahs and the rest of the marketing team are aiming towards that direction. But obviously it needs some work to set up some processes and it takes additional time to deal with all the details, and available time is scarce to begin with. There is also quite some friction around maintaining team pages on nixos.org, and there hasn’t been any significant progress on resolving that in the past year.

1 Like
5

Dear Community Member. If you had flagged this post, thank you for expressing your feelings, but before you flag it again, please be considerate and reciprocate the value or, at least, good intent, of this post to find a workable solution to @fricklerhandwerk assertion, that:

There is no automation in place […] Right now we do quite some copy-pasting and manual linking during and after the meetings because we take notes collaboratively and neither GitHub nor Discourse support that kind of workflow.

I know it’s an bit of a premature / non-entirely-serious suggestion, but we all love to unpack shiny new things:

MeetingMinutes 2023-04-29 { // script dispatch to Discourse Announcement (evtl. filtered)
  Agenda {
    fricklerhandwerk "Welcome everybody!"
    roberth "What do we have?"
  }
  General Discussion {
    item 1 "some"
  }
  #PR7788 { // script dispatch to PR comment in post-processing
    roberth "LGTM" {
      Ericson2314 "+1"
    }
    tomberek "Have we considered X?"
  }
}

The key of this suggestion is the non-verbosity of such a data description language [so that it can be used almost synchronously without getting in your way].