Tweag + Nix Dev Update #3

Development
1

Hello everyone,

Here is what Tweag’s Nix team has been up to lately.

Nix

  • Flakes: @edolstra worked on tab completion on flake outputs

  • Flakes: Nix can now nix-build inside a non-git tree, as long as the
    dir has a flake.nix.

  • @edolstra’s work on Nix 2.0:

    • removed Nix 2.0 commands from the NixOS tree. Since these have
      always been experimental, it isn’t good to depend on them.
    • did some work around the UX of the nix command, to allow
      for these commands to be categorized for discoverability. This
      categorization could also be used for things like porcelain /
      plumbing differentiation.
    • The command nix run is now called nix shell, and nix shell
      is now called nix dev-shell.
    • nix search can recurse in to package sets now

  • CAS: @thufschmitt has been working on extending the binary cache
    protocol to support substituting CAS builds.

  • Testing: @gilligan and @andir added over 75 unit unit tests to the Nix
    build after finding some peculiarities in Nix’s documentation of
    dirOf: Add unit tests by gilligan · Pull Request #3571 · NixOS/nix · GitHub

NixOps

class NeatCloudMachineDefinition(nixops.resources.ResourceDefinition):

  def __init__(self, xml):
    super().__init__(xml)
    self.store_keys_on_machine = (
      xml.find("attrs/attr[@name='storeKeysOnMachine']/bool").get("value")
      == "true"
    )

they can now define a type-checked and validated object, and get
confident configuration:

class NeatCloudMachineOptions(nixops.resources.ResourceOptions):
    storeKeysOnMachine: bool

class NeatCloudMachineDefinition(nixops.resources.ResourceDefinition):

    config: MachineOptions

    store_keys_on_machine: bool

    def __init__(self, name: str, config: nixops.resources.ResourceEval):
        super().__init__(name, config)
        self.store_keys_on_machine = config.storeKeysOnMachine

Simplify and make extensible:

  • @adisbladis has been working on moving NixOps’ evaluation to the module
    system, to deprecate the bizarre and somewhat inconsistent model
    of the previous evaluation model.
  • @adisbladis’s PR removing auto-luks and auto-raid0 has merged, and those
    modules now live in a separate repo, since they actually aren’t
    unique to NixOps: GitHub - nix-community/nixos-modules-contrib: NixOS modules that don't quite belong in NixOS.
  • @adisbladis has replaced the use of scp with rsync in NixOps, which
    works very much the same way as scp but supports specifying a remote
    command. This is useful for the rootless deployments.
  • @adisbladis reimplemented encryptedLinksTo as a plugin, but holding that
    for now. putting that on a pause until the XML to JSON PR is merged.
  • State backends:
    • Locks and Storage are now split so S3 storage could use something
      other than dynamodb for locking.
    • The memory backend is going to be removed, since it has some
      implications which shouldn’t be addressed in the same PR.
    • I’m blocking this merge until the testing is done.

Testing:

  • @adisbladis has been working on NixOps testing with podman. I have been
    experimenting with this PR, and working to make it super fast and
    super reliable, so we can easily extend and depend on tests written
    with it.

My NixOps theme for this week is focusing on plugins. A number of
changes merged recently have broken plugins when trying to use NixOps
master. I’m glad people are using it and keeping up, and it is a
bummer to find the plugins broken. In particular, I’ll be starting
with the list at the top of:
https://nixops.readthedocs.io/en/latest/plugins/authoring.html

Each week except for last week, which was a holiday in the UK @adisbladis
and I have been doing regular public calls to review PRs and issues
against NixOps. If you’d like to come participate, they typically
happen America/New_York’s morning around 11:30 on Fridays. We do some
other calls from time to time, too, but on less schedule.

Lorri

  • @Profpatsch has continued reviewing Lorri pull requests.

NixOS

  • I have been working on organizational questions around how to build
    and manage a security team with regular “on-call”-style rotation and
    responsibilities. This is with the goal of getting on the embargoed
    list and doing a better job overall. In this effort I’ve reached out
    to the Arch security team and have talked a bit with @andir and some
    NixOS users. I’m interested in talking with anyone who is interested!

Marketing

@garbas has been hard at work on the pagination of search results in
GitHub - NixOS/nixos-search: Search NixOS packages and options. This is almost on-par with the
existing Package and Option search.

@garbas has also been working on the landing page for nixos.org. It turns
out to be pretty hard to find the right words to say.

He’s been experimenting with something like three “buttons”, showing
a flow from “Develop” to “Build” to “Deploy”. Each of these would
feature unique aspects which makes Nix amazing.

Hydra

@gilligan worked with @edolstra and I to create an OpenAPI spec for Hydra:
Add hydra OpenAPI description by gilligan · Pull Request #750 · NixOS/hydra · GitHub He also spent a good bit of
time and did a nice clean-up of the Hydra issue tracker, closing out
old issues which don’t seem applicable anymore.

This makes me wonder about how to create a regular cadence of
reviewing tickets on these projects. I’m not sure what to do with that
yet.

Poetry2nix

@adisbladis has worked on assorted poetry2nix bugs, like this semVer-related
python version check: Fix semver pythonVersion checks by adisbladis · Pull Request #93 · nix-community/poetry2nix · GitHub

I want to talk a little about what inputs go into these weekly
updates. Tweag currently has @adisbladis , @garbas, @edolstra, and me allocated
full-time to work on Nix. Like all teams within Tweag, we synchronize
regularly:

  • daily voice standups (10 minutes max),
  • weekly written check-ins by everybody.

These updates that I publish on this Discourse are from these standups
and weekly check-ins. Transparency and honesty is incredibly important
to me, and I hope that rings true. If you have any questions or
comments about these updates, please ask. We’re always looking
for more ways to keep you informed about our progress.

Thanks everyone!

21 Likes
2

This is helpful to see where the ongoing effort is. Keep up the updates!

1 Like
3

Thank you all!

(Extra words because discourse has a minimum length for posts.)

1 Like
4

These kinds of questions are very interesting to me! I’ve been developing some ideas based on Pennarun’s talk that touches on triage, though nothing concrete yet. My current thought is to triage all tech-debt style tickets (those that could improve QoL but might never make it high on the product roadmap) into a Tech Debt Bucket. Then, each dev cycle would have a “debt payback” story that draws a few tickets from the bucket at random. As for the un-triaged backlog, I just try to triage a few more every day based on ideas from the talk.

It’s just an experiment, and one that hasn’t survived one whole cycle yet (although other ideas from that talk have already been useful to me). Maybe “a few at random” is a terrible criterion. :smile: We’ll see!

Good luck, glad to see so much progress!

4 Likes