Tweag + Nix Dev Update

Development
1

Hey everyone,

I was thinking it’d be cool / useful to post some info on a regular basis about Nix projects Tweag is working. This is the first one of many, where we’re shooting to do a weekly post or so. The motivation behind Tweag’s work is to supplement the community: doing work to create and improve the tooling and ecosystem overall. We work pretty hard to make as much of our work open source as possible – and when possible, generally applicable, and so talking about it a bit seems like a good next step!

Documentation Formats

A few weeks ago, I asked @nlewo and @adisbladis to reformat the Nixpkgs documentation in to RST and Asciidoc. I’m really glad and excited to see that they were able to publish their work on the RFC:

I was wanting to have someone do a markdown port, but that hasn’t materialized yet, and I’m not sure it is on the horizon. For now, if you want to take a crack at it, would say don’t wait up for us.

NixOps

@adisbladis and I have been working on NixOps quite a bit lately. It started as a bit of a lark a few weeks ago when I tried porting NixOps core to Python3, and ended up getting quite a good bit of community effort in adding mypy types. Since then, some pretty cool changes have taken place:

  • NixOps uses GitHub Actions
  • each PR is checked that it is formatted accoriding to black
  • each PR is checked that its tests pass
  • each PR is checked that it does not decrease type coverage

A big thank you from community TheWizardTower, AmineChikhaoui, doshitan, lovesegfault, bhipple, and adisbladis for this work.

We have also added a nixops deploy --test feature, which mirrors nixos-rebuild test’s behavior of switching to the new configuration, but not updating the bootloader. This and a PR outlining deploy targets, creating a sort of two-phase commit protocol for network deployments (Deploy Targets: Policy/Behavior-free Deployment Hooks (auto-rollbacks, drain events, etc.) by grahamc · Pull Request #1245 · NixOS/nixops · GitHub) was for http://www.yakkertech.com/.

@adisbladis has also submitted some very cool PRs:

He also submitted a PR allowing for stateless deployments: WIP: Add deploy without creating a state file by adisbladis · Pull Request #1248 · NixOS/nixops · GitHub which inspired a PR by me, allowing networks to define a state backend: Example NixOps State Backends by grahamc · Pull Request #1264 · NixOS/nixops · GitHub

My initial PR includes an in-memory backend for stateless deploys, a legacy backend for deployments keeping the state in ~/.nixops, and then an example S3 backend which stores the state in S3 with encryption and uses dynamodb for locking. The S3 backend does not belong in NixOps core (instead belonging in the nixops-aws plugin), this was just an example and will be removed.

I’ve been using this S3 state file mechanism in deployments for ofborg, r13y, my personal network, and a couple others since – and to great effect.

This change to state backends makes it possible to use NixOps statelessly (though additional changes needed, like @adisbladis’s PR to disable provisioning SSH keys automatically.) The S3 backend makes many company’s dreams come true: being able to deploy from a CI pipeline and from multiple developer machines without needing to check a the statefile in to the Git repository. This is a huge feature which, among other recent work, has reignited people’s excitement in NixOps. I have been grateful to be able to work with Adam on this.

In general, this has sparked an interest in creating a NixOps 2.0, an idea I’ve started writing down notes for at Release 2.0 old tracking issue · Issue #1242 · NixOS/nixops · GitHub. One thing I’m trying to stay cautious of is avoiding second-system syndrome, and keeping goals achievable.

Nix

I’ve been talking with @edolstra about using the CXX crate for integrating Nix’s C++ and Nix’s Rust. He was saying it is very similar to the method he’s already using. One advantage of using CXX is fixes there helps the overall community, and hopefully many eyes will make at least some of the potential bugs more shallow.

He’s also been working on backporting most of the non-flake development work on the Flakes branch back in to Nix master. For example, this includes fetcher abstraction layers and UI framework improvements.

fetchGit has also gained support for shallow Git fetches.

On the Flakes front, Eelco has reimplemented a lot of the logic as regular Nix expressions – making it much easier to iterate and experiment.

At any rate, this is just a sampling of things we’ve been up to, and going forward it will be much easier to be thorough. Let me know if you have any questions!

Graham

24 Likes
2

This all sounds great! Thanks for everyone’s hard work.

I wasn’t aware that Rust was used within Nix itself. What parts of Nix are currently in Rust?

1 Like
3

Is there any release plan for the flakes brach?

1 Like
4

Rust: Right now very little is in Rust, you can see it here: https://github.com/NixOS/nix/tree/master/nix-rust I believe it mostly is a few hash functions. So far the Rust work has been done in a way where it could be easily removed if it turns out to be too tricky to bridge the gap between C++ and Rust. Maybe CXX will get us over that bridge.

Flakes release: I don’t know of a timeline right now, there has been a good bit of experimentation even recently – but I think even if it merged today it’d be behind an experimental flag disabled by default.

2 Likes