I was thinking it’d be cool / useful to post some info on a regular basis about Nix projects Tweag is working. This is the first one of many, where we’re shooting to do a weekly post or so. The motivation behind Tweag’s work is to supplement the community: doing work to create and improve the tooling and ecosystem overall. We work pretty hard to make as much of our work open source as possible – and when possible, generally applicable, and so talking about it a bit seems like a good next step!
A few weeks ago, I asked @nlewo and @adisbladis to reformat the Nixpkgs documentation in to RST and Asciidoc. I’m really glad and excited to see that they were able to publish their work on the RFC:
- @nlewo’s, rst: https://github.com/NixOS/rfcs/pull/64#issuecomment-601876791
- @adisbladis’, asciidoc: https://github.com/NixOS/rfcs/pull/64#issuecomment-603492309
I was wanting to have someone do a markdown port, but that hasn’t materialized yet, and I’m not sure it is on the horizon. For now, if you want to take a crack at it, would say don’t wait up for us.
@adisbladis and I have been working on NixOps quite a bit lately. It started as a bit of a lark a few weeks ago when I tried porting NixOps core to Python3, and ended up getting quite a good bit of community effort in adding mypy types. Since then, some pretty cool changes have taken place:
- NixOps uses GitHub Actions
- each PR is checked that it is formatted accoriding to
- each PR is checked that its tests pass
- each PR is checked that it does not decrease type coverage
A big thank you from community TheWizardTower, AmineChikhaoui, doshitan, lovesegfault, bhipple, and adisbladis for this work.
We have also added a
nixops deploy --test feature, which mirrors
nixos-rebuild test's behavior of switching to the new configuration, but not updating the bootloader. This and a PR outlining deploy targets, creating a sort of two-phase commit protocol for network deployments (https://github.com/NixOS/nixops/pull/1245) was for http://www.yakkertech.com/.
@adisbladis has also submitted some very cool PRs:
- switching to Poetry for the nixops build: https://github.com/NixOS/nixops/pull/1256 I hope to be able to merge this one this week.
- disable automatic SSH key generation: https://github.com/NixOS/nixops/pull/1247
- support for deploying as a non-root user: https://github.com/NixOS/nixops/pull/1270
He also submitted a PR allowing for stateless deployments: https://github.com/NixOS/nixops/pull/1248 which inspired a PR by me, allowing networks to define a state backend: https://github.com/NixOS/nixops/pull/1264
My initial PR includes an in-memory backend for stateless deploys, a legacy backend for deployments keeping the state in ~/.nixops, and then an example S3 backend which stores the state in S3 with encryption and uses dynamodb for locking. The S3 backend does not belong in NixOps core (instead belonging in the nixops-aws plugin), this was just an example and will be removed.
I’ve been using this S3 state file mechanism in deployments for ofborg, r13y, my personal network, and a couple others since – and to great effect.
This change to state backends makes it possible to use NixOps statelessly (though additional changes needed, like @adisbladis’s PR to disable provisioning SSH keys automatically.) The S3 backend makes many company’s dreams come true: being able to deploy from a CI pipeline and from multiple developer machines without needing to check a the statefile in to the Git repository. This is a huge feature which, among other recent work, has reignited people’s excitement in NixOps. I have been grateful to be able to work with Adam on this.
In general, this has sparked an interest in creating a NixOps 2.0, an idea I’ve started writing down notes for at https://github.com/NixOS/nixops/issues/1242. One thing I’m trying to stay cautious of is avoiding second-system syndrome, and keeping goals achievable.
I’ve been talking with @edolstra about using the CXX crate for integrating Nix’s C++ and Nix’s Rust. He was saying it is very similar to the method he’s already using. One advantage of using CXX is fixes there helps the overall community, and hopefully many eyes will make at least some of the potential bugs more shallow.
He’s also been working on backporting most of the non-flake development work on the Flakes branch back in to Nix master. For example, this includes fetcher abstraction layers and UI framework improvements.
fetchGit has also gained support for shallow Git fetches.
On the Flakes front, Eelco has reimplemented a lot of the logic as regular Nix expressions – making it much easier to iterate and experiment.
At any rate, this is just a sampling of things we’ve been up to, and going forward it will be much easier to be thorough. Let me know if you have any questions!