NixOS vs Imperative Development Setups

I’m trying to switch my work laptop from Fedora to NixOS, but I’m still having problems how to map my software development workflow to NixOS.

Consider the following scenario. I have three (C/C++) projects: A, B, C. Each of these projects takes a considerable time to build and the dependency chain is C depends on B depends on A plus each of these has non-trivial additional dependencies. Once all of these are built, they need to deployed to a different machine for testing.

On Fedora, my development workflow looks like this: I install all dependencies of A,B,C globally using -devel packages. Then I checkout the repos of these projects next to each other and configure the build systems to consume these development checkouts. After one sequential build, I’m deploying everything to the target system with a shell script.

This is painful to setup, but once it is setup, it’s very fast to change code in any of the components and quickly build/deploy and test.

Moving to NixOS:

The easiest option to replicate this setup is to build all components with Nix after overriding all the dependencies to point to the right repository. Then I can deploy using nixops to my test machine. This is very easy to setup, but somewhat useless in practice, because any change in A requires a full rebuild of everything. Even with ccache this is prohibitively slow compared to the manual setup.

The alternative is using nix-shell. This only works for the package at the end of the dependency chain ( C ) and gets rid of most of the build times for C, because I can build incrementally. Also now I have a problem deploying the resulting binaries, because the nix store paths they reference may not be on the target machine.

I’m not sure what the best solution here is. Maybe I should create a unified shell.nix with dependencies for all projects. This would mean I get the incremental build experience as on my Fedora setup, but it still leaves the problem of how to deploy the dependencies to the target machine.

I’m eager to hear how other people do this!

1 Like

I rarely do development (in the sense of programming), so I’m watching this from a somewhat different perspective.
Your observations are correct, if you’ve got programs with long compile times that can be quite annoying. I notice that everytime my laptop has to rebuild virtualbox with that addon pack, which takes maybe 45 minutes (never really bothere to measure it).
I think development with nix-shell works pretty well. It’s up to you to decide how deterministic that has to be.
But once things get sent of to test and production systems I really want them to be reliable in the Nix-sense. Just replacing the production binary with a binary I built in an unclean development environment feels “dirty” and risky to me.
Development produces a result (source code, not binaries) and I test that result on a test system. If it survives the test, I can be pretty confident that it will work the same way on the production system, because everything is clearly defined and comes from a clean environment.
And you can even save time building for the production system, as the binaries for the test system is clean as well.

The big bonus is that you are forced to have a clean build when deploying to test. You can be confident that it will work and that you can reproduce the results at a later time as well.
Nix isn’t going for “it only has to work once”. Nix is about “let’s try to make sure it will work every time”.
If your development system dies, without Nix you’ve got to take care of the same setup and have to hope that everything still works because versions have changed and no dependencies broke.
With Nix you just copy over the files, tell it to get crunching, drink a coffee and you’re back in business.

Working in nix-shell (or lorri!) is very nice as long as it is one project you are working on. The moment you try to implement a new API in a library and at the same time prototypically implement it in some app, that’s where the complications start. I totally agree that “production” builds are better done with Nix itself, but even the most reproducible framework will not convince developers if it turns their 5s development turnaround time into a 15min.

That’s why I’m eager to hear how other people structure their development flow on NixOS.

Working in nix-shell (or lorri!) is very nice as long as it is one project you are working on.

Well, depending on what you call one project, you could have a single nix-shell with multiple repositories checked-out inside and add the local library build output directory into NIX_LDFLAGS or LDFLAGS or something like that?

If both machines have nix installed (not necessarily NixOS), you can also use nix-copy-closures to transfer the build outputs plus the dependency closure to the other machine.

I’m also very interested in this kind of workflow. Currently, I’m struggling with the separation into two worlds:

Editing source, building manually, running tests of a single project using nix-shell is all well and good, but it’s really only good for those single projects or when the combination of multiple projects is very simple.

Seeing that project in action (e.g., it’s an input to further nix expressions) and testing if all the expressions you have work as expected often requires nix-build, and then everything is built from scratch.

Examples of where this gets tricky are:

  • Building large projects like VirtualBox and integrating them for example into an initrd created by nix expressions, where you frequently want to edit/re-build VirtualBox and the resulting initrd for testing
  • Building custom binaries, a Linux kernel and initrd, and combining all of them into a GRUB standalone image for testing

In my opinion, what’s missing is a feature that swaps out the default source/build directory handling with a local checkout (so no copying of the source code or deleting the build folder). I could imagine something like this:

  1. nix-build --incremental --initialize foo.nix -A

This would just create a dedicated build directory, run the initial phases (e.g., configure), but point directly to a local checkout of the source, where I also do my editing. Of course this is a very limited special case where I have the source in a local directory under version control, but it’s the main use case for this kind of development model.

  1. nix-build --incremental foo.nix -A

This would just run the build and packaging steps on the existing build folder, essentially triggering an incremental build and then doing all the things it would usually do to create the desired output which can be used in further expressions.

Now, I’m well aware that there will be plenty of details that make this complicated or even a bad idea, but it’s the kind of semantics I’m missing. If there are other ways to achieve that, I’d be very happy to experiment.

1 Like

If the result of nix-build is what you also get via make install, you could think about packing this up and using this as build input to any dependencies you need to build.

So let’s say you had largeLibrary and largeProject. Instead of using largeLibrary’s derivation in largeProject, you change largeProject’s Nix expressions to be able to consume an incrementally built and installed version of largeLibrary.

I’m not sure I’m being very clear here. :wink:

If there was a convenient way to do that, of course. What I would not want is to have temporary modifications to largeProject’s expressions just to do what I need and then stash or remove them as soon as I succeeded in what I was trying to do. Ideally, the switch from the standard build process to using locally built/installed versions should not be much more than a flag, which preferably could be specified on the command line.

My suggested workflow is shared nix-shell for development + CI that builds and caches results so deploys are fast.

Unfortunately, that approach does not cover the scenario that is quite common in my workflow: Produce initrds or other forms of combined images (GRUB standalone) that contain those incrementally built binaries for testing on a test machine that boots via network (or, more generally speaking, retrieves those combined images for testing). It’s extremely convenient to create things like initrds and GRUB images in nix expressions, but once you left the nix-build world (and use nix-shell to build the individual components), you have to stay in that world and perform the remaining steps by hand as well. There is no way back into the nix-build world, or at least I’m not aware of one.

Here’s my personal experience on how to develop projects that depend on each other using nix-shell at my workplace (We’re using C++ and python).

  • Initially I did write a default.nix which contained a derivation per project and exclusively used nix-build -A projectA to build and run the projects tests. This got annoying pretty quickly as you cannot make use of incremental builds this way.
  • I did then create a shell.nix with all “common” dependencies of all projects and a single Makefile that would install each project into a mutable ./virtualenv directory using the python develop mode for python projects. I’m using this now for doing quick iterations without having to rebuild everything when working on the lowest level library. It does however have the downside of requiring a rm -rf ./virtualenv once in a while to get a really clean build.
  • Since I had the build descriptions of each project still lying around, I did set up a CI using the definitions in the default.nix from my initial configuration.
  • At one point I was a little unhappy that I did have to maintain both the individual builds + the shell.nix which contained all the shared derivations. So I tried re-using these individual expressions in a common multiShell via this: https://github.com/knedlsepp/nix-shell-for-python-multi-project-repos/blob/bd60bd0001448f8f44d6aeb71d2e186c7fd5d523/shell.nix. I didn’t yet completely move over to that solution however.

So I’m now at a point where I mostly use my Makefile + shell.nix with all common dependencies for interactive tasks. I do sometimes however use the nix-build locally to get a really clean build.

One of my colleagues has a really neat solution where his emacs is configured to always automatically switch to the environment that you would get from running nix-shell -A projectX whenever he switches his buffer to a file in projectX. This is all based on lorri and doesn’t require setting up the “shared dependency shell.nix”. It’s quite nice since you get the benefits of an always clean build + interactive incremental rebuilds.

There is also a new option for nix develop upcoming with flakes that is targetting exactly your use case: nix develop --redirect (See: nix develop: Add --redirect flag to redirect dependencies · NixOS/nix@f9438fb · GitHub). I have not personally tried that yet.

1 Like

Thanks for the thoughts. This is indeed very helpful. I particularly like the shared shell.nix idea. The upcoming nix develop command also looks interesting.

I think the last problem we have is working with projects that need to be extensively patched to build with Nix. VirtualBox comes to mind. Using buildFHSEnv to get around this seems promising, though.