NixOS vs Imperative Development Setups

I’m trying to switch my work laptop from Fedora to NixOS, but I’m still having problems how to map my software development workflow to NixOS.

Consider the following scenario. I have three (C/C++) projects: A, B, C. Each of these projects takes a considerable time to build and the dependency chain is C depends on B depends on A plus each of these has non-trivial additional dependencies. Once all of these are built, they need to deployed to a different machine for testing.

On Fedora, my development workflow looks like this: I install all dependencies of A,B,C globally using -devel packages. Then I checkout the repos of these projects next to each other and configure the build systems to consume these development checkouts. After one sequential build, I’m deploying everything to the target system with a shell script.

This is painful to setup, but once it is setup, it’s very fast to change code in any of the components and quickly build/deploy and test.

Moving to NixOS:

The easiest option to replicate this setup is to build all components with Nix after overriding all the dependencies to point to the right repository. Then I can deploy using nixops to my test machine. This is very easy to setup, but somewhat useless in practice, because any change in A requires a full rebuild of everything. Even with ccache this is prohibitively slow compared to the manual setup.

The alternative is using nix-shell. This only works for the package at the end of the dependency chain ( C ) and gets rid of most of the build times for C, because I can build incrementally. Also now I have a problem deploying the resulting binaries, because the nix store paths they reference may not be on the target machine.

I’m not sure what the best solution here is. Maybe I should create a unified shell.nix with dependencies for all projects. This would mean I get the incremental build experience as on my Fedora setup, but it still leaves the problem of how to deploy the dependencies to the target machine.

I’m eager to hear how other people do this!

1 Like

I rarely do development (in the sense of programming), so I’m watching this from a somewhat different perspective.
Your observations are correct, if you’ve got programs with long compile times that can be quite annoying. I notice that everytime my laptop has to rebuild virtualbox with that addon pack, which takes maybe 45 minutes (never really bothere to measure it).
I think development with nix-shell works pretty well. It’s up to you to decide how deterministic that has to be.
But once things get sent of to test and production systems I really want them to be reliable in the Nix-sense. Just replacing the production binary with a binary I built in an unclean development environment feels “dirty” and risky to me.
Development produces a result (source code, not binaries) and I test that result on a test system. If it survives the test, I can be pretty confident that it will work the same way on the production system, because everything is clearly defined and comes from a clean environment.
And you can even save time building for the production system, as the binaries for the test system is clean as well.

The big bonus is that you are forced to have a clean build when deploying to test. You can be confident that it will work and that you can reproduce the results at a later time as well.
Nix isn’t going for “it only has to work once”. Nix is about “let’s try to make sure it will work every time”.
If your development system dies, without Nix you’ve got to take care of the same setup and have to hope that everything still works because versions have changed and no dependencies broke.
With Nix you just copy over the files, tell it to get crunching, drink a coffee and you’re back in business.

Working in nix-shell (or lorri!) is very nice as long as it is one project you are working on. The moment you try to implement a new API in a library and at the same time prototypically implement it in some app, that’s where the complications start. I totally agree that “production” builds are better done with Nix itself, but even the most reproducible framework will not convince developers if it turns their 5s development turnaround time into a 15min.

That’s why I’m eager to hear how other people structure their development flow on NixOS.

Working in nix-shell (or lorri!) is very nice as long as it is one project you are working on.

Well, depending on what you call one project, you could have a single nix-shell with multiple repositories checked-out inside and add the local library build output directory into NIX_LDFLAGS or LDFLAGS or something like that?

If both machines have nix installed (not necessarily NixOS), you can also use nix-copy-closures to transfer the build outputs plus the dependency closure to the other machine.