Thanks for the writeup.
I was wondering as well, if nix flakes could be used instead of a makefile.
I am doing a similar thing: Generate some code, if the sources changed, compile a lib, use that lib in a UI framework, check the result in an emulator and release compile it to a binary - though not for a website, but a rust-flutter mobile app (GitHub - patmuk/flutter-rust-bridge_crux_style: minimal project applying flutter-rust-bridge in a crux style!).
That last link is my “template”, I recently created a flake.nix to work in my real project.
For the steps above I use a justfile. And now I execute the commands in it using nix (develop).
But can I get rid of the justfile (or makefile for others) completely? I don’t care about non-nix-user compatibility, prefer reduced complexity instead.
So, my concrete question: Is nix as well checking dependencies in a development project? Say my source files changed and code generation is needed - would nix detect that and execute the codegen step when I ask to run build? Nevermind the phases, would the check phase execute the build phase only, if the sources changed or always?
Or does one need any kind of make(, just, …) for that functionality?
@pxc How did your “nix only” path continued?
Oh, I didn’t write this! I just shared the article, which happens to use the first person, and that got captured in the autogenerated summary.
Ah 
Than thanks for sharing! And forwarding to the author.
By any chance - do you have experience with this setup?
Is nix as well checking dependencies in a development project? Say my source files changed and code generation is needed - would nix detect that and execute the codegen step when I ask to run build?
Technically yes, but on a per-repository level. So if any file in your sources changes, Nix will trigger a complete rebuild. Unless your builds are very fast, you probably don’t want to use Nix as the only build tool.
Now one could entertain the thought of splitting a Nix build up and, for a C example, build each object file as its own Nix recipe. But I don’t think there is tooling for that to any extend and you’d have to re-solve a lot of things in Nix that other build-systems already do.
I find what works well is a combination of a devShell for development where you work in a traditional way, i.e., (incremental) builds through Make/Ninja/Whatever and then a Nix recipe that builds everything in isolation and is the basis for releases. And making your Makefiles work well with Nix devShells, e.g., by using enviroment variables for paths to compilers and other tools, will also make them easier to adapt to other environments. It’s a good practice to get into.
Thanks a lot for the fast answer and your thoughts!
I hear you - so best practice is to use a nix flake to setup the environment and at most use it to script build phases, similar to npm (?).
(Let me repeat your advice with my own words, to check if I understood you completely:)
However, unlike a makefile or the likes, nix develop will execute the phases regardless if the sources changed.
So, the theoretical suggestion would be to break the project down, into several derivations, each having their own inputs and outputs, and build phases. Thus, if input A changed output B is created, which is input for another derivate producing output C …
But this approach falls flat in practice, as the granularity matters: Either there are too less derivations, thus one changed file leads to the whole project needed a rebuild, which can take too long. Or, to mitigate that, every single file has its own derivate, leading to too many derivations, and a quite complex setup.
Thus the practical advice: Use nix (besides the env setup) for npm-like scripts, which execute some flavor of make, which is made for incremental and dependency aware builds.
If this was all correct & what you meant, I wonder if the practical granularity isn’t good enough?
In my example: I have the core logic implemented in Rust. cargo build (rust’s compiler) uses rust’s dependency management to build (and cache) all dependencies (3rd party libs). So, I only want to rebuild the rust core project if any of my source code files changes. That would be my rust-core-package.
Its input would be my rust source files (is that possible?). It’s output a rust lib file.
This file would be used in a mobile app, build from flutter source files. Again, their 3rd party lib dependencies are handled by the flutter/dart toolchain.
This flutter project would get a flutter-app-package as well, the input being the rust lib and flutter source files. Output the mobile binary.
So, I imagine having just two nix files. Running “deploy” on the flutter file would trigger a re-compile of the rust project, if their sources changed.
… And I assume there this idea would not work, right? Probably nix would use the existing rust lib from the last build, and not check if the sources changed (“transitive dependency”), right?
Hi,
I discovered an additional option. And wonder if anyone uses this?
To restate, my goal is to make the setup easier by removing redundant tools. It is arguable, if having nix and make isn’t easy enough - or if nix alone is better (nix learning curve vs familiarity with make). Let’s not have that discussion here, but explore what is possible and still practical
I just saw that direnv has a watch_file option. I assume that using direnv with nix flakes for a development environment setup is best practice. If I am right in my above post, that nix would not evaluate transitive dependencies, using direnv could help:
Source files A would be watched by direnv, and any change to any file would trigger a flake, which would build output B. Another direnv configuration (maybe in another subdirectory) would watch output B for changes and trigger another flake, resulting in output C, and so on.
That would practically mimic make and allow for sub-project specific dev env as well.
Any thoughts/corrections on that?
I assume that using direnv with nix flakes for a development environment setup is best practice.
It definitely is a good idea, but mainly (and a bit simplified) to prevent having to type “nix develop” by automatically taking care of this when entering a project folder. Other than that, nix-direnv will also just watch some files and then execute some shell-code (mainly to re-load the environment) if one of them changes.
For me, it’s simple. Make/CMake/Cargo/Whatever build the software assuming that build dependencies are provided from somewhere (along with some capability of getting source-dependencies on their own). Nix/direnv are the tools that provide these dependencies in a reliable and repeatable way so there’s no more long READMEs about what packages you need to install and no more “works on my system”. Clear separation of concerns, works very nicely.
I’m not sure why you want to use the tools that provide the dependencies to do the build itself. The existing build tools work reasonably well for that and a lot of work went into them to solve many common issues. Sure, they have their sharp edges, but I’m pretty sure anything new based on Nix will have the same.
I suggest you just try your ideas with an easy example and see how far you get.
Many thanks for your thoughts.
My goal/idea is to reduce the number of needed tools (build systems) - especially if multiple languages are utilized in a project. But I get a more clear picture (since my last post I implemented direnv, and understand that better as well). And I agree with you: Best build tool for a toolchain … is their own build tool (like cargo) … and if there are multiple languages, make is the time-tested king. In my project I mix Flutter and Rust - and cargo as well as flutter’s pubspec are perfect for dependency management, I would not replace those. Though I looked at getting rid of the just file I use, which is actually more like npm’s scripts then makes targets: Shortcuts to run certain commands.
I will transfer this into a flake
As you suggested, I will play around with a simple setup, just to see how dependencies are managed. I understand now, that source code managed by nix would get copied into an immutable location, so that the build is reproducible. And selecting what needs to be rebuilt when something changes is more of a build management, which expects mutability
I am currently blocked by my flutter setup not working - but once it runs I will continue on this thought
There’s some murmurings about it, at least: GitHub - edolstra/nix-ccache: A flake to remotely build and/or cache C/C++ compilation, using recursive Nix
I think you can use a system-wide ccache on NixOS too.
Not really, the tooling is just lacking for a general build tool to do the latter automatically (though ccache+nix is a step to that). In a sense, make does exactly that, and nix only cannot do the same thing trivially because people are already using various automakes for C & co, and no other languages support granular build instructions.
I’ve always found bazel really interesting, since its goal is exactly to do good multi-language, incredibly granular builds (function-level, so even more granular than make). In theory it can do that without intermediate build tools, too, though in practice there are too many languages/projects reinventing the wheel to actually achieve that.
I find Bazel too cumbersome for any of my personal projects, and working on bazel itself can be painful because working with Java is painful, so I rarely use it these days. But the ideals are definitely right, and large organizations with heavy build requirements seem to be adopting it.
It’d be amazingly cool if the nix ecosystem got some development in that direction, since I think it could realistically bridge the niche bazel struggles to fill due to its lack of a good way to handle bootstrapping. Hard to see that happen without Google’s funding though.