What is the recommended way, requiring the least effort, of making binaries work on NixOS?
When I get a binary from an open source project, which, for example, is pretty difficult to compile on a generic operating system, how can I make this binary work the “right way” on NixOS, while not being forced to invest a lot of time into each binary?
I have found it difficult to make any third party application work, if there is no NixOS option already implemented for that.
What is the recommended way, requiring the least effort, of making binaries work on NixOS?
You’re not going to like this, but the recommended way is unfortunately not the one that requires the least amount of effort.
So if you are looking at effort only:
But in terms of the recommended approach, package it properly and get it upstreamed into nixpkgs.
NixOS is probably one of the easiest distributions to package something for although admittedly, it takes a bit of effort to cross the initial hurdle.
One great source is Arch Linux (both Arch proper and AUR) where you can see their build recipes.
When I get a binary which is pretty difficult to compile on a generic operating system
Do you have examples?
while not being forced to invest a lot of time into each binary?
It’s possible to package simple applications in 5 minutes. Yes, some take much longer of course.
I have found it difficult to make any third party application work, if there is no NixOS option already implemented for that.
You mention NixOS - do you mean applications that expose configuration via programs.foo.enable = true; and friends or applications simply packaged in nixpkgs but without any nixos configuration support?
I’m not familiar with submitting anything & am not sure how approachable it is. Does it take a lot of effort on the part of whoever is submitting it?
If submissions are fairly easy (low effort), then this recommended way isn’t bad, at all (except the time it needs for the package to be eventually available).
However, if submissions are difficult & discouraging to accomplish, then that’d be a problem.
I’m actually currently in the process of comparing Arch to NixOS, mostly in terms of usability & compatibility. Have started using both recently, once again. Of course, I count in all the NixOS declarativeness into the equation.
Well, I set up a Kubernetes cluster on NixOS, to get into NixOS in an area where NixOS shines: reproducibility. Having a reproducible server declaration would be really awesome.
When I looked up the NixOS options, I have found, that it’s naturally nice to configure all the stuff through the Nix configuration. However, as always, the documentation was very scarce and I had to read the modules’ source code a lot, to get a grasp of the Kubernetes specific Nix options.
For example, I wasn’t able to find the definition of packages in the context of Kubernetes’ CNI options.
There are other options like this in the Kubernetes related modules.
Another problem was, that the most obvious CNI choice, i.e. flannel, has lots of options.
However, what if I use a different CNI?
Well, that was troublesome to wrap my head around. It took me a fairly high amount of hours for something, that would’ve otherwise taken me 15 minutes on a generic system.
I manually installed an unsupported CNI, by providing it with custom CLI options, which changed paths to binaries, etc.
However, even after this, there is still something not working well. One of the places which always get re-generated by NixOS on switching/re-building configuration obviously does not save a manually placed binary there and therefore I had to cp it there, as a workaround.
I don’t know of any way, which would implement the placement of the binary in the right place, without making a huge Nix module out of it, by hand.
Now, even if I would submit a request to support this CNI, would someone write a module for it or just make it work as a bare minimum?
And I’m lucky for the binary to work out of the box on NixOS, anyway.
So, this is the story behind the question about making something work on NixOS.
I doubt it would be so simple for this CNI example. How would I go about that?
https://search.nixos.org/options
I’m not familiar with the official terminology. I mean “NixOS option”, as in, any option, that already wraps an existing program into a module. Basically, when I use services.kubernetes.kubelet, this is a prepared NixOS option, which already wraps a major part of the Kubernetes Kubelet’s functionality in Nix.
Basically, someone took the time to create a Nix module for a third party app.
Still, all that said, the biggest issue with NixOS is and always has been the lack of documentation. I always have to guess, what anything means. Like for example, just look at the Kubernetes, Kubelet and related modules. At least 60% of the options there are unclear about their functionality.
Things get worse, when options are deprecated and I don’t even know why.
For example, services.kubernetes.kubelet.networkPlugin was just deprecated like two days ago. Why?
That’s a nice one. I wish the information about this would be disaplyed more prominently.
If submissions are fairly easy (low effort), then this recommended way isn’t bad, at all
There is absolutely a learning curve and you will in most cases be getting some kind of feedback that will require some changes, but I consider that a good thing. I have personally submitted changes that were trivial that I would consider “let’s just merge and get on with life” and yet somebody who took the time to review it came back with some constructive feedback causing a “huh, I didn’t think of that - good point!”
(except the time it needs for the package to be eventually available).
There are ways around that. You can simply fork either the latest stable or unstable and apply your own changes which allows you to start running it straight away while the upstreaming process is ongoing.
I’m actually currently in the process of comparing Arch to NixOS, mostly in terms of usability & compatibility. Have started using both recently, once again. Of course, I count in all the NixOS declarativeness into the equation.
I used to use Arch but each machine requires handholding which is painful and (in our case), we would have a different OS per machine type (arch for workstations, Debian for servers). >
Well, I set up a Kubernetes cluster on NixOS, to get into NixOS in an area where NixOS shines: reproducibility. Having a reproducible server declaration would be really awesome.
It has been quite some time since I looked at K8s on NixOS and I might be doing a disservice to the people who have put effort into running it on NixOS, but at that time, it was shaky at best. K8s moves fast and with the resources we have available, we (the community) are not able to keep up.
I would not recommend running a full blown K8s cluster on NixOS.
However, what if I use a different CNI?
Then you would need to add proper support by packaging the CNI properly and writing/changing a NixOS module to support it and then continue fixing it when something changes - it’s a lot of work.
I don’t know of any way, which would implement the placement of the binary in the right place, without making a huge Nix module out of it, by hand.
There is nothing preventing you from using NixOS to copy in an arbitrary binary and putting it somewhere you need it that being said as long as you can reference it your configuration, the location really shouldn’t matter.
Now, even if I would submit a request to support this CNI, would someone write a module for it or just make it work as a bare minimum?
Would someone randomly pop up and write it for you? No, very likely. If somebody else has a need for it, you might get help, but by default you should assume that you will need to write it.
I doubt it would be so simple for this CNI example. How would I go about that?
K8s with custom network providers is in any case not a good place to begin your journey. Start simple.
Basically, someone took the time to create a Nix module for a third party app.
In your case, you would need module support. Many other applications do not.
That’s good for every Pull Request.
That’s one really nice. Didn’t think of that.
I managed to get it running somehow and once it is set up, “it works”, but it requires a lot of effort in advance, before anything is running, at all.
I guess, this is how NixOS is supposed to work, in general, though. Make a huge configuration up-front, then fix the bugs and then you are done for (mostly) good. Right?
Indeed, that’s my problem. It’s already a lot of work to deal with advanced Kubernetes setups in the first place. It would complicate things, if it would be required to make a officially approved Nix module for every yet unsupported component. 
Yes, though this sounds very dirty, like an anti-pattern. Or is it expected on NixOS?
That again cements the time & effort problem.
I’m familiar with Kubernetes & am used to it. So, I thought I’m going to set it up on NixOS, having to deal with NixOS mostly, since the Kubernetes area is nothing new to me.
I guess, there is no templating or generation tool for this, yet? All this would be a manual job, I suppose.
The biggest issue with all this is the time & effort that NixOS compatibility through Nix modules apparently requires.
One needs to create entire modules for something to work, in some cases, like e.g. this one.
Perhaps, I would create a module for that particular CNI at some point, though am not sure when I get to that, in terms of time.
Either way @peterhoeg, you have been a great help. I appreciate you clearing up some thoughts & questions I had about NixOS & its tools.
I am slowly getting into how NixOS works & am curious how things will continue from now on.
I guess, this is how NixOS is supposed to work, in general, though. Make a huge configuration up-front, then fix the bugs and then you are done for (mostly) good. Right?
Complex systems such as k8s that require a lot of configuration elsewhere, also require a lot of configuration on NixOS.
It would complicate things, if it would be required to make a officially approved Nix module for every yet unsupported component.
You will have to have to modify the module system if you are adding a different CNI, but it doesn’t have to be officially approved. Nobody is preventing you from just applying the patches to your own fork.
Alternatively, can you use one of the existing CNIs for now? I am assuming that this k8s cluster of yours isn’t running large production loads, so maybe for now, just stick to one of the supported options while you get your head around NixOS?
Yes, though this sounds very dirty, like an anti-pattern. Or is it expected on NixOS?
When you run nginx (as an example), it will run /nix/store/somepath-to-nginx/bin/nginx -c /nix/store/some-path-to-the-config-nginx.conf.
In the configuration for nginx, you are referring to pkgs.nginx and some builder that writes out the config file.
It will be exactly the same in your case. You will reference some builder that “generates” your binary in your configuration which is no more or less dirty than the nginx case.
That again cements the time & effort problem.
Sure, if it isn’t supported, someone will have to add support.
I’m familiar with Kubernetes & am used to it. So, I thought I’m going to set it up on NixOS, having to deal with NixOS mostly, since the Kubernetes area is nothing new to me.
I meant in terms of NixOS. Either start using k8s with the NixOS supported configuration or try using NixOS for something !k8s while you wrap your head around it.
I guess, there is no templating or generation tool for this, yet? All this would be a manual job, I suppose.
You’ll likely copy an existing module, but there is no automated way, no.
The biggest issue with all this is the time & effort that NixOS compatibility through Nix modules apparently requires.
Again, it depends on what you are trying to do. k8s is complex, its module(s) is/are complex. On the flip side, you can do a quick module to wrap simple applications in no time at all.
I am slowly getting into how NixOS works & am curious how things will continue from now on.
We use NixOS for generating appliances that run on customer sites, which we can still push updates to from the outside. Before that, it was Debian and ansible. NixOS is miles ahead.
In this specific case, the culprit was solely NixOS, in terms of what I counted as “taking up too much time”.
A major part of that was the installation command of the CNI. On a normal system, this job would’ve been executed & done. On NixOS, I had to find a way to pseudo-install it and then manually fixing the installation, so it can work with the custom directory generation NixOS offers.
As mentioned earlier, this wouldn’t be such an issue, if NixOS had official documentation proportional to how much NixOS offers in features, quirks & differences. There is some documentation available, even a “manual”, but compared to how different & special NixOS is, the available manuals & documentation is a tiny fraction, at most 1% of the amount of documentation that should be there, if one is aiming for a pretty complete documentation, covering all reasonable use cases. A great example is the Arch Linux documentation. Not even that one is even remotely close complete, but it’s the best I have seen regarding any operating system and this is by far not the most popular one. Before it come Debian, Ubuntu & its derivatives.
The last time I set up something advanced on Arch Linux, it just took me an hour of reading Arch Linux documentation & it just worked after that. On NixOS, I cannot dig too much, because there is nothing to dig into. The manual, which seems pretty exhaustive, when looking at it from a 100 miles away, gets suddenly very terse, imprecise & insufficient once one wants to get specific & detailed information about a specific component, like e.g. the Firewall.
The only way one can guess how the entirety of the Firewall works specifically on NixOS is by searching the NixOS options in the online search tool & the source code. That’s it. There is no Firewall manual, which goes beyond opening a couple of ports.
Well, my point is, the NixOS project isn’t so recent anymore, that it can afford to have such a minuscule amount of official documentation available & without documentation all the NixOS specialties are just hard to swallow, if one has to investigate lots of source code, trying to guess what one is supposed to do.
I’ll try to make it work like this.
Could you elaborate on that? I’m not sure, whether I have the same definition of “push updates” in my mind.
In this specific case, the culprit was solely NixOS, in terms of what I counted as “taking up too much time”.
Sure, no-one is disputing that using a different component from what is supported by NixOS especially for someone who is new to NixOS is going to be a lot of work and thus time. But what is more important - getting your cluster running or using a particular unsupported CNI?
The last time I set up something advanced on Arch Linux, it just took me an hour of reading Arch Linux documentation & it just worked after that.
Arch Linux isn’t that different from Fedora or Ubuntu or Debian. NixOS is very different and although I agree our documentation is sub-par, doing something “simple” like swapping in a different CNI for k8s, isn’t actually that simple on NixOS because it is so different and there is a fairly large of body of knowledge that one needs to build up before being able to address it.
There is no Firewall manual, which goes beyond opening a couple of ports.
It’s not the ideal answer, but I will recommend to read the code. nix (the language) is straight-forward despite its quirks and shortcomings.
I’m not sure, whether I have the same definition of “push updates” in my mind.
We build the initial appliances using plain ol’ nix. Any changes are pushed out using colmena (used to be nixops) when needed.