I’d like to share my major pain issues that makes NixOS hard to “sell” for a company as a viable OS.
When talking about NixOS, it’s almost perfect, you control version deployed, you can rollback, you don’t need third party tools like ansible/salt to handle your systems, you can easily build a newer version to try and replicate environments almost perfectly.
BUT
Very short life cycle: most companies just can’t handle upgrading every 6 months and regularly keep track of the unstable version to check for breaking changes. This is why most companies run Red Hat or Ubuntu LTS with 5 / 10 years of support and slow changes
When a program isn’t packaged, you often have to dive deep into Nix language and nixpkgs framework, while using NixOS itself doesn’t require much Nix knowledge and didn’t prepare you for that
Using proprietary software can be a real pain, or impossible, nix-ld can help sometimes but it’s still not straightforward
I don’t have solutions for this, and not really looking for them, but I’d be curious to read about other and why they can’t use NixOS on company servers.
I used to work for a smallish consultancy; we had some internal infrastructure that was managed by a rather excellent ops team, lead by a friend.
His response was “it’s too different, I don’t have the time to learn this”. Which is fair, the company is filled with Debian and GNOME maintainers. They still primarily use ansible to my knowledge, though one internal project has a flake.nix in it nowadays (that has probably become mostly unused after I and two of the other people I got enthused about nix left).
When talking to customers they typically already know roughly what they want, and while we could give them useful advice on whatever they were planning (and then implement it), it’d usually be a call for help adopting some other technology.
Offering something relatively obscure with a significantly smaller community they’d never heard of as an alternative just… Well, if I suggested it internally it was primarily as a joke. Ironically, the only time nix came up in a discussion was because they’d mentioned a negative experience with it.
Then again, I was also working for a consultancy that doesn’t have much of a background with nix, so I probably shouldn’t be surprised nobody ever asked for it.
Trying to get us to be more involved was mostly met with indifference, as the company is more interested in bazel.
All this is to say… Maybe NixOS/nix just needs more, better marketing so it doesn’t sound so exotic?
I personally wouldn’t have that much of a problem with running a non-LTS system.
When I look at all the changes we had to make to jump from 18.04 to 22.04 it probably would’ve been easier to follow the releases.
However with those systems there can be something left behind which can cause problems.
We probably wouldn’t have that problem with NixOS.
But 1 month transition time is very little time to do testing and fixing possible problems.
Ubuntu drops a new LTS every two years and semi-annual updates to the hardware-enablement stack. And LTS support is limited to a (comparatively) small set of core packages.
In theory, we could make a LTS-release based on NixOS small (using LTS kernel), but this would put additional load on the maintainers. If we tackle this, we would also have to invest a lot of up-front work in maintenance policies. (e.g. list of supported packages, single Python version,…)
Writing a proper Debian package isn’t fun at all and creating a package for Nixpkgs seems even simpler to me. Plus: Being nice and maybe handing out a contract to one of the Nix-companies helps a lot.
People face similar issues when working with Yocto, so I’s say this issues isn’t unique for NixOS.
When proprietary software request a special distro, this can be honoured using distrobox.
We probably need better documentation on that side or even a proper wrapper/helper.
I have pretty similar reasons for being extremely hesitant in pushing NixOS. In my last company I was so overhyped with NixOS that a coworker once jokingly gifted me a shirt saying something like “With NixOS… …that would’nt have happened” (but in swiss german).
My main blockers are currently:
No proper LTS release and way too short update time. I think this also requires a “mindshift”. Old Legacy python2 software is NOT easier to maintain than an almost up to date system. But I doubt people will learn that
A “proper” company backing with good (and obvious) open source presence. Working more and more with customers using RHEL and SLES, I can’t imagine them ever using a distribution without full company support and guarantees. IIRC there used to be Mayflower OS which also gave you security patches for selected packages. I think it was a nice try, a bit sad that this apparently didn’t last. Not sure what other offers are in that regard though.
A company backing would also include some opinionated definition on “how to use NixOS” - with a proper and up to date documentation. There is so much tooling around NixOS and it’s growing by the day. I have no clue which tool I would use to manage a customer landscape of maybe 20-150 servers. And the approach I’ll go with will most certainly be completely unique, even though many things shouldn’t have to be unique (the downside of having a turing complete language with nix). With an opinionated definition on “how to use NixOS”, a wiki which is not just community based (or an in between solution like e.g. “owned pages”, where a company/person makes some guarantees about the state of selected documentations), I would feel much better telling my colleagues about NixOS.
Flakes are still experimental. I’ve told myself somewhere around 2 years ago that I want to make a company internal NixOS presentation once Flakes are stable and I don’t have the confusion of stuff like --experimental-features 'nix-command flakes'. The flakes ecosystem seems (to me) far easier to explain to people than the “legacy” tooling. I know it’s a minor thing, but not even knowing when flakes will become stable is really annoying. And if we’re talking about enterprise, we can’t recommend experimental features.
CVE Handling seems not very nice in NixOS. The vulnerability roundups have stopped and many of the generated issues are still open. I’ve triaged some of these CVEs a few years ago thinking that maybe I could at some point get a job doing that if some companies are interested in getting these things handled, but life happened (and apparently nobody cares enough, otherwise there would still be something like the vulnerability roundups).
Have to second the holes in the CVE management / software supply chain management. If your company is serious about security (or at least trying to be) that’s going to be a big issue with using nix.
Another issue that nobody has brought up yet is combining nix and non-nix packages: it won’t come up in the initial “sell” of NixOS, but:
non-NixOS binaries on NixOS: after engineers start using it they’ll notice that the prebuilt binaries in Maven jars or pypi wheels don’t work because they assume you are running a debian / redhat like distro
nix binaries on non-NixOS: after engineers start building containers they’ll notice that if they want to put their nix binaries in a layer on top of a non-NixOS base image it’s going to blow up the image size, and they’ll hit issues loading native code built in Nix in a non-Nix JVM / python interpreter
I think pushing NixOS is quite tough because generally people don’t intuitively understand how much the choice of distro / OS impacts everything, so once they see how much using NixOS impacts every aspect of development they might feel like it’s too big of a risk.
I question whether we should be “pushing NixOS to companies” to begin with. Of course it’s understandable that in a daily sysadmin capacity, people would prefer working with NixOS - but there are also serious risks associated with this.
Corporate environments often have operational expectations/requirements that are at odds with best practices from a community perspective; for example, when people talk about “security” in a corporate setting, it’s usually actually about compliance, which frequently requires compromising on actual security to accomplish.
Trying to actively push NixOS to companies and accommodate those sorts of requirements runs the risk of corporate needs deciding the direction of the project, at the expense of everybody else, as already typically happens in FOSS projects that see corporate adoption. I would rather not see that, especially given the current highly collaborative nature of the community that I’d like to keep.
So personally I feel like it would be better to focus on what issues people run into with NixOS personally (there are more than enough of those!), and address those instead; this should be sufficient to make it adoptable in commercial environments that don’t place strange ‘enterprise burdens’ on FOSS communities, while reducing the risk of ‘demanding corporations’ taking over project direction.
We don’t have needs like this in our company.
We could easily run Debian instead of Ubuntu for example since we don’t have an active support contract with Canonical.
The only compromise in terms of security we had to make was when we configured the SSL on the proxy so that customers with old devices could still connect to our services.
However I’m talking of a company with 20 employees and not hundreds.
I see a lot of benefit in using Nix/NixOS especially because we’re so small, I feel the overhead to manage my servers at home got much lower since I switched from Ubuntu to NixOS and from Ansible to nixos-rebuild.
When I presented Nix(OS) to our Ops-Team and some picked staff members across coding teams, ops was really interested, though also said, that we do not have much of a choice of the platform.
Service and Compliance agreements with our (international) customers, which are much larger than us (in Germany alone the biggest customer has 800 times as many employees than we).
Due to the fact that Ops are just 3 and are doing customer hostings, internal infra, and also handing out and maintaining the employees equipements, they do not really want to have yet another completely different beast to maintain.
So all hostings are using an LTS version of Ubuntu, as thats the distro allowed across all customers. For employees devices we can choose between Mac or Windows.
We are free to use Nix on Mac or WSL, but we can not use NixOS.
And what I already realize, the Ubuntu LTS schedule already causes problems, as 5 years seems to be “to fast paced”. Due to the slow acting customers everything has to be planned ahead massively. And despite having a year in planning the move, small problems can cascade to seviere outages.
So even if the customer would allow for using NixOS, having to do hard switches like that every 6 months on all hostings is considered dangerous and burdened. With Ubuntu we can balance the hostings to be one of the still alive LTS’.
I’ve been pushing Nix-based development environments into my company. We are a large semiconductor manufacturer and often, creating firmware for a single SoC requires multiple compilers for different ARM cores, DSP cores, probably RiscV soon. Not all of these cores are under our control, so we may need to support config blob generators for other firmwares, all kinds of internal analysis tooling written in a plethora of languages from Perl to Java. Some of the most common issues I’ve come across:
People like using Nix, but it’s very hard to find people who both want to spend the time and have time to spend to learn Nix to a useful capacity. Be prepared to do this alone for quite a while. Also, you will now be the guy where everyone goes if there is issues with the dev env (though, if you are that guy already, it will probably still save you a ton of time).
Initially, creating packages for some proprietary software is quite a pain. But once it’s done, unless there’s major changes, it’s usually a matter of minutes to package updates.
It sometimes works so well, that people forget why you put in the effort in the first place. I can’t remember when we last had an issue to the likes of “works on my machine”.
Get the CI team on board early.
You will need infrastructure at some point in time. A Hydra here, a binary cache there (because a remote site has a slow connection), a reasonably stable place to host your build sources, these things tend to add up.
Cross-compiling for Windows is still pretty bad. I still don’t know how to get mingw-w64 with pthreads to run. Win32 thread model just does not cut it many times. Unfortunately, this is often required. I only realized that the Linux Dev Bubble is actually pretty small once I joined this company.
Probably lots more. Overall, I’m quite happy that I started with these dev envs. I think it saved the team lots of time, we could get new employees ready to build firmware for 5 different cores in 30 minutes (most of it waiting for 20GB of toolchains to download), prevented many issues with diverging environments.
I also started doing a NixOS experiment for our work machines (usually containers that you connect to using some remote desktop software). This is still heavily in Beta state. I got quite a lot of stuff working now, including some proprietary remote desktop solution. But really, this is something I take more as an exercise right now with it being quite a bit away from actual usage.
This is one of the main points that I’m thinking about a lot. I’m still learning Nix myself but I see what we could gain by using it.
Vagrant, Ansible, etc. require a ton of work as well.
However, when you’re already the only DevOps person it’s quite a difficult decision to make.
When you’re stuck with one of the conventional tools it’s quite accepted to have a bug, if I would push something like Nix I reckon it would be expected that I should be able to solve it.
It is too “enterprisy” to me. Indeed I live on the unstable channel, basically.
Supporting a whole ecosystem for five years require a huge cache and at least a dedicated team cherry-picking security changes.
Seriously, it applies to other distros too.
When something is not packaged, you are on your own.
When I go to Repology seeking for installation scripts, there are few package databases providing them. The Debian- or Fedora-based ones didn’t provide afaicr, but AUR and pkgsrc have plenty of them.
On other distros though, you can use some prebuilt GH artifacts though, try this on NixOS…
And to be honest, doing quick’n’dirty packages using fpm based on this prebuilts is pretty straight forward, and does not requiry nearly as much work as nix does.
that’s another story when you need to push this for a company. Yes, it’s fine to use unstable as a single person, but as a team with high availability requirement, unstable isn’t ideal, and pushing Nix is hard.
I do want to shout out the Security team here . They do a lot of work and it often seems to go unappreciated.
I suspect the CVE digests just took too much time and were too boring to do. Fixing actual security issues is probably a good deal more engaging. Both NixOS Security Triage (#security:nixos.org to track work on CVEs) and NixOS Security Discussion (#security-discuss:nixos.org for more back and forth discussion of more fundamental issues) are active, and mostly driven by a small group.
If companies want/need a more easily digestable format than keeping tabs on those rooms I don’t think it’s unreasonable that they should sponsor such work, either financially or with manpower. Too many still expect a free lunch.
. They do a lot of work and it often seems to go unappreciated.