What would you change in Nix or NixOS?

[7c6f434c]

@FRidh

• topologically sorted phases: not being able to set the order of phases blocks progress;
• structured attributes: ad hoc variables used by hooks becomes a mess;
• clean separation between inputs such as phase arguments and what is passed to derivation as attributes: package expression author should not need to know derivations are used;

… and avoidance of clearly-pointless rebuilds for silly implementation reasons, right? I agree that would be nice, I even once tried…

@FRidh

multi-derivation packages: less building time, especially with CA.

Going towards full *-to-NixMake, I hope!

[peterhoeg]

FRidh

• structured attributes: ad hoc variables used by hooks becomes a mess;
• clean separation between inputs such as phase arguments and what is passed to derivation as attributes: package expression author should not need to know derivations are used;
• multi-derivation packages: less building time, especially with CA.

Yes, yes, a thousand times yes on these 3.

[NobbZ]

@Solene

For example, I can’t figure on which commands I can use --substituters .

On all of them that eventually realize a given derivation. But only if you are a trusted user or the mentioned substituter is already in the trusted-substituters list and has a public key available.

Or well, is it actually --option substituters? I never remember which options can be overriden using arbitrary flags and which require usage of --option…

@Solene

The second thing is not being able to upgrade a NixOS system if a package doesn’t build anymore (claws-mail recently, or zim on a regular basis), of course this is normal that the build is failing but we could have a simple mechanism to tell it’s ok to try an older package version or something.

Not (easily) possible in nix. Though indeed it should be easier to know what exactly pulls a broken package.

Even if we manage to build the broken config and use --why-depends we might get stuck with a meaningless information.

I wish there was some tooling that would allow to query the configuration itself for reference of build inputs by store path.

[TLATER]

@tejing

In short, a well-formed flake should always specify a fully reproducible set of outputs, but it should also be thought of as specifying automatic processes to modify itself in various intended ways, such as updating inputs, overriding inputs, and altering arguments.

I keep running into this. Flakes are almost a generic replacement for things like github actions that can easily and reproducibly manage silly project chores like “update my Cargo.lock”.

I’ve been hacking it through putting scripts in apps, but that’s not what those are intended for. Some integration with nix flake update would also be great. Maybe this is where the “type” attribute could actually be used?

[Sandro]

A smarter remote builder protocol that makes a difference between big-parallel builds and normal ones and cpu usage of different remote builders.
Also I want CA to really take off and multi-derivation packages to become a thing.

[NobbZ]

1. I would make URLs in flakerefs work like URLs, with a host where the host belongs, and not in a query perameter (Leverage the "uniform" in URLs for flake refs more · Issue #6304 · NixOS/nix · GitHub)
2. I would make sure that self can properly depend on git submodules (despite my aversion against them)
3. I’d make the flake schema more modular, such that we can provide own “labels” or “checks” for custom outputs, to not get annoyed by “darwinModules/homeManagerModules is not a know attribute” anymore

[nrdxp]

While I was a big fan of it for a long time, the more I use alternatives at work (i.e. std), the more I am starting to dislike the module system. It’s just too complex, it is the main source of confusing error messages that are no help, and I think that the suboptimal merge strategy of the Nix expression language would be solved better by using a different, but similar language; preferably one with static types, and type inference.

[smkuehnhold]

I think I am also slowly coming to the same conclusion as I learn more about the module system.

On a similar note, I wish the module system separated “Types” from “Merging Strategy”.
Coming from an imperative background, the idea that an option defined in multiple areas of a code base could result in value that is a merging of all those definitions was quite surprising and fairly difficult to grasp. I feel it would be beneficial to those coming in from similar backgrounds if this was spelled out more clearly.

[nrdxp]

Well I only mentioned it because we are explicitly talking about a wish list here, but it seems like it would be very difficult to decouple NixOS from the module system at this point, at least without losing valuable information.

Thinking a bit further on this though, it would provide a clean boundary between config and packaging to just use an entirely different language for configuration all together. Maybe Nix doesn’t need to be the end all be all configuration language, maybe we could just use a better language for that (Nickel?) and just focus on using Nix for packages?

[7c6f434c]

@nrdxp

Thinking a bit further on this though, it would provide a clean boundary between config and packaging to just use an entirely different language for configuration all together.

For experienced users the fact that config can seamlessly include a bit which is closer to a package definition (why package list would not contain an override…) is quite a powerful escape hatch, though.

[7c6f434c]

If so many people talk types, I want a completely different thing abot types: reduce the usage of booleans and add a type actually suitable for what we use booleans. Binary yes/no does fit in some cases (we either run the tests or not), but more often our reality is «known-good / unknown / known-bad» or sometimes even multiple gradations of «known», and hammering this into binary yes/no leads to pretty absurd looking discussions… Like all the history of changes around enableParallelBuilding is the story how painful it is to use booleans for handling «reliable-yes / unknown / flaky-no / reproducible-no»

[cdepillabout]

@Solene

If you had a huge team of engineers and a lot of budget. What would you change in Nix or NixOS and why?

I’ve heard on at least one occasion that it would be difficult for a company to adopt Nix/NixOS without having a vendor they can go to for support.

If I had a team of engineers and a lot of budget, I think it would be interesting to create a vendor that focuses on Nix/Nixpkgs/NixOS. First order of business might be to pick a random NixOS release and turn it into an LTS, where it would be supported for 2 years or so (instead of the normal 6-months).

It would be interesting to see if creating a vendor like this would spur more interest in using Nix/NixOS professionally.

[tobiasBora]

Personnally I would try to improve the developper experience. NixOs fits poorly with many “language package managers” (e.g. npm) that tries to download pre-built binaries, and it would be great to automatically patch these files, provide a loader in NixOs or provide a way to enter a kind of FHS without creating a VM (VM or even steam-run are poorly integrated with the host system). Just trying to program with electron is already a challenge! EDIT I discovered meanwhile nix-ld whose goal is exactly to provide a system-wide loader… Really cool, this should be more documented ! That said this won’t help to package stuff in nixpkgs as this only works in NixOs.

Another improvement would be regarding caching when creating/debugging a derivation: I spent so many time recompiling a big package 10 times because of a few typos in the install phase… So adding a simple cache between the build phase and install phase would be really cool. I’m aware of nix-shells but in any case I’ll need to compile at least twice to test the final derivation, which is a waste of time. Ideally, nix-build could even provide a debug option where when we build twice, it tries to use to files that have already been compiled in the previous try, providing basically the benefits of nix-shells directly into nix-build, which would save the need for manually calling the various phases in a nix-shell.

[Ericson2314]

@7c6f434c

If I had a huge team of engineers… I would immediately sink in coordination issues, as it always happens.

This is why the most important thing we do is not any new features, but emphasizing layering.

The more modular the Nix ecosystem is, but more isolated groups of people will be able focus down on individual areas of improvement in isolation, and yet we can still try out their work through mixing in matching.

[7c6f434c]

@Ericson2314

This is why the most important thing we do is not any new features, but emphasizing layering.

Well, yes, my wishlist has a ton of «more separated and hot-pluggable .*»

Unfortunately «pull whatever as a nix subcommand without a clear way to switch it out for something similar» is the way development will actually go…

[rehno]

A consistent, uniform and robust lang2nix approach for all major package sets. Fixes/RFCs upstream to various tools to make things work the way they ought to work if necessary.

Every build system would have a single lang2nix that is good enough for both nixpkgs and development environments.

[turion]

From useful/important to least important:

• Useful type checker. When one forgets/incorrectly names/… attributes, the error messages are often completely useless. The right kind of type system could help a lot. But this is a nontrivial task.
• Fully automate minor version updates to existing packages. Such PRs should just be automatically merged if all ofborg tests are green. Reserve human attention to more important tasks and never “burn through” contributors.
• Secret handling. Like passwords, deployment keys, database passwords, and so on. Each module/service ships its own fragile, noncomposable “solution”.
• Clean nixpkgs pipelines. Merge trains, a core package set that has to have all tests succeeding before a merge.
• Better security story. CVEs have to be handled quickly and transparently, by a dedicated, responsive team. And before they are publicly known. This is often a blocker for commercial adoption.
• Make tests into separate derivations. When packages fail because of tests, disabling them or fixing them shouldn’t need to rebuild everything, but only the test. Similarly, disabling tests should not rebuild everything.
• Easier entry/migration for existing projects to NixOS. Especially at work we have the situation that someone writes code, infrastructure, CI, etc. without nix, and then at some point people start wishing for reproduceability, fast (cached) pipelines, deterministic builds. But if you’ve set up all the machines without nix already, it’s hard to use nix without switching to NixOS. I don’t know a good solution. Maybe bundling? Ansible integration?
• Stabilise the nix development. Don’t bring out features so quickly. Test and review them better with a beta user crowd to receive community feedback before breaking stuff for many people.
• Better meta-programming story, or rather better import-from-derivation story, mainly to update lock files. In particular on Hydra. There are reasons it is not enabled there, but it’s annoying. A lot of “organically evolved” update scripts are lying around because of this. Alternatively, use nix flakes in nixpkgs (and deprecate non-flakes).
• Modularise toplevel/all-packages.nix and avoid huge files in the future. (Maybe let's sort and/or categorize all-packages?)

[Sandro]

@turion

Fully automate minor version updates to existing packages . Such PRs should just be automatically merged if all ofborg tests are green. Reserve human attention to more important tasks and never “burn through” contributors.

That only works for projects that follow semver very strict and there are enough which don’t or don’t follow it all the time. Also this would not remove upstreamed patches, update compile flags, dependencies or meta data, summarized we would by default not react to any upstream changes if the package still compiles and tests pass which would make bit rotting an even bigger problem.

@turion

Clean nixpkgs pipelines . Merge trains, a core package set that has to have all tests succeeding before a merge.

merge-trains do not really solve the problems we are having. A good amount of the PRs to nixpkgs are fully independent of each other and the PRs to staging which would benefit by this are not viable right now because of rebuild counts and the required compute power. We could run the checks required for the unstable channel to advance but most packages would not benefit from that because they do not influence those tests. This could fully change with CA and after some optimization for it which is one of the reasons why I am looking very much forward to it.

@turion

Modularise toplevel/all-packages.nix and avoid huge files in the future. (Maybe let’s sort and/or categorize all-packages?)

A good idea that came up in the nixpkgs architecture matrix chat is to just sort packages by their first letter into subdirectories which would make automated sorting easy.

[turion]

@Sandro

@turion

Fully automate minor version updates to existing packages . Such PRs should just be automatically merged if all ofborg tests are green. Reserve human attention to more important tasks and never “burn through” contributors.

That only works for projects that follow semver very strict and there are enough which don’t or don’t follow it all the time. Also this would not remove upstreamed patches, update compile flags, dependencies or meta data, summarized we would by default not react to any upstream changes if the package still compiles and tests pass which would make bit rotting an even bigger problem.

One could opt-in for packages that follow semver. Minor version changes simply should not trigger the problems you are mentioning.

Anyways, I’m noticing that a lot of maintenance work is “reviewing” and merging minor version updates. This scares away people. Newcomer PR reviewers are looking through the massive number of PRs, say “these are all just boring version bumps, what could I possibly sensibly review here”, and look for another OS project where they can spend more meaningful time. (I’m claiming this after having conversations with NixOS newcomers.)

This is a problem and I’ve suggested a change.

[Sandro]

Then we need to improve the documentation for newcomers that the amount of open PRs are not that scary and they get the knowledge to navigate them.

One way is to filter out r-ryantm PRs Pull requests · NixOS/nixpkgs · GitHub

and the open PRs are a lot but currently we are merging ~900 a week which is also a lot. nixpkgs scale is just big.