Who uses NixOS? Who are you people? (And good-bye)

How can this be fixed, and what are you putting into derivations. Some software doesn’t really like to be put into derivations, because their ecosystem and language package managers, are badly broken, no matter what operating system you run it on.

How does one get highly unreproducable software into a reproducible build system?

Can you name some in implicit assumptions and divergent partial solutions.

10 years!!!

I guess

'you are the people?, (and hello). ’

May the (4th) nix be with you, always.

2 Likes

Rust:

  • buildRustCrate
  • buildRustPackage
  • cargo2nix
  • crate2nix
  • crane

There is always some combination of Rust functions that is missing, whether it is workspaces (i.e., how I work when I’m developing both a library and a GUI app at the same time, but in separate crates), or cross-compile targets, or partial build caching. And maybe, somewhere, there is the right combination of setting up a rust package and injecting it into [buildRustCrate|buildRustPackage|crane|whatever] so that I can mix and match the things that I need, but damned if I can find all the disparate parts.

Then with JS, there’s node2Nix and buildNodePackage and buildNPMDependencies.

And then, if I want to include a package in the Nixpkgs repository, there’s no way for me to know what the build standards are. I’ve had packages rejected for not using buildRustPackage. So I had to either have one build environment for Nix and a different, more practical one for myself, or just give up and accept unacceptable build times for my work.

There’s no movement forward happening on this, and though I’ve offered to help, I won’t do it in a vacuum because I’ll come up with a solution that won’t get accepted.

5 Likes

Thanks for the examples.

nixpkgs will rejected Import from derivation, which is not allowed.

However, it quite possible that certainly ‘2nix’ tools are not adopted, what i’ve seen of the npm ones they generate quite large files in the nixpkgs repo… so it now always a code problem, it’s a size problem!!!

Maybe flakes could solve a problem there… flakes are quite free form to do what ever they want, they really restore the ‘policy free’ way of package management.

I’m not familiar with the rust ecosystem, but maybe someone who is can help you.

1 Like

Apologies for the fragmentation.

There’s some efforts on the Node.js side driven to have buildNodePackage as one single point of entry for everything. The problem with the other approaches is that they worked at a time and everytime Node.js breaks everything or some downstream packages does it. So it’s frustrating and tiring.

There’s no movement forward happening on this, and though I’ve offered to help, I won’t do it in a vacuum because I’ll come up with a solution that won’t get accepted.

Can you tell me what did you try? I can take a look and see how to mend the things.

I can see definitely movements on all points you brought but unfortunately sometimes it takes time, communication is lost, etc, etc. Apologies for this.

4 Likes

I’ve tried all of the tools I listed. Each one of them behaves differently and has different limitations in terms of dependency caching, selecting the toolchain, selecting the target architecture, or producing outputs that could then be consumed by a different toolchain. Additionally, none of them have even remotely similar APIs. Going in any deeper feels too much like I’m ranting.

Instead of ranting, this is what I want:

  • a well documented API which build tools should generally implement.
    • all the build tools to more or less accept the same set of parameters: this is the source, this is the compiler to use, this is the list of dependencies. Here is how you override inputs to a particular dependency, in case it doesn’t play nice with the default environment.
    • more or less the same results of the derivation, especially in a form that can be added to the next build tool: here is a list of libraries, here are the include files, here are the executables, maybe also here is an environment that you can enter that has all of the build tools
    • a clear way to specify that the target architecture of a build is not the same as the host architecture of the build (eg: host is x86, target is arm64).
  • caching of intermediate dependencies, even those built in the same language as the final end result. I cannot emphasize how important this in a large project. It’s easy in Rust-land to end up with hundreds of dependencies, and some of those (like GTK) are themselves multi-minute builds. It’s even important to cache each dependency separately, because a large project can easily have dozens or more different [language-specific term for a Rust crate] involved, and they may be developed in parallel.

For the most part, doing this means discarding Cargo/Pip/Bundler/NPM and building nix-friendly tools for making it all work in properly hermetic ways. Plus, it probably also means building a standard for toolchains like oxalica, so that the things that the toolchain generates are going to easily plug in to the derivations that we want to build the toolchains for.

So, this is a huge amount of effort and certainly not something I could accomplish on my own (much less making it usable by anyone other than myself).

2 Likes

Have you stumbled upon dream2nix yet?

At least seems to be the goal of that project, even if it is IMO quite far from that yet. It seems to support the rust things you’re asking through crane already at least, while providing the higher level API standardization you’re looking for.

1 Like

I’ve tried dream2nix for js a few weeks ago and:

  • it was super slow on bad internet. I would expect it to work as well as any other node package manager, but that wasn’t the case
  • Had weird issues when cancelling commands mid execution (would sometimes seem like I’d get stuck in an invalid state)
  • The devshells never worked
  • failed builds would require me to run a different command to see all the logs. I get why it works that way, but it felt super unproductive, especially combined with my previous point. I couldn’t find a way to automatically show the logs. Them generating in a new file each time meant I’d have to copy paste the path after each attempt. There might be a fix for this, but if that’s the case, it should be more easy to find.

I have tried with both the old and new system they have for js, and neither worked perfectly. I don’t remember specifics, but I could go search through my discord history to see what exactly went wrong. I ended up giving up in the end :frowning: I was quite disappointed, as dream2nix looked like a very neat tool.

1 Like

So I get that this is the nix bashing thread, but c’mon. Those strike me as minor polish issues - which should really be expected given how young the project is - sprinkled with misunderstandings about nix and flakes.

There are much bigger problems with the concept of integrating with npm, api concerns, etc.

Not meant to be taken personally, but slight rant incoming…


I can imagine a bit of processing overhead, even if download speed reduction seems odd.

Of course it would be cool to see some actual comparable performance comparisons, and do some deep dive benchmarking. But this project is very young, the API hasn’t even settled yet, and your first complaint is that it’s slower than the specifically optimized original? For cached inputs you should only download once no less?

And even then, maybe you’re conflating the speed of downloading nodejs and all its deps on top of the packages? Maybe even using unstable for a flake input on an otherwise stable/outdated system so you’re downloading all of glibc and up as well? Or maybe you’re unwittingly comparing to a well-primed npm cache?

Yes, we can say all three of the above are UX problems. But they’re ultimately just unfair comparisons caused by misunderstandings.

You’re sure this isn’t some slow terminal getting stuck with escape sequences? Maybe something like an untweaked emacs, vi or vscode shell that doesn’t know how to handle ansi escape sequences, because somehow that’s still hard in 2023?

It seems like odd criticism of dream2nix, though, given it doesn’t control the output. Hell, I don’t think it’s even fair criticism of nix, shells and terminal emulators should do a better job at not breaking themselves. IME this is often caused by broken PS1s anyway.

Yes that’s slightly unusual, but it’s a nix feature, not a dream2nix decision. I think the reasoning for the feature was that it actively speeds up builds, because most terminals can’t keep up with truly large builds, and there’s nothing worse than making your CPUs spin idle because you’re printing enough text to keep an avid reader occupied for 10 lifetimes.

Stick -L after the command for verbose logging. It’s shown in the help output and the manual under “logging-related options”, though I appreciate nobody ever rtfm.

Maybe there could be a feature request to give this hint on the output as well where it currently instructs you to use nix log?

At least I got some constructive idea out of this rant.


Personally, my main problem with using it for npm was that the build script used for it did some utterly insane reshuffling of directories until recently, which completely broke some build tools. It was something like building in $out and a nested node_modules dir. But that was fixed.

Other problems I’ve run into is the usual unreproducibility of npm projects that like downloading random precompiled binaries as if that isn’t a huge problem.

Running something like the npm elm repos via dream2nix just isn’t going to work ootb either.

dream2nix has pretty good handling of overrides to fix problems like these though, which I like a lot.

I’ve also had surprisingly good experiences with the devshells, personally, it handles complex things like typescript language server envs out of the box for me. Not really for rust, but that’s more of a side effect of the silly situation with rust-analyzer.

I think it ended up being quite concise and pretty for my personal website. And that’s while using parcel, which I didn’t really expect to work at all, given the nested build toolingness.

I think dream2nix is impressive for having gotten as far as it has. There are very few projects that actually attempt to bridge the gaps between all these modern build/packaging tools, and so far dream2nix is doing a great job.

2 Likes

Hi, thanks for the info about the flag for logging! Did not know about it.

When talking about invalid states, I did not mean my terminal getting into an invalid state. I just mean the devshell… not working without any error message. I can dig through old messages I sent in the nixos discord server for more details perhaps.

I was a bit tired when I wrote that last message and I think I shoul’ve been more careful with the tone. Dream2nix seemed really cool and I was a bit disappointed. I know there’s probably a lot of work which went (and is still going into it), so I’m very sorry for being so negative. I know the rough edges will get polished one day.