You mention nushell with its caveats (though I think build closure size isn’t that big of a deal, as you said yourself, only build servers need to install the build scripts’ interpreter), also there is oils. Perhaps raw python.
I agree there is bikeshedding that’d need to be done, but alternatives exist that at least solve some of the main painpoints.
In my eyes the issues that need to be solved are mainly documentability and implicit dependency propagation; those two issues are the core of what makes developing in the bowels of nixpkgs hard. This could probably even be solved with some bonus tooling and enforced rules around how to with bash, but once you go to those lengths you might as well dream big and get rid of the footguns.
It’s not actually the closure size, it’s that special care would be needed in packaging everything in its closure, since it couldn’t be packaged with stdenv. Making that include rustc would be tough on the bootstrapping architecture.
At the cost of how many new pain points, though? That’s what I’m really worried about. I can’t imagine python actually working for this problem domain. Way too hard to do the things that are easy in bash, that we need to be easy to write derivations easily.
Isn’t rustc becoming part of the stdenv kind of a foregone conclusion? It’s hard to see a world in which that doesn’t become mandatory.
But yeah, I’m with you that a POSIX-shell alternative besides nushell is hard to find, especially if you limit it to projects that can be built with the current stdenv. Most modern replacements of anything will be written in go or rust.
Oils might actually be a good middle ground, but I admit I haven’t tested it in anger.
Maybe. But the bootstrapping issue will probably keep it out for as long as we can feasibly do so.
I don’t know much about oil, but on cursory examination, it does look like it might be decent for the problem space. Python is in the stdenv build closure already, iirc, because the gcc build process needs it, I think? Might be worth looking more into.
oils uses (vendors?) python2, which is pretty unacceptable for a stdenv dependency. nushell on the other hand is just a worse powershell (lacks basic functionality ime). I think we are stuck with bash for now, unless we made our own bespoke language.
According to the docs, oils release tarballs don’t need any python to build them, just a c++ compiler. You need python to properly develop it. It’s a 2-stage build process, and they don’t normally expect builders to do the python-to-c++ stage. So technically nixpkgs could package it without involving python2. Not saying that isn’t a point against it, though, from the perspective of the reliability of the project going forward. Using long-EOL software isn’t a good sign.
Not saying that this is better than bash in the context of nixpkgs, but I’ve been thinking about this question for a long time, and my two best contenders are:
They have interpolation syntax that makes it possible to create process spawning DSL which is both convenient and safe and they have good support for concurrency (which is less relevant if your task is just building stuff).
Janet in particular is also very nice from bootstrapping perspective, it’s a single C file that takes less than a second to compile.
Because worse is better! Janet isn’t really a lisp: it’s a VM which supports pedestrian data types (mutable & immutable hash tables, mutable & immutable arrays, and mutable and immutable byte-arrays, as a special case), implements GC & event loop with fibers & images.
Basically, my reading is that Scheme and Common Lisp are focused on being the purest expression of LISP, while janet is focused on runtime features (mutable hash tables!) and on most economical way to expose those features to the programmer, a difference of values.
Janet does look interesting, but going back to my original problem statement:
Janet appears to address the latter, assuming a janet-based stdlib would forego hook propagation via nativeBuildInputs in favor of something a bit less magic, but the examples from that page don’t strike me as what you’d expect from a community that is very doc-oriented.
Do you think the doc story is actually better than bash’?
I know lisps can do it, but frankly the mess that is emacs doesn’t make me very confident this would be any better - though that’s admittedly more because emacs has poor import hygiene and scope creep, emacs’ doc story is actually among the best out there IMO.
Having had a few showers to ponder the situation, I’m kinda starting to think that a neoshell specifically designed for scripting, rather than interactive use, would actually be a useful thing to design from scratch.
Nushell gets closer, but is also not ideal. You’d want good tooling support, like a full LSP, for the kinds of use cases that nixpkgs belongs to.
I have been wondering the same thing, hence the “shell-like evaluator” I’ve mentioned before. Though, it was probably not a good description - I’m not good with technical terminologies.
As I understand it, you’re proposing the creation of something with an actual, modern syntax (instead of keeping it POSIX-compliant), which would definitely be interesting.
To be fair though, I would explicitly not make it POSIX-compatible. Not even remotely. It’d need proper modern language features, like actual arrays, function args, modules, proper imports instead of nested shell invocations. As well as doc comments with a proper doc generator, and tooling to tell people off for not writing them.
POSIX shell simply isn’t a good scripting language. I’m not convinced a good scripting language exists; all I know of are terminal-oriented shells and programming languages that run in an interpreter and lack any convenience features for managing subprocesses.
I agree with your analysis. For me, Janet pareto-dominates bash for scripting, as it is less worse and more better, but it doesn’t really have a killer advantage (Janet doesn’t assume POSIX, which is massive, but it isn’t really relevant for nixpkgs). TypeScript does have such an advantage (types & interfaces, and the docs are downstream of that), but it comes with baggage of its own (strings being UTF-16-ish, and byte buffers being second class is a big liability in this domain).
If writing derivations started to require writing lisp, any variant of it, I’d be out. End of story. Some programmers like lisp, for reasons I fundamentally cannot understand, but for me, personally, it’s just horrific. Even bash is 1000x better to me.
I’m sure I’m not the only one who would feel this way…
Sorry to burst your bubble, but I see absolutely no chance of rewriting stdenv in something other than a shell language. It’s certainly possible to create a stdenv-ng that re-implements the standard build phases of stdenv.mkDerivation, maybe supports some common build system like cmake and then you opt-in some packages, but that’s it.
Rewriting the support for all the languages, frameworks and build hooks (there are 49 + 39 more build hooks documented in the Nixpkgs manual) is going to take forever, unless you can call or somehow reuse the existing code.
Something like Oil could work in theory, but it’s not going to be easy to get maintainers to suddendly start rewriting their packages in an even more obscure language than Nix.
It made for interesting reading at the time, and puts things a bit in perspective. Personally I’d agree with most above sentiments that the effort is probably better spent on something else, but since stdenv is relatively “pluggable” one could always experiment away of course.
Nix uses every bashism there is. It depends heavily on (assoc) arrays, converting that to POSIX is hard and makes it even less readable.
Anyways, I’ve patched out the things which Oils hasn’t included 2 years ago (e.g. declare -i) to test how Oils works as POSIX Shell for building packages. But a gnarly bug that was only fixed recently in Oils resulted in a perl build error and prevented me from building the stdenv itself with Oils. So I switched to building alpine linux packages instead - and that’s what we’ve now almost finished as part of an NLNet grant. Realistically trying to build all of Nixpkgs with Oils would now probably result in only a few build issues based on Oils bugs/POSIX incompatibilities and thus be a reasonable target. I can pick the nixpkgs work back up if there is (still!) interest.
Oils does have a very sane modern shell language, with a smooth upgrade path. You can take POSIX/Bash and upgrade it somewhat step-by-step. The biggest advantage for nixpkgs would be structured data and a better eval API, which could make the whole hooks logic massively more maintainable! It’s still somewhat experimental, though, since e.g. the eval API has only been challenged/tested by me until now. oh and it’s also OK if people keep writing Bash in the package definition. They can use a single shopt flag to indicate that they want modern features, and then start using Ysh. So with Oils no package would need to be rewritten.
Andy (Oils maintainer) is currently very absorbed with personal matters, so the progress on the current NLNet fund has slowed down since January. Therefore there will probably not be another NLNet fund lead by him this year. His plan would be focussing on Ysh (the new parts) as well as a Rust Port/a “liboils” library for building tools like LSP, linter, etc. Unless someone else has time to take it up, that’ll have to wait for next year. But I can imagine upgrading the nixpkgs internals to ysh as part of the grant could be great for the project.
And re python2; It sounds fucked up, but is completely reasonable in this project. I guess we’d need to use the vendored version from the project itself or stick to the tarball. I recently thought of asking Andy to add all tarball releases to a git repository as part of the release automata. Since it’s all readable code, that might be a reasonable compromise for Maintainers seeking an observable target. E.g. build the tarball on each commit and add the diff to a “release repo”. Though I’m not sure if that would just be useless overhead, what do you think?
It’s not reasonable for nixpkgs to use python2, as stdenv would end up being marked insecure (therefore not built on hydra). (That’s also not really an explanation beyond “trust me, it’s fine”.)
The Oils code is written in a few thousand lines of a very strict statically typed Python2 subset. This code is then transpiled to C++. That allows the codebase itself to stay extremely small (right now ~35k lines of python2 + <7k C++ code compared to 140k C++ for Bash itself. And Bash has no garbage collection or YSH features!)
Python2 only ever runs the trusted code from the repository itself. Once the Code is transpiled to C++, it does NOT require python2 anymore. The transpiled tarball only depends on a C++ Compiler, a POSIX shell and a libc (plus optionally libreadline). Oils als houses an extensive test suite, which among other things makes sure that the Python and the C++ version don’t differ.
Since Oils vendors a python2 (AFAIU even a stripped down version only supporting the required subset), I would argue if we don’t want to trust the tarball, then we should instead compile the vendored python2 as part of a oils-tarball-package. The result of this package - clean C++ code - is then the input for the oils-bin package. And no python2 is ever near the oils-bin package, or in any buildinputs.
And to reiterate; If the concern for not using the generated C++ tarball is a lack of observability, what speaks against using a git repository with all the generated C++ code tracked? Or are there other reasons against the C++ version?
You’re talking about the reproducible bootstrapping distro. In practice it doesn’t matter much, of course, but it definitely feels wrong to make the distro whose entire point is clean reproducible builds dependent on what is effectively a compiled code generation artifact.
This means you can’t .overrideAttrs the python source for custom changes, recreate the C++ tarball from a git repo, or do any of the source provenance proving that we all want from $reproduciblebuildtool.
If it isn’t part of the stdlib I suppose it’s usually alright to do stuff like this. But in stdlib this would make the entirety of nixpkgs not-reproducible from source.
With that aside, why on earth are oils still using python2? Some weird dogmatic choice or belief that python2 is superior? If so, that’s also a red flag. If it’s just maintenance lag, well, alright, but that’s also kind of a red flag, it’s been almost 20 years since python2 was superseded.
I can’t think of any excuse for still using python2 that isn’t indicative of a project you shouldn’t build your stack around.
