(Re-)packaging cosmopolitan / cosmocc / llamafile

Howdy,

First post, be gentle.

Putting the main ask up front, for those who don’t want to read the context: is there a canonical example of how to write a “nonfree” nix package, i.e. a package that just fetches some binary blob and puts it in the nix store?

Now, the context:

I would like to try and figure out how to provide modern cosmopolitan and cosmocc nixpkgs. This seems to be a prerequisite to providing a nixpkg for llamafile, which seems to also be desirable.

There is a cosmopolitan and cosmocc package already, but it is ancient, and the build process has changed since then.

Nowadays, the only supported compiler for cosmopolitan libc is cosmocc, which is a fairly light wrapper around a cosmopolitan build of gcc. So we are in a situation in which new versions of the compiler are bootstrapped from existing versions of the compiler. (There is not really a concept of “platform” in cosmopolitan libc; the idea is that the same set of binaries should work everywhere, so the FreeBSD distribution is bit-for-bit identical with the Linux distribution, the Windows distribution, etc.)

I guess there are a few ways to tackle this. One that seems like it’d allow a good bit of flexibility would be to have a cosmocc-bootstrap (or somesuch) package that just fetches and unpacks the zip file for the latest version, and then a cosmocc package that depends on that package and uses it to build a distribution from the git repository.

Now a certain version of cosmopolitan wants to be built by a certain version of cosmocc, so the two packages would often need to update in lockstep. But this scheme would, I suppose, make it easier to do development on the cosmocc code from within nix? Otherwise it seems pointless to have the layer of indirection and the cosmocc-bootstrap package may as well just be named cosmocc and behave like a “nonfree” package.

Anyway, once there is a current cosmocc, it should be more or less trivial to make a cosmopolitan package. And from there, it should be easy enough to make a llamafile package.

Any suggestions welcome, thanks.

Oh for the record my familiarity level with Nix/NixOS is “I have a laptop and a VM running it and I can edit configuration.nix, and I’ve made lightweight edits to existing packages with lots of hand-holding in the past.”

I am interested in trying this tomorrow!

The term is just “opaque blob” instead of “nonfree”.

well, the ‘canonical way’ is to install on $out/bin and patchelf the binary to make it locate the libs.

We need to find an old release or Git commit of cosmocc that can be built by pure gcc.

1 Like

An alternative to doing it the bootstrap way (which seems to be much in the vein of what currently happens with Rust as well) would be to have -bin packages for those that are not built from source and instead are simply distributing the upstream binaries. There’s already a lot of those derivations in nixpkgs, so maybe that could be the “easy” way to get cosmopolitan and, more specifically cosmocc, up and running for making llamafile distributions.

The way I feel about it is that although cosmocc and the rest of the cosmopolitan ecosystem can have a great benefit to the Nix ecosystem, they’re not actually required to be packaged at all, and in fact work extremely well being simple and lose files on the system. After all, one of the benefits of these are that it’s very easy to bring your own applications that run on all platforms, including with the ability to maybe modify them too (considering the PKZIP part).

1 Like

I guess there are a few ways to tackle this. One that seems like it’d allow a good bit of flexibility would be to have a cosmocc-bootstrap (or somesuch) package that just fetches and unpacks the zip file for the latest version, and then a cosmocc package that depends on that package and uses it to build a distribution from the git repository.

Now a certain version of cosmopolitan wants to be built by a certain version of cosmocc, so the two packages would often need to update in lockstep. But this scheme would, I suppose, make it easier to do development on the cosmocc code from within nix? Otherwise it seems pointless to have the layer of indirection and the cosmocc-bootstrap package may as well just be named cosmocc and behave like a “nonfree” package.

This is exactly how rustc is currently packaged. Upstream only guarantees that rustc version N can be compiled with rustc version N-1 (iirc there was even one version where version N could not boostrap itself, only version N-1 worked).
The boostrap compiler is here nixpkgs/pkgs/development/compilers/rust/bootstrap.nix at 291683af8f01662a84d5b648caf5a5ab95bf3a31 · NixOS/nixpkgs · GitHub and the actual compiler used elsewhere in nixpkgs is compiled from source nixpkgs/pkgs/development/compilers/rust/rustc.nix at 291683af8f01662a84d5b648caf5a5ab95bf3a31 · NixOS/nixpkgs · GitHub but with the binary rustc compiler: nixpkgs/pkgs/development/compilers/rust/default.nix at 291683af8f01662a84d5b648caf5a5ab95bf3a31 · NixOS/nixpkgs · GitHub