Making a lib available to compiler without using nix-shell -p

Hi,

Apologies if there’s an obvious answer to this, but is there a canonical “NixOS” way for ensuring that a lib that I generally always need for compiling is available without having to use nix-shell -p ...?

For example, I’m playing with V (vlang), and when building/running something using its vweb framework, the one and only library dependency is openssl, which is installed.

Unless I drop into nix-shell via nix-shell -p openssl the compiler can’t find the openssl library.

Openssl is the kind of library that I’m always going to want available to compilers, so is there a way to make that so in the system’s configuration.nix?

Have you considered using direnv and a shell.nix in those projects to make the dependency explicit?

I’m trying to avoid that at the moment, wondering if there’s a general solution for making a lib available system wide?

But yeah, if there’s no system wide solution that’ll work in any project I’ll definitely dive into direnv and shell.nix for project specific set up.

The fact that dumping a simple shell.nix and using it through direnv is actually easier than installing libraries system wide is actually the bug pro of using nix in my opinion.

Though if you say, about every project in that language has to use that library, perhaps it should be a propagatedBuildInputs of the buildtools/compiler then?

Yeah, I do need to look into that for sure.

So, the current version of the vlang package does use propagatedBuildInputs that include all 3 potential dependencies, including openssl needed for the web stuff.

However, that’s a super old version, V is in rapid development, and the latest PR seems to have stalled.

I actually need a more recent version still, and have just built it locally and linked into my personal $PATH because I couldn’t get an overlay version to properly compile. That simple build works perfectly from a quick git clone and make.

I guess the real solution here is for me to work out how to do “proper” local builds from a git clone of my own GitHub fork so that I have the latest and greatest, and hopefully that propagatedBuildInputs will do the trick.

If I understood what you mean, and you want to modify nixpkgs and install your own vlang compiler globally, then possibly something like:

cd ~/programming
git clone git@github.com:ianmjones/nixpkgs # your own fork
cd nixpkgs
# Edit the vlang file
nix-env -f ~/programming/nixpkgs -iA vlang 

As for making the libraries available, apart from the propagatedBuildInputs, it’s whatever search path the compiler is looking for, set it to include ~/.nix-profile/path/you/want (e.g. ~/.nix-profile/lib/vlang or whatever it is).

As a bonus, if you are also hacking on the vlang compiler, instead of having a src= fetchFromGitHub you could have a path, e.g. src = ~/programming/vlang.

Of course, if you do manage to get the vlang compiler updated to the latest then perhaps you could try submitting a pull-request for it :slight_smile:

How does that overlay look like?

Perhaps we can make it build together?

Instead of forking, an overlay seems to be much more clean.

Nah, I’m already set up for nixpkgs development for a couple of other packages I maintain, but if I do get a more recent release of V up and running I might very well do this and submit a PR.

Yeah, this is more what I was meaning, at least for the time being while V is rapidly changing. I’ve already submitted a couple of PRs up stream for V, and would like to use that same local clone to also properly build for NixOS. This is great info, I’ll have a play with that different form of src.

Yup, sounds like a plan!

Thanks! I’ll have another crack at it, and if I get stuck I’ll be sure to post here.

Yeah, wasn’t talking about a fork of the nixpkgs pkg, I was talking about my personal “github fork” of vlang/v that easily builds with make locally, and works fine apart from that openssl thing. Sorry I wasn’t clear, but I was talking more in line of the second idea @KoviRobi mentioned for local dev of V itself.

Thanks to you both, great ideas here, I’ll have a play as soon as I get a chance and will be sure to post here if I get stuck.

Ok, I’m stuck!

Here’s as far as I’v got. This overlay is based on recent(ish) vlang/v commits that I know to be “good” (I can compile via cloned source and make, but ultimately get the same issues with openssl that you’ll see at the end of this post)…

self: super: {
  vlang = super.stdenv.mkDerivation rec {
    pname = "vlang";
    version = "0.1.25-imj";

    ###src = /home/ian/Projects/v;

    src = super.fetchFromGitHub {
      owner = "vlang";
      repo = "v";
      rev = "522de0871a6ad87c1e9da2c9ac771e0984b83528";
      sha256 = "090q1gd8nyrx9wzimibxz7znqavn14637h4zx4pinx1cbvd96091";
    };

    vc = super.fetchFromGitHub {
      owner = "vlang";
      repo = "vc";
      rev = "f5207d7ce2e1daf482420d32f4b656fed73b999b";
      sha256 = "06y4xy36z4fy647jayliyqi0dwvjr1q9ajshb8mz5yqw9mdmdw9m";
    };

    propagatedBuildInputs = [ self.glfw self.freetype self.openssl ];

    buildPhase = ''
      runHook preBuild
      echo "Using cc to build v.c..."
      cc $CFLAGS -std=gnu11 -w -o v2 $vc/v.c $LDFLAGS -lm
      echo "Using vc to build v..."
      ./v2 -o v3 cmd/v
      echo "Using v to build v (prod)..."
      ./v3 -o v -prod cmd/v
      runHook postBuild
    '';

    installPhase = ''
      runHook preInstall
      mkdir -p $out/{bin,lib,share}
      cp -r examples $out/share
      cp -r {vlib,thirdparty} $out/lib
      cp v $out/lib
      ln -s $out/lib/v $out/bin/v
      runHook postInstall
    '';

  };
}

$vc is the C source that V produces when something is changed in the core compiler, it’s what can be used to build the v compiler when v isn’t already available. The v compiler is effectively a v => c => binary machine.

So, v.c is used to compile v2, a “true” v. Then in this overlay, we use v2 to compile a v3, i.e. v self compile, then v3 to v itself in a production mode (as a good test and final small binary).

This ultimately produces this output…

unpacking source archive /build/522de0871a6ad87c1e9da2c9ac771e0984b83528.tar.gz
building '/nix/store/4mpdwkvl9fsbi2sp2wwqbf33ayjvsm5l-vlang-0.1.25-imj.drv'...
unpacking sources
unpacking source archive /nix/store/xibzz443kyjacl759lxn2d2f5vflx217-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
Using cc to build v.c...
Using vc to build v...
V panic: No such file or directory
builder for '/nix/store/4mpdwkvl9fsbi2sp2wwqbf33ayjvsm5l-vlang-0.1.25-imj.drv' failed with exit code 1
cannot build derivation '/nix/store/kz6m6gqfxyfahysc5ka212m8g2k2m83l-system-path.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/v2jkcf6n4wjaas6b622080q55cg6za7v-nixos-system-ians-apollo-20.03pre211190.a21c2fa3ea2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/v2jkcf6n4wjaas6b622080q55cg6za7v-nixos-system-ians-apollo-20.03pre211190.a21c2fa3ea2.drv' failed

Something is not happy in the v compiler, it can’t find some file, I’m presuming this is due to the way NixOS works when building a derivation in sandbox mode (because I want this to also work on Hydra if converted to a PR for the package).

If I instead reduce the build phase to …

buildPhase = ''
  runHook preBuild
  echo "Using cc to build v.c..."
  cc $CFLAGS -std=gnu11 -w -o v $vc/v.c $LDFLAGS -lm
  runHook postBuild
'';

… i.e. just use the C compiler to produce v, then the overlay works.

However, this for some reason does not “propagate” the libraries, e.g. this is what I get when trying to run one of the examples that relies on the openssl lib…

ian@ians-apollo:~/Projects/v/examples/vweb/vweb_assets (master) $ which v
/run/current-system/sw/bin/v
ian@ians-apollo:~/Projects/v/examples/vweb/vweb_assets (master) $ v run .
warning: /nix/store/yhisgrsfls9i5scfqn0w9f33pc8gxqnl-vlang-0.1.25-imj/lib/vlib/vweb/assets/assets.v:197:3: `am` is declared as mutable, but it was never changed
==================
/home/ian/.cache/v/vweb_assets.tmp.c:423:10: fatal error: openssl/ssl.h: No such file or directory
  423 | #include <openssl/ssl.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.
...
==================
(Use `v -cg` to print the entire error message)

V error: C error.

Please make sure that:
- You have all V dependencies installed.
- You did not declare a C function that was not included. (Try commenting your code that involves C interop)
- You are running the latest version of V. (Try running `v up` and rerunning your command)

If you're confident that all of the above is true, please try running V with the `-cg` option which enables more debugging capabilities.

ian@ians-apollo:~/Projects/v/examples/vweb/vweb_assets (master) $ nix-shell -p openssl
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/h3iim69ysaha4v7lgk6s26i57vmwffaw-bash-interactive-4.4-p23-dev
copying path '/nix/store/h3iim69ysaha4v7lgk6s26i57vmwffaw-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...

[nix-shell:~/Projects/v/examples/vweb/vweb_assets]$ v run .
warning: /nix/store/yhisgrsfls9i5scfqn0w9f33pc8gxqnl-vlang-0.1.25-imj/lib/vlib/vweb/assets/assets.v:197:3: `am` is declared as mutable, but it was never changed
Running a Vweb app on http://localhost:8081 ...

So that’s a lot of information there, sorry!

But maybe someone can see the errors in my ways?

I’d be super happy if we could just get the v.c compiled version to work properly with propagatedBuildInputs, that would be a major win!

uhhhh I’m trying to build vlang my self as well @ ianmjones any luck?