R packages & the `renv` library manager

gwmngilfen · February 14, 2020, 1:07pm

Hi all, brand new to NixOs so do forgive me if I’ve overlooked something obvious…

I’m a R user for my day-to-day work. I’m currently struggling to get the r-openssl to be available in a certain way. I’m using the “nixos.rWrapper” package for all these trials…

What works:

Doing this the “NixOS way” seems fine - I define the config.nix snippet suggested at https://github.com/NixOS/nixpkgs/blob/20bc3aa323fc4f20e2369be83c5b2d500bab1645/doc/languages-frameworks/r.section.md but I add “openssl” to the list of packages. Then when I start R, “openssl” is already available:

$ R
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
...
> library('openssl')
> openssl::sha1('yay')
[1] "4511f8c0e213c8ec51559eb4a1ffccbc7e42710e"

What breaks:

Great. However, I like to use the renv package manager in my R work, to keep projects from polluting each other with dependencies - whilst I realise that NixOS effectively does the same thing, I have to deploy to non-NixOS production hosts, and then the renv lockfile is a saviour.

So, if I enable renv it all goes horribly wrong. Openssl is (of course) not available, because renv has it’s own library. So, I try to build it, with the appropriate libs:

$ nix-shell -p pkgconfig openssl_1_1
[nix-shell]$ R
* Project '~/Test' loaded. [renv 0.9.3]
> install.packages('openssl')
Retrieving 'https://cran.rstudio.com/src/contrib/openssl_1.4.1.tar.gz' ...
        OK [file is up to date]
Installing openssl [1.4.1] ...
        FAILED
Error installing package 'openssl':
===================================

* installing *source* package ‘openssl’ ...
** package ‘openssl’ successfully unpacked and MD5 sums checked
** using staged installation
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include
Using PKG_LIBS=-L/nix/store/z9vsvmll45kjdf7j9h0vlxjjya6yxgc0-openssl-1.1.1d/lib -l:libssl.so.1.1 -l:libcrypto.so.1.1
** libs
rm -f aes.o base64.o bignum.o cert.o compatibility.o diffie.o envelope.o error.o hash.o info.o keygen.o keys.o onload.o openssh.o password.o pbkdf.o pem.o pkcs12.o pkcs7.o rand.o rsa.o signing.o ssl.o stream.o write.o x25519.o openssl.so bcrypt/libstatbcrypt.a bcrypt/bcrypt_pbkdf.o bcrypt/blowfish.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c aes.c -o aes.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c base64.c -o base64.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c bignum.c -o bignum.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c cert.c -o cert.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c compatibility.c -o compatibility.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c diffie.c -o diffie.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c envelope.c -o envelope.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c error.c -o error.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c hash.c -o hash.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c info.c -o info.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c keygen.c -o keygen.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c keys.c -o keys.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c onload.c -o onload.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c openssh.c -o openssh.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c password.c -o password.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c pbkdf.c -o pbkdf.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c pem.c -o pem.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c pkcs12.c -o pkcs12.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c pkcs7.c -o pkcs7.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c rand.c -o rand.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c rsa.c -o rsa.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c signing.c -o signing.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c ssl.c -o ssl.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c stream.c -o stream.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c write.c -o write.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c x25519.c -o x25519.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c bcrypt/bcrypt_pbkdf.c -o bcrypt/bcrypt_pbkdf.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -I"/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/include" -DNDEBUG -I/nix/store/2v45sw2109m2c7xzld4gc8g5bvjzybc4-openssl-1.1.1d-dev/include    -fpic  -g -O2  -c bcrypt/blowfish.c -o bcrypt/blowfish.o
/nix/store/ajrrkivdfvp8dp4vdg5hp1h5hblmanc9-binutils-2.31.1/bin/ar rcs bcrypt/libstatbcrypt.a bcrypt/bcrypt_pbkdf.o bcrypt/blowfish.o
/nix/store/291ldi6fqsbmkbvbs8is4mcg3jb64ld4-gcc-wrapper-8.3.0/bin/cc -shared -L/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/lib -o openssl.so aes.o base64.o bignum.o cert.o compatibility.o diffie.o envelope.o error.o hash.o info.o keygen.o keys.o onload.o openssh.o password.o pbkdf.o pem.o pkcs12.o pkcs7.o rand.o rsa.o signing.o ssl.o stream.o write.o x25519.o -Lbcrypt -lstatbcrypt -L/nix/store/z9vsvmll45kjdf7j9h0vlxjjya6yxgc0-openssl-1.1.1d/lib -l:libssl.so.1.1 -l:libcrypto.so.1.1 -L/nix/store/aky16nzr2y7ld2na5x1knafznmkqm3h0-R-3.6.1/lib/R/lib -lR
installing to /home/demo/Test/renv/staging/1/00LOCK-openssl/00new/openssl/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘openssl’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/demo/Test/renv/staging/1/00LOCK-openssl/00new/openssl/libs/openssl.so':
  libssl.so.1.1: cannot open shared object file: No such file or directory
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/home/demo/Test/renv/staging/1/openssl’
Error: install of package 'openssl' failed
Traceback (most recent calls first):
14: install.packages("openssl")
13: install(pkgs)
12: renv_install(records, library)
11: renv_install_staged(records, library)
10: renv_install_default(records, library)
 9: handler(package, renv_install_impl(record))
 8: renv_install_impl(record)
 7: withCallingHandlers(renv_install_package_local(record), error = function(e) {
        vwritef("\tFAILED")
        writef(e$output)
    })
 6: renv_install_package_local(record)
 5: renv_install_package_local_impl(package, path, library)
 4: r_cmd_install(package, path, library)
 3: r_exec(package, args, "install")
 2: r_exec_error(package, output, label)
 1: stop(error)
>

So it looks to me like R does find the openssl 1.1 library and builds “openssl” just fine, but for some reason the tests on the finished package can’t find the openssl libs anymore. Contrast this to the upstream rPackages.openssl package, which clearly does manage to pass this test.

It suggests to me that renv is doing some magic paths stuff that doesn’t work with NixOS’s worldview, but I don’t know how to debug it. Does anyone know where to start? I don’t want to give up using renv, it’s too integrated into my workflow on other OSs…

Side note: using nix-shell -p ... to build R packages in renv is working for other packages. For example, I had to do the same thing with zlib, and the build passed fine, and renv could load it afterwards without nix-shell (see https://pastebin.com/mNhAn05Y for an illustration). There’s just something odd about openssl…

seamsay · March 8, 2022, 1:27pm

For anyone that finds this in the future, as far as I can tell the only way around this is to use buildFHSUserEnv (the following is what I needed for one of my projects, different projects will need different dependencies):

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }:
    utils.lib.eachDefaultSystem (system:
      let pkgs = import nixpkgs { inherit system; };
      in {
        devShell = (pkgs.buildFHSUserEnv {
          name = "2542";
          targetPkgs = pkgs:
            (with pkgs; [
              binutils
              cargo
              curl.dev
              gcc
              libgit2
              libxml2.dev
              openssl.dev
              pandoc
              pkg-config
              R
              rPackages.renv
              rustc
              zlib.dev
            ]);
        }).env;
      });
}

The only thing (that I’ve tried) that fails with this is devtools::check(), but even that works if you set vignettes = FALSE (i.e. devtools::check(vignettes = F)).

I wonder if it’s possible to create some kind of renv2nix project so that this just works, but I don’t know renv’s (or in fact R’s) internals well enough.

alexv · March 8, 2022, 9:43pm

renv lock file format is documented here:

bsaul · October 11, 2022, 8:05pm

I’m not sure if my setup is different (I’m new to NixOS), but with the following flake.nix, I’m able to interact with an existing project that uses renv:

# https://discourse.nixos.org/t/r-packages-the-renv-library-manager/5881/2   
{
  description = "A basic flake for the mbbs package";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.05";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = nixpkgs.legacyPackages.${system};
    in {
      devShells.default =  pkgs.mkShell {
        nativeBuildInputs = [ pkgs.bashInteractive ];
        buildInputs = [
          pkgs.R
        ];
      }; 

    });
}