R installation within conda cannot find default libraries

Help
1

I installed conda as described in the wiki. conda-shell opens a shell, in which all conda-related commands work as expected (i.e. working with python environments works flawlessly). However, installing an R environment leads to a broken R installation: It seems that several standard-libraries cannot be found by the R interpreter:


(base) conda-shell-chrootenv:moritz@moxps:~$ conda create -n r_env r-essentials r-base
... everything installs fine ...
(base) conda-shell-chrootenv:moritz@moxps:~$ conda activate r_env
(r_env) conda-shell-chrootenv:moritz@moxps:~$ R

R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Error: package or namespace load failed for ‘utils’:
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: error in running command
Error: package or namespace load failed for ‘stats’:
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: error in running command
During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found

I failed in finding similar errors on the web, so I suspect it’s a nixos-specific error, however I am completely out of ideas on how to find out why R cannot find these packages. utils for example seems to be present in the environment:

$ cd .conda/envs/r_env/
$ find | grep utils
...
./lib/R/library/utils/libs/utils.so
./lib/R/library/utils/misc
./lib/R/library/utils/misc/exDIF.csv
./lib/R/library/utils/misc/exDIF.dif
./lib/R/library/utils/DESCRIPTION
./lib/R/library/utils/INDEX
./lib/R/library/utils/NAMESPACE
./lib/R/library/utils/iconvlist
./lib/R/library/Rcpp/include/Rcpp/utils
./lib/R/library/Rcpp/include/Rcpp/utils/tinyformat
./lib/R/library/Rcpp/include/Rcpp/utils/tinyformat/tinyformat.h
./lib/R/library/Rcpp/include/Rcpp/utils/tinyformat.h
...

Thank you for any help on this issue

2

my experiences with conda were also very poor on nixos, I ended up resorting to using a VM (it was just a 1 or 2 time use case).

I’m not familiar with R, is there some way to convey libraries? For example, I do the following with my python venvs:

    PYTHONPATH=$PWD/$venvDir/${python.sitePackages}:$PYTHONPATH

which will make the local files available to the python interpreter

3

This is a total shot in the dark, but can you try running R after setting R_LIBS to point to where your missing R packages are stored?

4

I was hoping for such a hint. Unfortunately, setting R_LIBS to any R-associated lib directory in the environment did not change anything.

Thank you

5

Thanks for the feedback. Good to hear that I am not the only one struggling with conda.

Since I don’t use R too often, it now suffices for me to have one global R-installation and put all the packages there.

No need to mess with conda for now…

6

Shoot. There are a couple of other environment variables to try, plus the .libPaths() function which shows the full list of search paths (might be helpful for debugging).

Here’s a blog post and an Rstudio discourse thread that provide more details.

I hope this works out!

roni

7

Thank you @roni.
libPaths seems to point to the correct location. I was reading a bit about the other environment variables, but didn’t see a light. I’m trying to use the nix-version of R for now. Thank you for the support!!

> .libPaths()
[1] "/home/moritz/.conda/envs/r_env/lib/R/library"

$ find "/home/moritz/.conda/envs/r_env/lib/R/library" | grep stats
/home/moritz/.conda/envs/r_env/lib/R/library/stats
/home/moritz/.conda/envs/r_env/lib/R/library/stats/Meta
...
8

Drat!

If you do figure this out, please report back here–I’d love to know what the correct solution is for this. Good luck!

roni

9

@roni I’ve got another promising trace:

https://github.com/ContinuumIO/anaconda-issues/issues/11133

R depends on which, which might not be present in the conda R environment. Maybe it could be as simple as to add the which-derivation to conda?

Placing ‘which’ into the conda binary folder did however not fix the issue (the R console still complains about not being able to load stats and utils packages)

10

Hi, the utils package from R_HOME. this issue appeared in R 4.0 or other environments.

  1. https://github.com/GTrunSec/jupyterWith/blob/c1ccbe1b0ee5703fd425ce0a3442e7e2ecfde352/kernels/ijulia/default.nix#L45

solution: set R_HOME path

  1. R_packages path set: https://github.com/GTrunSec/jupyterWith/blob/c1ccbe1b0ee5703fd425ce0a3442e7e2ecfde352/lib/directory.nix#L9
11

Why do you need conda?

Have you tried to use mach-nix?

12

mach-nix looks great! Thanks for pointing me to it @igel. Unfortunately, I am using a system that requires conda (snakemake) to run the way I want it to.

@GTrunSec although it heavily looks like this is the issue for me, I still get the same error even after setting R_HOME (it was indeed unset before…):

(r_env) conda-shell-chrootenv:moritz@moxps:~/.conda/envs/r_env/bin$ R_HOME=/home/moritz/.conda/envs/r_env/lib/R ./R

Do you have a suggestion what I could still try?

Thank you both for the help!

13

@moritzschaefer Your pointer to which was a good one. Using strace -f you can find:

[pid  1144] execve("/usr/bin/which", ["/usr/bin/which", "uname"], 0x1800008 /* 177 vars */) = 0

which here reports success. I got there after adding which to conda-shell:

conda = pkgs.conda.override {                         
    extraPkgs = [ pkgs.which ];                                                                                                                                                                                                                                                
};

with this R no longer prints failures.

3 Likes
14

That is so great to hear! Do you want to start a PR for nixpkgs to get it fixed?

I added an overlay with your override, and can confirm that it works!

1 Like
15

@moritzschaefer, looks like you found a solution! Woo hoo!

roni

1 Like
16

Hi @unode - I am running into the same problems, however with a different Conda setup over julia, concretely pkgs.julia_1.9-bin

The overwrite you mention does not work there… what am I doing wrong?

In nix repl '<nixpkgs/nixos>' I am doing the following

nix-repl> pkgs.julia_19-bin.override {extraPkgs = [ pkgs.which ];}
error: anonymous function at /nix/store/4yxy56fn7phrrkvapvd2cc8kzy958krd-nixos-23.05/nixos/pkgs/development/compilers/julia/1.9-bin.nix:1:1 called with unexpected argument 'extraPkgs'

       at /nix/store/4yxy56fn7phrrkvapvd2cc8kzy958krd-nixos-23.05/nixos/lib/customisation.nix:80:16:

           79|     let
           80|       result = f origArgs;
             |                ^
           81|
17

If found your answer unode at R 3.6.0 depends on `which` and that's not always present · Issue #11133 · ContinuumIO/anaconda-issues · GitHub

I can confirm what @moritzschaefer said and also that adding which to /usr/bin/which fixes the problem.
The path is hardcoded, simply having which in $PATH is not sufficient.

18

@schlichtanders which was added to the conda environment. In your case you are trying to modify julia’s derivation directly. conda environments in nix are a little special because they use a FHS sandbox that fakes the presence of some binaries in /usr/bin. This doesn’t usually happen with other software unless the software has special requirements.

It’s not clear from your message if the errors in julia are also caused by which missing from /usr/bin nor if julia-bin is hardcoded to look for binaries under /usr/bin/.