Problem with a rust build (crane/fenix)

blaggacao · February 25, 2023, 4:02pm

Fails:

nix run github.com:paisano-nix/mdbook-paisano-preprocessor -- --help

Works:

tmp=$(mktemp -d);
git clone https://github.com/paisano-nix/mdbook-paisano-preprocessor $tmp;
direnv allow $tmp;
direnv exec $tmp \
   cargo run -- --help

What entropy (ordering?) is potentially crane bringing to the table to make the build fail?

figsoda · February 25, 2023, 5:11pm

This is probably because you are using the latest profile, switching to default fixes the issue for me

--- a/nix/repo/rust.nix
+++ b/nix/repo/rust.nix
@@ -3,4 +3,4 @@ let
 in
   # export fenix toolchain as it's own package set
   # change "stable" to "latest" for nightly rust
-  builtins.removeAttrs (fenix.packages.latest // {inherit (fenix.packages) rust-analyzer;}) ["withComponents"]
+  builtins.removeAttrs (fenix.packages.default // {inherit (fenix.packages) rust-analyzer;}) ["withComponents"]

blaggacao · February 25, 2023, 7:42pm

While not a fix for the root cause of X, it’s a very welcome un-blocker! Thank you! It wouldn’t have occurred to me.

And looking into how the shell was set up send me onto a useful tangent, as well. So it was fully worth it: feat: add RUST_SRC_PATH in case we can detect it by blaggacao · Pull Request #246 · numtide/devshell · GitHub — may you review if that’s a sound approach?

blaggacao · February 25, 2023, 8:30pm

Oh, just stumbled over this. Worthy of opening an issue in fenix?!?

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ la /nix/store/67flfpia8dl0i5g1gj1lp758pfd7qppi-devshell-dir/bin
Permissions Size User      Date Modified Name
[ ... ]
lrwxrwxrwx    85 blaggacao 31 Dec  1969  cargo -> /nix/store/vdqdj12nv0p8fs76d35sbmyyhy7kk6qk-rust-nightly-default-2023-02-25/bin/cargo
lrwxrwxrwx    92 blaggacao 31 Dec  1969  cargo-clippy -> /nix/store/vdqdj12nv0p8fs76d35sbmyyhy7kk6qk-rust-nightly-default-2023-02-25/bin/cargo-clippy
[ ... ]

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly took 7s
❯ cargo clippy
error: failed to parse manifest at `/home/blaggacao/src/github.com/paisano-nix/mdbook-paisano-preprocessor/Cargo.toml`

Caused by:
  the cargo feature `edition2021` requires a nightly version of Cargo, but this is the `stable` channel
  See https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.
  See https://doc.rust-lang.org/cargo/reference/unstable.html#edition-2021 for more information about using this feature.

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ cargo --version
cargo 1.69.0-nightly (9d5b32f50 2023-02-22)

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ cargo clippy --version
clippy 0.1.54 (a178d03 2021-07-26)

Observed when using from .#default.toolchain.

figsoda · February 25, 2023, 9:01pm

I’m not sure what the issue is since I’m not familiar with std and devshell. Looking at devshell's implementation, it might be because that it is not using withComponents. rust-analyzer should also be able to find rust-src correctly without setting $RUST_SRC_PATH if withComponents or combine is used

blaggacao · February 25, 2023, 9:04pm

Oh sorry, I should have presented the issue in terms of nix build github:nix-community/fenix#default.toolchain. My bad!

❯ /home/blaggacao/src/github.com/nix-community/fenix/result/bin/cargo --version
cargo 1.69.0-nightly (9d5b32f50 2023-02-22)

❯ /home/blaggacao/src/github.com/nix-community/fenix/result/bin/cargo clippy --version
clippy 0.1.54 (a178d03 2021-07-26)

❯ /home/blaggacao/src/github.com/nix-community/fenix/result/bin/cargo clippy
error: failed to parse manifest at `/home/blaggacao/src/github.com/paisano-nix/mdbook-paisano-preprocessor/Cargo.toml`

Caused by:
  the cargo feature `edition2021` requires a nightly version of Cargo, but this is the `stable` channel
  See https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.
  See https://doc.rust-lang.org/cargo/reference/unstable.html#edition-2021 for more information about using this feature.

figsoda · February 25, 2023, 9:06pm

cargo clippy picks up cargo-clipppy from $PATH, can you try nix shell instead of nix build and see if that works?

blaggacao · February 25, 2023, 9:08pm

Marvelous! Thx! Will hunt this down further!!

❯ /home/blaggacao/src/github.com/nix-community/fenix/result/bin/cargo-clippy --version
clippy 0.1.69 (c5c7d2b 2023-02-24)

❯ /home/blaggacao/src/github.com/nix-community/fenix/result/bin/cargo-clippy
warning: the cargo feature `edition2021` has been stabilized in the 1.56 release and is no longer necessary to be listed in the manifest
  See https://doc.rust-lang.org/cargo/reference/manifest.html#the-edition-field for more information about using this feature.
   Compiling proc-macro2 v1.0.51
   Compiling quote v1.0.23
   Compiling unicode-ident v1.0.6
   Compiling syn v1.0.107
   Compiling version_check v0.9.4

blaggacao · February 25, 2023, 9:12pm

Strange thing: (devshell one’s works, but cargo picks up something else

~
❯ which cargo-clippy
cargo-clippy not found

~
❯ which cargo
cargo not found

mdbook-paisano-preprocessor on  main [!] is 📦 v0.4.0 via 🦀 v1.67.1 took 7s
❯ which cargo-clippy
/nix/store/fxs6nmshhxijx92bphcymlr2rsw5k9xd-devshell-dir/bin/cargo-clippy

mdbook-paisano-preprocessor on  main [!] is 📦 v0.4.0 via 🦀 v1.67.1
❯ /nix/store/fxs6nmshhxijx92bphcymlr2rsw5k9xd-devshell-dir/bin/cargo-clippy
warning: the cargo feature `edition2021` has been stabilized in the 1.56 release and is no longer necessary to be listed in the manifest
  See https://doc.rust-lang.org/cargo/reference/manifest.html#the-edition-field for more information about using this feature.
   Compiling proc-macro2 v1.0.51
   Compiling quote v1.0.23
   Compiling syn v1.0.107

But cargo clippy failed.

blaggacao · February 25, 2023, 9:18pm

The installation root is determined, in order of precedence:

--root option
CARGO_INSTALL_ROOT environment variable
install.root Cargo config value
CARGO_HOME environment variable
$HOME/.cargo

Got it!

❯ $HOME/.cargo/bin/cargo-clippy --version
clippy 0.1.54 (a178d03 2021-07-26)

figsoda · February 25, 2023, 9:22pm

Is there some way I can reproduce that? I tried this on HEAD and it worked as expected

~/mdbook-paisano-preprocessor on  main
❯  which cargo cargo-clippy
/nix/store/67flfpia8dl0i5g1gj1lp758pfd7qppi-devshell-dir/bin/cargo
/nix/store/67flfpia8dl0i5g1gj1lp758pfd7qppi-devshell-dir/bin/cargo-clippy

~/mdbook-paisano-preprocessor on  main
❯  cargo clippy --version
clippy 0.1.69 (c5c7d2b 2023-02-24)

~/mdbook-paisano-preprocessor on  main
❯  cargo clippy
warning: using `Result.or_else(|x| Err(y))`, which is more succinctly expressed as `map_err(|x| y)`
  --> src/reference.rs:67:3
   |
67 | /         std::fs::read_to_string(s)
68 | |             .or_else(|_| Err(askama::Error::Fmt(std::fmt::Error)))
   | |__________________________________________________________________^ help: try this: `std::fs::read_to_string(s).map_err(|_| askama::Error::Fmt(std::fmt::Error))`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#bind_instead_of_map
   = note: `#[warn(clippy::bind_instead_of_map)]` on by default

warning: using `Result.or_else(|x| Err(y))`, which is more succinctly expressed as `map_err(|x| y)`
  --> src/reference.rs:97:3
   |
97 | /         cmark(events.iter(), &mut buffer)
98 | |             .or_else(|_| Err(askama::Error::Fmt(std::fmt::Error)))?;
   | |__________________________________________________________________^ help: try this: `cmark(events.iter(), &mut buffer).map_err(|_| askama::Error::Fmt(std::fmt::Error))`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#bind_instead_of_map

warning: binary comparison to literal `Option::None`
  --> src/lib.rs:84:11
   |
84 | ...                   if conf.registry == None {
   |                          ^^^^^^^^^^^^^^^^^^^^^ help: use `Option::is_none()` instead: `conf.registry.is_none()`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#partialeq_to_none
   = note: `#[warn(clippy::partialeq_to_none)]` on by default

warning: the borrowed expression implements the required traits
   --> src/lib.rs:181:32
    |
181 |     let mut sysevl = Command::new(&cmd);
    |                                   ^^^^ help: change this to: `cmd`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
    = note: `#[warn(clippy::needless_borrow)]` on by default

warning: the borrowed expression implements the required traits
   --> src/lib.rs:196:30
    |
196 |     let mut eval = Command::new(&cmd);
    |                                 ^^^^ help: change this to: `cmd`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: `mdbook-paisano-preprocessor` (lib) generated 5 warnings (run `cargo clippy --fix --lib -p mdbook-paisano-preprocessor` to apply 5 suggestions)
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s

figsoda · February 25, 2023, 9:27pm

Ah, didn’t see your new message. I’m assuming your problem with clippy is resolved?

blaggacao · February 25, 2023, 9:28pm

mdbook-paisano-preprocessor on  main [!] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ cargo clippy --version
clippy 0.1.54 (a178d03 2021-07-26)

mdbook-paisano-preprocessor on  main [!] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ export CARGO_HOME=$CARGO_INSTALL_ROOT

mdbook-paisano-preprocessor on  main [!] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ cargo clippy --version
clippy 0.1.69 (c5c7d2b 2023-02-24)

So CARGO_HOME needs to be set to the root of the drv or else it may pick up something else…

But I don’t see it set in fenix, either:

fenix on  main
❯ rg 'CARGO_HOME'

so maybe we can tweak things a little?

AHHH - one thing leads to another …

❯ env | rg 'CARGO'
CARGO_HOME=/nix/store/vdqdj12nv0p8fs76d35sbmyyhy7kk6qk-rust-nightly-default-2023-02-25

❯ cargo b
error: failed to get `anyhow` as a dependency of package `mdbook-paisano-preprocessor v0.4.0 (/home/blaggacao/src/github.com/paisano-nix/mdbook-paisano-preprocessor)`

Caused by:
  failed to create directory `/nix/store/vdqdj12nv0p8fs76d35sbmyyhy7kk6qk-rust-nightly-default-2023-02-25/registry/index/github.com-1ecc6299db9ec823`

Caused by:
  Permission denied (os error 13)

blaggacao · February 25, 2023, 9:52pm

This shell init hook snipped managed to solve the issue. It prepares a writable CARGO_HOME in the isolated scope of the repository (under .gitignore) and then links *.toolchain's folder tree into it (so that also the right clippy is picked up):

{
      devshell.startup.link-cargo-home = {
        deps = [];
        text = ''
          # ensure CARGO_HOME is populated
          mkdir -p $PRJ_DATA_DIR/cargo
          ln -s -t $PRJ_DATA_DIR/cargo $(ls -d ${cell.rust.toolchain}/*)
        '';
      };

      env = [
        {
          # ensure subcommands are picked up from the right place
          # but also is writable
          name = "CARGO_HOME";
          eval = "$PRJ_DATA_DIR/cargo";
        }
        {
          # ensure we know ghere rustup_home will be
          name = "RUSTUP_HOME";
          eval = "$PRJ_DATA_DIR/rustup";
        }
      ];
}

figsoda · February 25, 2023, 9:53pm

Do you get the correct version if you run cargo-clippy? Weird that cargo would pick up something different. CARGO_HOME does a whole lot more so fenix should probably not change that.

CARGO_HOME — Cargo maintains a local cache of the registry index and of git checkouts of crates. By default these are stored under $HOME/.cargo

--root option
CARGO_INSTALL_ROOT environment variable
install.root Cargo config value
CARGO_HOME environment variable
$HOME/.cargo

This seems to be documentation for cargo install, I’m not sure if this is related to cargo clippy at all

blaggacao · February 25, 2023, 9:54pm

Yes. The issue is that cargo clippy picks up $HOME/.cargo/bin/cargo-clippy.

That is consistent with: cargo/lib.rs at 06178d7a7c35cc2c058a4cf9391a21a4c5662eb0 · rust-lang/cargo · GitHub

Can you check what $HOME/.cargo/bin/cargo-clippy gives for you? Maybe it happens to be the same version?

figsoda · February 25, 2023, 9:56pm

I don’t have anything installed in my $HOME/.cargo/bin so maybe that’s why I’m not experiencing that

blaggacao · February 25, 2023, 9:59pm

~
❯ mv .cargo .cargi

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly took 6m41s
❯ unset CARGO_HOME

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ cargo clippy --version
clippy 0.1.69 (c5c7d2b 2023-02-24)

~
❯ mv .cargi .cargo

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ cargo clippy --version
error: no override and no default toolchain set

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ unset RUSTUP_HOME

mdbook-paisano-preprocessor on  main [!+] is 📦 v0.4.0 via 🦀 v1.69.0-nightly
❯ cargo clippy --version
clippy 0.1.54 (a178d03 2021-07-26)

Yes, what you just concluded is the case.

So in resume:

$HOME/.cargo's existence has an “exorbitant privilege” in plugin resolution
It’s existence has therefore unintended side effects on the cargo environment

figsoda · February 25, 2023, 10:01pm

I put a something in my .cargo/bin/cargo-clippy and I’m still not experiencing this

~
❯  cargo clippy
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

~
❯  nix shell nixpkgs#direnv

~
❯  direnv hook fish | source

~
❯  mdbook-paisano-preprocessor/
direnv: loading ~/mdbook-paisano-preprocessor/.envrc
direnv: using std ./nix //repo/shell:default
direnv: Watching: ./nix/repo/shell.nix
🔨 Welcome to Paisano MdBook

[general commands]

  adrgen        - A command-line tool for generating and managing Architecture Decision Records
  mdbook        - Create books from MarkDown
  menu          - prints this menu
  treefmt       - one CLI to format the code tree

[rust dev]

  cargo         - Downloads your Rust project's dependencies and builds your project
  rust-analyzer - A modular compiler frontend for the Rust language
  rustc         - A safe, concurrent, practical language
  rustfmt       - A tool for formatting Rust code according to style guidelines

nixago: updating repositoriy files
nixago: 'treefmt.toml' link is up to date
nixago: 'docs/book.toml' copy is up to date
nixago: '.conform.yaml' link is up to date
nixago: '.editorconfig' copy is up to date
nixago: '.github/settings.yml' copy is up to date
nixago: 'lefthook.yml' link is up to date
nixago: 'adrgen.config.yml' link is up to date
direnv: export +CFLAGS +DEVSHELL_DIR +LIBRARY_PATH +NIXPKGS_PATH +NODE_PATH +PKG_CONFIG_PATH +PRJ_CACHE_DIR +PRJ_DATA_DIR +PRJ_ROOT +RUST_SRC_PATH +name ~PATH ~XDG_DATA_DIRS

~/mdbook-paisano-preprocessor on  main
❯  cargo clippy
warning: using `Result.or_else(|x| Err(y))`, which is more succinctly expressed as `map_err(|x| y)`
  --> src/reference.rs:67:3
   |
67 | /         std::fs::read_to_string(s)
68 | |             .or_else(|_| Err(askama::Error::Fmt(std::fmt::Error)))
   | |__________________________________________________________________^ help: try this: `std::fs::read_to_string(s).map_err(|_| askama::Error::Fmt(std::fmt::Error))`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#bind_instead_of_map
   = note: `#[warn(clippy::bind_instead_of_map)]` on by default

warning: using `Result.or_else(|x| Err(y))`, which is more succinctly expressed as `map_err(|x| y)`
  --> src/reference.rs:97:3
   |
97 | /         cmark(events.iter(), &mut buffer)
98 | |             .or_else(|_| Err(askama::Error::Fmt(std::fmt::Error)))?;
   | |__________________________________________________________________^ help: try this: `cmark(events.iter(), &mut buffer).map_err(|_| askama::Error::Fmt(std::fmt::Error))`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#bind_instead_of_map

warning: binary comparison to literal `Option::None`
  --> src/lib.rs:84:11
   |
84 | ...                   if conf.registry == None {
   |                          ^^^^^^^^^^^^^^^^^^^^^ help: use `Option::is_none()` instead: `conf.registry.is_none()`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#partialeq_to_none
   = note: `#[warn(clippy::partialeq_to_none)]` on by default

warning: the borrowed expression implements the required traits
   --> src/lib.rs:181:32
    |
181 |     let mut sysevl = Command::new(&cmd);
    |                                   ^^^^ help: change this to: `cmd`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
    = note: `#[warn(clippy::needless_borrow)]` on by default

warning: the borrowed expression implements the required traits
   --> src/lib.rs:196:30
    |
196 |     let mut eval = Command::new(&cmd);
    |                                 ^^^^ help: change this to: `cmd`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: `mdbook-paisano-preprocessor` (lib) generated 5 warnings (run `cargo clippy --fix --lib -p mdbook-paisano-preprocessor` to apply 5 suggestions)
    Finished dev [unoptimized + debuginfo] target(s) in 0.16s

Is your cargo proxied by rustup by any chance?

blaggacao · February 25, 2023, 10:04pm

I completed the above snipped with reverting the exercise, proving that $HOME/.cargo has the described side effect. But somehow RUSTUP_HOME also needed to be unset for that to have effect. With RUSTUP_HOME set, it errored out.

❯ which cargo
/nix/store/7wr49ymmiq85c58fdbq0qd33zwfmxqsx-devshell-dir/bin/cargo

In all these investigations, I’m always accessing ${default.toolchian}/bin/cargo.