Issues with nix reproducibility on MacOS, trying to build nokogiri (ruby): error: unknown warning option

Hello,

I got tired of running into ruby version and dependency errors every year or so when building my Jekyll-based blog on MacOS (most dependencies via homebrew), so I thought this sounded like a perfect fit for my first useful project with nix.

I thought a flake would be the best way to make sure everything is totally reproducible, so I put things together into this flake which I I’ve been running happily via nix develop for the last couple of months:

{
  description = "Reproducible setup for n8henrie.com via GitHub Pages";
  inputs = {
    # ruby 2.7.3
    # https://pages.github.com/versions/
    # https://lazamar.co.uk/nix-versions/?channel=nixpkgs-unstable&package=ruby
    nixpkgs.url = "https://github.com/NixOS/nixpkgs/archive/860b56be91fb874d48e23a950815969a7b832fbc.tar.gz";
  };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      gems = pkgs.bundlerEnv {
        ruby = pkgs.ruby;
        name = "n8henrie.com";
        gemdir = ./.;

        gemConfig.nokogiri = attrs: {
          buildInputs = [ pkgs.zlib ];
        };
      };
    in
    {
      devShell.${system} =
        let
          pkgs = import nixpkgs { inherit system; };
        in
        with pkgs;
        mkShell {
          buildInputs = [
            bundix
            gems
            libffi
            pkgconfig
            ruby
          ];
          shellHook = ''
            export LANG="en_US.UTF-8"
            make develop
          '';
        };
    };
}

Unfortunately yesterday I tried to work on a new blog post (in part about setting up a nixos VM on my M1 Mac) and ran into errors, without changing anything in my flake (the dirty git repo was from the new blog post itself):

$ nix develop
warning: Git tree '/Users/n8henrie/git/n8henrie.com' is dirty
error: builder for '/nix/store/g73www98mimm1xs1zy4lbdbmcinjlv2x-ruby2.7.3-nokogiri-1.13.8.drv' failed with exit code 1;
       last 10 log lines:
       >      from extconf.rb:774:in `<main>'
       >
       > To see why this extension failed to compile, please check the mkmf.log which can be found here:
       >
       >   /nix/store/brhgiv7z1xhv2w3ghgc4vkb2scgy5pvl-ruby2.7.3-nokogiri-1.13.8/lib/ruby/gems/2.7.0/extensions/arm64-darwin-20/2.7.0/nokogiri-1.13.8/mkmf.log
       >
       > extconf failed, exit code 1
       >
       > Gem files will remain installed in /nix/store/brhgiv7z1xhv2w3ghgc4vkb2scgy5pvl-ruby2.7.3-nokogiri-1.13.8/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.8 for inspection.
       > Results logged to /nix/store/brhgiv7z1xhv2w3ghgc4vkb2scgy5pvl-ruby2.7.3-nokogiri-1.13.8/lib/ruby/gems/2.7.0/extensions/arm64-darwin-20/2.7.0/nokogiri-1.13.8/gem_make.out
       For full logs, run 'nix log /nix/store/g73www98mimm1xs1zy4lbdbmcinjlv2x-ruby2.7.3-nokogiri-1.13.8.drv'.
error: 1 dependencies of derivation '/nix/store/yybx749z1v3gby0c2kay06gff30s6s5g-n8henrie.com.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ki4v28h21m2nzx1cdin3al1izhb5ghb7-nix-shell-env.drv' failed to build

Reading through mkmf.log, this looks like the error:

error: unknown warning option '-Werror=unused-command-line-argument-hard-error-in-future'; did you mean '-Werror=unused -command-line-argument'? [-Werror,-Wunknown-warning-option]

Searching for related issues, I see a few jekyll issues on nixpkgs, some questionably related issues on bundix, but nothing that looks like this error specifically.

This looks to be the same error: Unknown warning option '-Werror=unused-command-line-argument-hard-error-in-future' · Issue #1258 · sparklemotion/nokogiri · GitHub but looks like the root cause is changes to clang and Xcode (which there was an XCode update recently)… but I thought nix was the solution to these kinds of “dependencies changing underneath me” errors, right? I thought “something must be polluting my nix environment, otherwise how could this happen?” so I tried nix develop -i – same error. Then I remembered how even nix develop -i was not really as pure as nix-shell --pure, so I converted the flake into a default.nix and ran it in nix-shell --pure… same issue.

So am I totally mistaken about nix and reproducibility? I thought that the entire dependency chain would be built into the nix store and using a flake with a pinned nixpks should give me something that was virtually guaranteed to work consistently. Instead, I was only able to use it a few times before I’m left feeling like my old approach was much simpler and no less resistant to breakage. What did I misunderstand here? I assume the issue is something about MacOS?

Does this flake as-is work on your x86_64 Mac, but not on your aarch64 one? Or has it worked on your aarch64 one before now?

I suspect this is because builds on Darwin aren’t entirely pure. We use Libsystem impurely for instance, so if you’ve had macOS updates in the meanwhile, that can have changed under you.

Also, as a further caveat, unless the entire dependency stack is built reproducibly you cannot have a guarantee that your project will build reproducibly. Nix can only do so much until we have fully reproducible software stacks.

I no longer have an x86_64 Mac; I wrote this originally on aarch64-darwin.

We use Libsystem impurely for instance, so if you’ve had macOS updates in the meanwhile, that can have changed under you.

Maybe something changed here with the Xcode update I referred to. No major OS upgrades, I don’t recall when the last minor or patch was.

Nix can only do so much until we have fully reproducible software stacks.

This looks like relevant reading: https://daiderd.com/2020/06/25/nix-and-libsystem.html

I don’t think the clang error is the problem. When I attempted to build your flake, it said it couldn’t find mini_portile2. I made some changes and was able to get it to build, but it gives an error regarding concurrent-ruby-1.10. I also had to hand edit the lockfile to remove the platform-specific version of nokogiri (not sure if related to Add support for platform-dependant pre-compiled gems by lavoiesl · Pull Request #68 · nix-community/bundix · GitHub).

diff --git a/Gemfile b/Gemfile
index 8d0cbff..a5e316f 100644
--- a/Gemfile
+++ b/Gemfile
@@ -3,3 +3,4 @@ gem 'github-pages', group: :jekyll_plugins
 gem 'guard'
 gem 'guard-livereload'
 gem 'html-proofer'
+gem 'mini_portile2', '~> 2.8.0'
diff --git a/Gemfile.lock b/Gemfile.lock
index 48aad2f..5528906 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -234,6 +234,7 @@ GEM
     lumberjack (1.2.8)
     mercenary (0.3.6)
     method_source (1.0.0)
+    mini_portile2 (2.8.0)
     minima (2.5.1)
       jekyll (>= 3.5, < 5.0)
       jekyll-feed (~> 0.9)
@@ -241,7 +242,7 @@ GEM
     minitest (5.16.3)
     multi_json (1.15.0)
     nenv (0.3.0)
-    nokogiri (1.13.8-arm64-darwin)
+    nokogiri (1.13.8)
       racc (~> 1.4)
     notiffany (0.1.3)
       nenv (~> 0.1)
@@ -300,6 +301,7 @@ DEPENDENCIES
   guard
   guard-livereload
   html-proofer
+  mini_portile2 (~> 2.8.0)
 
 BUNDLED WITH
    2.3.20
diff --git a/flake.nix b/flake.nix
index 4ff5409..f210f53 100644
--- a/flake.nix
+++ b/flake.nix
@@ -16,8 +16,24 @@
         name = "n8henrie.com";
         gemdir = ./.;
 
+        gemConfig.ffi = attrs: {
+          buildInputs = [ pkgs.libffi ];
+        };
         gemConfig.nokogiri = attrs: {
-          buildInputs = [ pkgs.zlib ];
+          buildFlags = with pkgs; [
+            "--use-system-libraries"
+            "--with-zlib-dir=${zlib.dev}"
+            "--with-zlib-lib=${zlib.out}/lib"
+            "--with-zlib-include=${zlib.dev}/include"
+            "--with-xml2-lib=${libxml2.out}/lib"
+            "--with-xml2-include=${libxml2.dev}/include/libxml2"
+            "--with-xslt-lib=${libxslt.out}/lib"
+            "--with-xslt-include=${libxslt.dev}/include"
+            "--with-exslt-lib=${libxslt.out}/lib"
+            "--with-exslt-include=${libxslt.dev}/include"
+            "--with-iconv-dir=${libiconv}"
+          ];
+          dependencies = attrs.dependencies ++ ["mini_portile2"];
         };
       };
     in
@@ -31,8 +47,6 @@
           buildInputs = [
             bundix
             gems
-            libffi
-            pkgconfig
             ruby
           ];
           shellHook = ''
diff --git a/gemset.nix b/gemset.nix
index 7d10656..5af7f17 100644
--- a/gemset.nix
+++ b/gemset.nix
@@ -771,6 +771,16 @@
     };
     version = "1.0.0";
   };
+  mini_portile2 = {
+    groups = ["default"];
+    platforms = [];
+    source = {
+      remotes = ["https://rubygems.org"];
+      sha256 = "0rapl1sfmfi3bfr68da4ca16yhc0pp93vjwkj7y3rdqrzy3b41hy";
+      type = "gem";
+    };
+    version = "2.8.0";
+  };
   minima = {
     dependencies = ["jekyll" "jekyll-feed" "jekyll-seo-tag"];
     groups = ["default" "jekyll_plugins"];

Thanks for your input

I’m seeing that now as well, in nokogiri-1.13.8/gem_make.out. Why would that suddenly change? I didn’t modify or change my

Also, I’m still seeing this in nokogiri-1.13.8/mkmf.log:

error: unknown warning option '-Werror=unused-command-line-argument-hard-error-in-future'; did you mean '-Werror=unused-command-line-argument'? [-Werror,-Wunknown-warning-option]

A few related issues:

Still unclear to me why this would suddenly start happening.

I was able to work around the issue with BUNDLE_FORCE_RUBY_PLATFORM=true nix run nixpkgs#bundix -- --lock. I can apparently configure this locally in something like .bundle/config; is there some way for me to instead put this in my flake?

I think this is the issue: bundix doesn't like architecture-specific gems · Issue #71 · nix-community/bundix · GitHub. I don’t know why it worked before. It’s possible the path had the binary gem before you created the flake (e.g., because it picked up an impurity from your environment that allowed it to build), the path got GCed, so now it’s failing because flakes are more strict about purity.

If you want to export BUNDLE_FORCE_RUBY_PLATFORM from your flake, you can add it as an attribute on the attrset passed to mkShell. That will make it available in the dev shell’s environment. However, that didn’t work for me. I had to use the workaround specified in the issue I linked. That did work. It forces Bundix to download and install the platform-specific gem.

diff --git a/flake.nix b/flake.nix
index 4ff5409..29a6662 100644
--- a/flake.nix
+++ b/flake.nix
@@ -17,6 +17,7 @@
         gemdir = ./.;
 
         gemConfig.nokogiri = attrs: {
+          version = attrs.version + "-arm64-darwin";
           buildInputs = [ pkgs.zlib ];
         };
       };

Regarding the error in mkmf.log, I think that’s expected. Nokogiri is using append_cflags in its extconf.rb to check if your compiler supports that flag. If it fails, which is what happens with the clang in Darwin’s stdenv, the flag is not appended. There would only be a problem if it went ahead and used the flag anyway in the built itself.

Thanks for such a thorough response, I really appreciate your time.

Huh, so any environment variable can just be passed in as an attribute. I hadn’t picked up on that. It looks like there is an example here: https://nixos.org/manual/nixpkgs/stable/#notes-on-environment-variables-in-android-projects. Cool!

I don’t think that will work however, as it needs to be set for the bundix --lock step (which generates the gemset.nix, not for my nix develop step which happens later. I suppose I could turn that step into a dependency for the devShell.

Maybe this is why we are getting different results here – how are you running bundix? I’m just running, in a regular shell, nix run nixpkgs#bundix -- --lock. Without that environment variable set, it’s trying to use the platform-specific nokogiri, which does not depend on mini_portile2 (Native gems: ensure that mini_portile2 is not a dependency · Issue #2078 · sparklemotion/nokogiri · GitHub). From my Gemfile.lock (without BUNDLE_FORCE_RUBY_PLATFORM):

...
  nenv (0.3.0)
    nokogiri (1.13.8-arm64-darwin)
      racc (~> 1.4)
    notiffany (0.1.3)
...

As opposed to with BUNDLE_FORCE_RUBY_PLATFORM=true:

...
  nenv (0.3.0)
    nokogiri (1.13.8)
      mini_portile2 (~> 2.8.0)
      racc (~> 1.4)
    notiffany (0.1.3)
...

Related: tapioca sync fails because of nokogiri native gem having different dependencies than compiled gem · Issue #208 · Shopify/tapioca · GitHub

So it seems that:

  • bundix is correctly putting the platform-specific gem into Gemfile.lock
  • bundlerEnv is not requesting the platform-specific nokogiri from rubygems, which results in a missing dependency (which is only required for the non-platform-specific case).

My workaround of forcing bundle to generate a Gemfile.lock with the non-platform-specific nogokiri forces the “missing” dependency to be installed.

EDIT:

Actually, looking at gemset.nix, it looks like bundix might be putting the platform-specific dependency into Gemfile.lock but not into gemset.nix:

 nokogiri = {
    dependencies = [ "racc" ];
    groups = [ "default" "jekyll_plugins" ];
    platforms = [ ];
    source = {
      remotes = [ "https://rubygems.org" ];
      sha256 = "0g7axlq2y6gzmixzzzhw3fn6nhrhg469jj8gfr7gs8igiclpkhkr";
      type = "gem";
    };
    version = "1.13.8";
  };

I tried BUNDLE_FORCE_RUBY_PLATFORM=true nix run nixpkgs#bundix -- --lock with and without the impure. I think I even tried nix shell nixpkgs#bundix and then ran it. Now that I think about it, I may have misunderstood what I was expecting it to do. Your additional explanation (regarding what went into the lockfile) was helpful.

I think that’s what the issue I linked is about and why my workaround of changing the name and hash works.

Of the various solutions, I like the one that builds from source most (because it lets you control the native dependencies). Unfortunately, Nokogiri defaults to vendoring some of its dependencies, so you need to pass the extra buildFlags to make it use them from nixpkgs.

1 Like