Help building specific versions of ruby

ldeck · May 24, 2022, 7:40pm

Hi all, and @manveru @marsam in particular since you’re listed as maintainers of nixpkgs/pkgs/development/interpreters/ruby/default.nix.

There are specific revisions of ruby available in nixpkgs like ruby_2_7, ruby_3_0 and ruby_3_1. If I want an older one I can refer to Nix Package Versions to see what’s been provided in the past.

BUT, some point revisions were never part of the nixpkgs history, unfortunately. For example, ruby_2_6 seems to have gone as far as 2.6.8 but skipped 2.6.9.

So I tried to override the derivation to make 2.6.9. But this seems to fail because it needs a bundle update or something.

Building specific nodejs versions we have the nixpkgs function buildNodeJs = pkgs.callPackage <nixpkgs/pkgs/development/web/nodejs/nodejs.nix> {};. There doesn’t appear to be such a function for ruby.

So what suggestions do you have for how to build a derivation for ruby versions that weren’t released in nixpkgs? Obviously if ruby had binaries I’d use that, but a project I need to use wants 2.6.9 and nix isn’t giving that to me atm.

let
    rubyVersion = import <nixpkgs/pkgs/development/interpreters/ruby/ruby-version.nix> { inherit lib; };

  # https://lazamar.co.uk/nix-versions/?package=ruby&version=2.6.8&fullName=ruby-2.6.8&keyName=ruby_2_6&revision=2cdd608fab0af07647da29634627a42852a8c97f&channel=nixpkgs-unstable#instructions
  pkgs_ruby268 = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/2cdd608fab0af07647da29634627a42852a8c97f.tar.gz";
    sha256 = "1szv364xr25yqlljrlclv8z2lm2n1qva56ad9vd02zcmn2pimdih";
  }) {};

  ruby_2_6_8 = pkgs_ruby268.ruby_2_6;

  ruby_2_6_9 = {
    version = rubyVersion "2" "6" "9" "";
    package = ruby_2_6_8.overrideAttrs(oldAttrs: rec {
      version = ruby_2_6_9.version;
      src = builtins.fetchurl {
        url = "https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.9.tar.xz";
        sha256 = "6a041d82ae6e0f02ccb1465e620d94a7196489d8a13d6018a160da42ebc1eece";
      };
      passthru = oldAttrs.passthru // {
        version = ruby_2_6_9.version;
      };
    });

  rubies = with pkgs; [
    ruby_2_6_8
    ruby_2_6_9.package
    ruby_2_7
    ruby_3_0
    ruby_3_1

in
{
  # start pseudo code
  install each ruby into home.file="~/.rubies/ruby-<version>"
  # end pseudo code
}

Output log:

...
installing default gems from lib:   /nix/store/3c6kbcq87ih12bpkzr8phhrmkm3cipjx-ruby-2.6.9/lib/ruby/gems/2.6.0 (build_info, cache, doc, extensions, gems, plugins, specifications)
                                    bundler 1.17.2
                                    cmath 1.0.0
                                    csv 3.0.9
                                    e2mmap 0.1.0
                                    fileutils 1.1.0
                                    forwardable 1.2.0
                                    ipaddr 1.2.2
                                    irb 1.0.0
                                    logger 1.3.0
                                    matrix 0.1.0
                                    mutex_m 0.1.0
                                    ostruct 0.1.0
                                    prime 0.1.0
                                    rdoc 6.1.2.1
                                    rexml 3.1.9.1
                                    rss 0.2.7
                                    scanf 1.0.0
                                    shell 0.7
                                    sync 0.5.0
                                    thwait 0.1.0
                                    tracer 0.1.0
                                    webrick 1.4.4
installing default gems from ext:   /nix/store/3c6kbcq87ih12bpkzr8phhrmkm3cipjx-ruby-2.6.9/lib/ruby/gems/2.6.0 (build_info, cache, doc, extensions, gems, plugins, specifications)
                                    bigdecimal 1.4.1
                                    date 2.0.2
                                    dbm 1.0.0
                                    etc 1.0.1
                                    fcntl 1.0.0
                                    fiddle 1.0.0
                                    gdbm 2.0.0
                                    io-console 0.4.7
                                    json 2.1.0
                                    openssl 2.1.2
                                    psych 3.1.0
                                    sdbm 1.0.0
                                    stringio 0.0.2
                                    strscan 1.0.0
                                    zlib 1.0.0
installing bundled gems:            /nix/store/3c6kbcq87ih12bpkzr8phhrmkm3cipjx-ruby-2.6.9/lib/ruby/gems/2.6.0 (build_info, cache, doc, extensions, gems, plugins, specifications)
/private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/installer.rb:179:in `initialize': undefined method `dir_mode=' for #<String:0x00007fc12c87df08> (NoMethodError)
        from ./tool/rbinstall.rb:725:in `initialize'
        from ./tool/rbinstall.rb:878:in `new'
        from ./tool/rbinstall.rb:878:in `block (2 levels) in <main>'
        from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:768:in `block (2 levels) in each_gemspec'
        from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:767:in `each'
        from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:767:in `block in each_gemspec'
        from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:766:in `each'
        from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:766:in `each_gemspec'
        from ./tool/rbinstall.rb:870:in `block in <main>'
        from ./tool/rbinstall.rb:937:in `block in <main>'
        from ./tool/rbinstall.rb:934:in `each'
        from ./tool/rbinstall.rb:934:in `<main>'
make: *** [uncommon.mk:364: do-install-all] Error 1
error: builder for '/nix/store/1jaaicdvrhnkwbmympqm2wzfmi8lyssb-ruby-2.6.9.drv' failed with exit code 2;
       last 10 log lines:
       >     from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:768:in `block (2 levels) in each_gemspec'
       >        from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:767:in `each'
       >    from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:767:in `block in each_gemspec'
       >   from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:766:in `each'
       >    from /private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/specification.rb:766:in `each_gemspec'
       >    from ./tool/rbinstall.rb:870:in `block in <main>'
       >      from ./tool/rbinstall.rb:937:in `block in <main>'
       >      from ./tool/rbinstall.rb:934:in `each'
       >         from ./tool/rbinstall.rb:934:in `<main>'
       > make: *** [uncommon.mk:364: do-install-all] Error 1
       For full logs, run 'nix log /nix/store/1jaaicdvrhnkwbmympqm2wzfmi8lyssb-ruby-2.6.9.drv'.
error: 1 dependencies of derivation '/nix/store/wn57jk9rhm0dk20yx6z7kgv3z5yyj944-home-manager-files.drv' failed to build
error: 1 dependencies of derivation '/nix/store/z8fsxr7wmkmx6hb0bhjmbd2klf95las3-home-manager-generation.drv' failed to build

jeff-hykin · May 25, 2022, 12:04am

I’ve been working on a more detailed version of Lazamar’s tool

But even mine didn’t have 2.6.9 … which is because it never existed on nixpkgs
commit: c6b949c

However, as you can see in the screenshot (link to newer file here) there is a way to pick the version. It’ll take a bit of work but basically you can build a version yourself.

Not ideal, or fun, so hopefully in the future there will be a better method.

ldeck · May 25, 2022, 12:14am

Thanks @jeff-hykin — yes I saw that was never part of nixpkgs unfortunately… and as you can see I’ve tried overridingAttrs for the 2.6.8 derivation, supplying the source for 2.6.9 — but this runs into bundler errors.

The generic function used by the nixpkgs ruby/default.nix appears to be private, so it’s not as simple as calling it in the same way as the builtin ones do. Hence, overrideAttrs.

The trouble is that the patches for various versions may no longer align, and so on… so I’m really hoping someone has experience in building 2.6.9 and other custom versions reliably might be able to provide insights and lessons they’ve learned here.

jeff-hykin · May 25, 2022, 3:07pm

Oh I must apologize. I really dislike when people don’t read the whole post, and here I am doing just that Thank you for patiently restating the problem.

jeff-hykin · May 25, 2022, 8:02pm

Like you said about patches, I don’t think an override is going to make it work. I forked from the nixpkgs at the 2.6.8. version and then edited stuff there (link to file/branch) take a look at the commits on that branch to see what I had to change.

You can test it out in a shell with this:

nix-shell -p ruby_2_6 -I nixpkgs=https://github.com/jeff-hykin/nixpkgs/archive/b68ecd09c2a0752132546b53c2fa63c2d0d44fd5.tar.gz

(and in general just swap that^ url for any of the examples on Lazamar’s site)

ldeck · May 26, 2022, 11:07pm

Yeah, using a version of ruby that was historically available via nixpkgs is not the problem I’m trying to solve. It’s obtaining a version that was not released in nixpkgs.

jeff-hykin · May 27, 2022, 3:56am

Does this work for you though?
It should build exactly ruby 2.6.9 on all platforms
(the url is not to the nixpkgs repo)

nix-shell -p ruby_2_6 -I nixpkgs=https://github.com/jeff-hykin/nixpkgs/archive/b68ecd09c2a0752132546b53c2fa63c2d0d44fd5.tar.gz

ldeck · June 9, 2022, 8:38pm

Hey @jeff-hykin, apologies for the delayed reply, yes it does work (though, of course, being ruby it takes an age to build).

Okay, so you’re suggesting (if I’m reading between the lines) that the most viable options are these:

use or update the relevant project to a version available in nixpkgs
otherwise, fork nixpkgs and append the version required and tag the release

A question: how do the patchSets work? Where do they come from? I couldn’t find “01-fix-broken-tests-caused-by-ad” in the repo for example.

So how do you know which patchsets will work?

Btw, the reason for my delayed reply is that I was able to use option 1 above on the offending project

rickhull · January 6, 2023, 2:51am

I’m very curious how to build Ruby with Nix, particularly making sure support for things like MJIT or YJIT are compiled in. Also maybe building from the latest available upstream.

Also, I noticed:

installing bundled gems:            /nix/store/3c6kbcq87ih12bpkzr8phhrmkm3cipjx-ruby-2.6.9/lib/ruby/gems/2.6.0 (build_info, cache, doc, extensions, gems, plugins, specifications)
/private/tmp/nix-build-ruby-2.6.9.drv-0/ruby-2.6.9/lib/rubygems/installer.rb:179:in `initialize': undefined method `dir_mode=' for #<String:0x00007fc12c87df08> (NoMethodError)

Particularly lib/rubygems/installer.rb:179:in initialize’: undefined method dir_mode=' for #<String:0x00007fc12c87df08> (NoMethodError)

Kinda weird error to kick all the bad stuff off.

jeff-hykin · January 22, 2023, 2:28pm

otherwise, fork nixpkgs and append the version required

I believe that^ is the brute force kinda-always-works option. However, I still don’t have enough experience to know if it’s the recommended option. There might be some method of using overlays or package arguments. Even if overlays and package arguments were used though, forking from an old nixpkgs means all the upstream dependencies for ruby worked for 2.6.8, so they probably also work for 2.6.9. If you use a new version of nixpkgs, and then overlay/add-args to get to 2.6.9 the upstream dependencies might be incompatible. Then you’d have to manually overlay/add-args for different versions of each of the upstream versions until everything worked. Which can be a very very painful process.

On the flip side, you can have the reverse problem that the forked-nixpkgs includes old dependencies that mess with, for example, a modern version of python in the same nix-shell as the old ruby 2.6.9

how do the patchSets work?

I’m not sure. The closest thing I know is that, instead of modifying the source repositories, nixpkgs maintainers often clone the source repo, make changes, save the changes as a .patch file. and Then they’ll have the .nix file download the code, apply the patch file to the downloaded code, and then try building the package. So my guess is patchSets are some kind of tool built around that idea.

bobvanderlinden · January 24, 2023, 6:51am

I’ve been working on this as well, but i took the approach of building each Ruby version against the latest nixpkgs.

It nowadays also has ci and pushes builds of every ruby version to cachix (so you don’t have to wait for ruby builds).

You can find it here: GitHub - bobvanderlinden/nixpkgs-ruby: A Nix repository with all Ruby versions being kept up-to-date automatically