While mounting Ceph filesystem, got a modprobe not found

Help
1

I’m in the process of trying to get a Ceph installation working. For the most part, I have it down, but then I hit a roadblock when I tried to mount the cephfs file system. Eventually, I want to do this at boot for my Plex server but I can’t even mount it manually, through fileSystems, or systemd mount.

In all cases, when I try to mount, I get this error:

systemd[1]: Mounting /media...
mount[3174]: sh: line 1: modprobe: command not found
mount[3173]: mount error 22 = Invalid argument
mount[3173]: failed to load ceph kernel module (127)
systemd[1]: media.mount: Mount process exited, code=exited, status=32/n/a
systemd[1]: media.mount: Failed with result 'exit-code'.
systemd[1]: Failed to mount /media.

Looking up the code, I found this discussion which talks about how modprobe is hard-coded to not have a path but the PATH has been wiped out because of the setuid mount program which means it can’t mount it at all.

I do have the Ceph modules loaded already in the system.

> lsmod | grep ceph
ceph                  499712  0
libceph               471040  1 ceph
netfs                  49152  1 ceph
libcrc32c              16384  1 libceph
fscache               389120  1 ceph

I’m not really sure how to move forward with this. I tried a number of ways of trying to override that package to try getting in ${kmod}/bin/modprobe, but kept getting an infinite recursion so I suspect that is a poor understanding of Nix.

The various Ceph commands are working fine, so I suspect it is currently hung up on that modprobe.

If it helps, I’m working with a flake-based setup using colmena to deploy. Everything else is pretty stable.

Any help or suggestions would be greatly appreciated. My children are harassing me about not having their Plex running. :smiley:

Thank you.

2

It can’t find the modprobe command in PATH. You can either expand PATH to contain the directory if the SUID wrapper but I am not sure if ceph has even permissions to run that. You could also remove the modprobe command entirely or add a modprobe dummy command which always returns true.

3

Would you be so kind as to give me hints (or show me) of how to do this? I’ve tried a few times and nothing is working.

4

The suid wrappers are under /run/wrappers/bin. How do you mount your filesystem? The mount service is custom right?

5

Either:

  fileSystems."/mnt/media" = {
    device = "192.168.2.35:6789:/media";
    fsType = "ceph";
    options = [
      "secretfile=/etc/ceph/media.secret"
      "user=media"
    ];
  };

or

    systemd.mounts = [
      {
        type = "ceph";
        what = "192.168.2.35:6789:/";
        where = "/mnt/media";
        mountConfig = {
          Options = "user=media,secretfile=/etc/ceph/media.secret";
        };
      }
    ];
  
    systemd.automounts = [
      {
        wantedBy = ["multi-user.target"];
        automountConfig = {
          TimeoutIdleSec = "600";
        };
        where = "/mnt/media";
      }
    ];

Or just calling mount directly. I got the same behavior in all three cases.

6

FWIW, I get similar errors when I need to remount a cephfs volume by hand. (eg mount /mnt/ceph). But despite the errors, the volume does get mounted.

Comparing your config to mine, I think you maybe need to swap user for name. AFAICT that’s the only difference. If that works, my vague guess is that the actual error is that, and the shown error message is just a red herring.

1 Like
7

I tried swapping the name for user, but still got the modprobe issue. I eventually got it working last night, but I had to recompile all of ceph to change this line from “modprobe” to ${kmod}/bin/modprobe. Takes about 2-3 hours to run but it actually worked and now fileSystems, FUSE, and manually mounting are all good.

Well, I also found an issue with building the tools where it appeared to be using the wrong version of lua and I reported an issue an issue with that also.

8

Would you mind sharing how you made the override? I tried making a pr for the ${kmod} change but the build wouldn’t succeed.

1 Like
9

I ended up going with a more generic and ham-fisted approach.

    preConfigure = ''
      substituteInPlace src/common/module.c --replace "/sbin/modinfo"  "modinfo"
      substituteInPlace src/common/module.c --replace "/sbin/modprobe" "true"
      substituteInPlace src/common/module.c --replace "/bin/grep" "grep"

      # install target needs to be in PYTHONPATH for "*.pth support" check to succeed
      # set PYTHONPATH, so the build system doesn't silently skip installing ceph-volume and others
      export PYTHONPATH=${ceph-python-env}/${sitePackages}:$lib/${sitePackages}:$out/${sitePackages}
      patchShebangs src/script src/spdk src/test src/tools
    '';

I just replaced it with true instead of ${kmod}/bin/modprobe. I had a problem with the kmod also, but I couldn’t really tell what the “correct” fix was and I already had it modprobe’d, so I could just ignore the statement entirely.

10

There is a chance replacing it with /run/current-system/sw/bin/modprobe would work, I saw that on a later issue with rclone.

11

some text that the post is not empty

12

I’m sorry, I didn’t understand.

13

If Sandro didn’t have made that additional comment, the forum system would have denied the post as “empty”, as just quoting a previous post is not enough. The forum is unaware of the fact that the quote actually contains corrections.

Each post requires 20 chars or so of “fresh” content.

1 Like
14

Hi, thanks for the pointers; I’m still a nix noob so i’m still trying to make the override.

The problem is would seem is that i can’t access the ceph-python-env and sitePackages because they’re not attributes of the ceph package.

Is overrideAttrs the correct function? I could perhaps use the super variable inside the override function?

{ pkgs, config, hostname, ... }:
  let 
  ceph = pkgs.ceph.overrideAttrs (oldAttrs: { 
      ceph-python-env = oldAttrs.ceph-python-env;
      sitePackages = oldAttrs.sitePackages;
      preConfigure =''
      substituteInPlace src/common/module.c --replace "/sbin/modinfo"  "modinfo"
      substituteInPlace src/common/module.c --replace "/sbin/modprobe" "true"
      substituteInPlace src/common/module.c --replace "/bin/grep" "grep"
      # install target needs to be in PYTHONPATH for "*.pth support" check to succeed
      # set PYTHONPATH, so the build system doesn't silently skip installing ceph-volume and others
      export PYTHONPATH=${oldAttrs.ceph-python-env}/${oldAttrs.sitePackages}:$lib/${oldAttrs.sitePackages}:$out/${oldAttrs.sitePackages}
      patchShebangs src/script src/spdk src/test src/tools
    '';
    });
in 
{
  environment.systemPackages = [ ceph pkgs.ceph-client ] ;
}

Thanks for any and all help.

PS: I asked chatgpt already, no really a lot of help

15

After a lot of trail and error, a colleague helped out.

I’m trying to make my first pr trying to fix the problem. If anyone has any pointers.
https://github.com/NixOS/nixpkgs/pull/206259

override code if there are any others

{ pkgs, config, hostname, kmod, lib, ... }:
  let 
  src = builtins.fetchurl {
    url = "http://download.ceph.com/tarballs/ceph-${version}.tar.gz";
    sha256 = "16mjj6cyrpdn49ig82mmrv984vqfdf24d6i4n9sghfli8z0nj8in";
  };
  python = pkgs.python39;
  version = "17.2.5";
  getMeta = description: with lib; {
     homepage = "https://ceph.io/en/";
     inherit description;
     license = with licenses; [ lgpl21 gpl2 bsd3 mit publicDomain ];
     maintainers = with maintainers; [ adev ak johanot krav ];
     platforms = [ "x86_64-linux" "aarch64-linux" ];
   };
  ceph-common = python.pkgs.buildPythonPackage rec{
    pname = "ceph-common";
    inherit src version;

    sourceRoot = "ceph-${version}/src/python-common";

    checkInputs = [ python.pkgs.pytest ];
    propagatedBuildInputs = with python.pkgs; [ pyyaml six ];

    meta = getMeta "Ceph common module for code shared by manager modules";
  };
   ceph-python-env = python.withPackages (ps: [
        ps.sphinx
        ps.flask
        ps.cython
        ps.setuptools
        ps.virtualenv
        # Libraries needed by the python tools
        ps.Mako
        ceph-common
        ps.cherrypy
        ps.cmd2
        ps.colorama
        ps.python-dateutil
        ps.jsonpatch
        ps.pecan
        ps.prettytable
        ps.pyopenssl
        ps.pyjwt
        ps.webob
        ps.bcrypt
        ps.scipy
        ps.six
        ps.pyyaml
      ]);
  sitePackages = ceph-python-env.python.sitePackages;

  ceph = pkgs.ceph.overrideAttrs (oldAttrs: { 
      preConfigure =''
      substituteInPlace src/common/module.c --replace "/sbin/modinfo"  "modinfo"
      substituteInPlace src/common/module.c --replace "/sbin/modprobe" "${pkgs.kmod}/bin/modprobe"
      substituteInPlace src/common/module.c --replace "/bin/grep" "grep"
      # install target needs to be in PYTHONPATH for "*.pth support" check to succeed
      # set PYTHONPATH, so the build system doesn't silently skip installing ceph-volume and others
      export PYTHONPATH=${ceph-python-env}/${sitePackages}:$lib/${sitePackages}:$out/${sitePackages}
      patchShebangs src/script src/spdk src/test src/tools
    '';
    });
in 
{
  environment.systemPackages = [ ceph pkgs.ceph-client pkgs.ksmbd-tools];
}
1 Like