Different Behavior of PIL in Docker vs Nix

Help
1

Hi everyone. I’m currently trying to port a big Python project from a Docker development setup to Nix. In the process I’ve run into a weird issue, and I think I need help.

We depend on Pillow (v9.5.0) for calculating the size of a block of text. For that, we use ImageFont.getsize.

But this function call has different results depending on whether we use Nix as an environment or install pillow in a “normal” debian environment.

I’ve created a reduced example for this behavior.

I’m assuming this is because some library is different somewhere down the line, but as far as I can tell at least libfreetype is of the same version. Checking for identity of the libraries is not that easy because the Nix libraries have been treated with patchElf.

If anyone knows what knobs I can turn to fix this or at least get more information about the underlying problem I would be very greatful.

1 Like
2

I did find one difference. poetry2nix seems to choose Pillow-9.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl while pip in the docker container chooses Pillow-9.5.0-cp310-cp310-manylinux_2_28_x86_64.whl

No idea why yet, or if this influences the difference in any way.

3

layout_engine seems to make the difference:

layout engine docker.io/python flake.nix
basic (125, 13) (125, 13)
Raqm (121, 13)

UserWarning: Raqm layout was requested, but Raqm is not available. Falling back to basic layout.

(125, 13)
1 Like
4

Oh, wow, I don’t think I would have found that anytime soon, thanks a lot!
Now I just have to think of a way to get Raqm into the nix variant (aside from wanting to change as little as possible I think we will need RAQM for rtl text)

5

Just guessing by imitating the existing packaging, this override allows it to work for me:

python3Packages.pillow.overridePythonAttrs (attrs: {
  preConfigure = attrs.preConfigure + ''
    export LDFLAGS="$LDFLAGS -L${libraqm}/lib"
    export CFLAGS="$CFLAGS -I${libraqm}/include"
  '';
})

I suspect an overlay via pythonPackagesExtensions might be the way to go but I’m not very familiar with Python packaging.

Probably also worth opening an issue in nixpkgs to either include this by default or make it easy to enable.

1 Like
6

That might take a bit of work still. We’re currently on Pillow 9.5 which seems to have problems building from source in poetry2nix^^
We might be able to upgrade, etc.

What I’m not quite sure about is… how does the docker version get libraqm? I tried searching for any mention of raqm in the file system and couldn’t find anything. Is it statically built into _imagingft.cpython-310-aarch64-linux-gnu.so or something? O.o

In any case, thanks so much for your help. I’ll resume the experimentation tomorrow :sweat_smile:

7

Oh I see, poetry2nix downloads the wheel from PyPI instead of using nixpkgs.

This might explain what’s happening in the Docker container as docker.io/python:3.10 does include libfribidi:

Pillow wheels since version 8.2.0 include a modified version of libraqm that loads libfribidi at runtime if it is installed.

I’m not sure how to provide that for the poetry2nix environment.

1 Like
8

Good catch!
I’m currently trying messing around with patchelf and adding fribidi to the propagated build inputs, but so far no luck.
If I understand this correctly, technically all that would need to happen is dlopen needs to find libfribidi.so. I think the two things in the way of that is having libfribidi.so in the search path and teaching dlopen to actually search for it (which I believe is turned off by nix normally?)

9

I somwhat gave up on getting the RPATH of the raqm wrapper to work. Something deletes my modified RPATH and I haven’t found out what.

But luckily I managed to upgrade the dependencies of our project to a point where libraqm is in nixpkgs.

This means I was able to do the following override:

pillow = (super.pillow.override {
  preferWheel = false;
}).overridePythonAttrs (old: {
  # Needed for RAQM support.
  # See https://discourse.nixos.org/t/different-behavior-of-pil-in-docker-vs-nix/37064/5
  preConfigure = (old.preConfigure or "") + ''
    export LDFLAGS="$LDFLAGS -L${libraqm}/lib"
    export CFLAGS="$CFLAGS -I${libraqm}/include"
  '';
  # Needed to build pillow from source.
  # See:
  # https://github.com/nix-community/poetry2nix/issues/1139
  # https://github.com/python-pillow/Pillow/pull/7069
  # https://github.com/sigprof/nix-devenv-qmk/commit/e6350e6df8e21aeb99c1abc96676a7a05d860a18
  patches = (old.patches or [ ]) ++ lib.optionals (old.version == "9.5.0") [
    (fetchpatch {
      url = "https://github.com/python-pillow/Pillow/commit/0ec0a89ead648793812e11739e2a5d70738c6be5.diff";
      sha256 = "sha256-rZfk+OXZU6xBpoumIW30E80gRsox/Goa3hMDxBUkTY0=";
    })
  ];
});

Now our dependency properly formats text with RAQM.
Thanks again for all your help @_Andrew
I will mark this as the “solution” so future people with the same problem have an easy time finding it, but it’s really none of my work^^