Confused about dockerTools buildimage (not) including copyToRoot content

Help
1

There’s a hole in my understanding and I’m having trouble finding documents to fill it in.

I wanted to try my hand at building simple packages, and that worked out okay, until I tried to then generate a docker container from the simple packages. It creates the container just fine, but dependencies are missing. I ripped out everything specific to what I was trying to package up and I’ll demonstrate it below with just python3 and including the magic library.

If I do something like nix-shell -p python3Packages.magic then I can import magic just fine in the resulting shell. But if I build a container like the following, the resulting container cannot import magic.

{                                                                                                                                                                     
  lib,                                                                                                                                                                
  stdenv,                                                                                                                                                             
}:                                                                                                                                                                    
let                                                                                                                                                                   
pkgs = import <nixpkgs> { config = {}; overlays = []; };                                                                                                              
in pkgs.dockerTools.buildImage {                                                                                                                                      
  name = "stripped";                                                                                                                                                  
  tag = "nix";                                                                                                                                                        
  created = "now";                                                                                                                                                    
  copyToRoot = pkgs.buildEnv {                                                                                                                                        
    name = "image-root";                                                                                                                                              
    paths = [pkgs.python3Minimal pkgs.python3Packages.magic];                                                                                                         
  };                                                                                                                                                                  
  config = {                                                                                                                                                          
    cmd = [ "python" ];                                                                                                                                               
  };                                                                                                                                                                  
}

When I docker image load -i result and then docker run --rm -ti stripped:nix …

# docker run --rm -ti stripped:nix python
Python 3.11.8 (main, Feb  6 2024, 21:21:21) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import magic
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'magic'

Poking around the python shell a bit and using os.listdir I could see that the magic libraries indeed were not included in the container.

Anyone know what I’m doing wrong? Or able to piece together what I’m not understanding correctly?

2

Further exploration (tar xf $(tar tf results | grep layer)) and tar tf */layer.tar | grep magic) has revealed the shared object and modules are in the container, but python isn’t configured in a way that it can find those paths? There are full paths in the nix store for some of them, and some of them are relative to site-packages, but that (`/lib/python3.11/site-packages’) is not in the sys.path. I can manually add an instruction to my python script to add it, and then I can import it and use it, but that seems less than ideal.

What’s the right way to do this?

Edit:

https://nixos.org/manual/nixpkgs/stable/#setup-hook-python

This! How do I get this!?

3

It should probably be:

    paths = [ (pkgs.python3Minimal.withPackages (python-pkgs: [
      python-pkgs.magic
    ]))
];

Edit: The above didn’t work but this works but I don’t know why the minimal version doesn’t work:

    paths = [ (pkgs.python3.withPackages (python-pkgs: [
      python-pkgs.magic
    ]))
];
1 Like
4

It’s unfortunate that it doesn’t work with the minimal, the container is at 25M using minimal, and 116M without :frowning:

5

Let me know if you find out how to make it work.

6

Interestingly, the value in NIX_PYTHONPATH would make it all work if it would just get exported to PYTHONPATH, but my attempts at adding an env to the config for the container have all failed from within the .nix file.

But, for clarity, the minimal built image can do the following…

import os
import sys
sys.path.append(os.environ['NIX_PYTHONPATH'])
import magic
# no errors here...