Trying to package the triton inference server

Help
1

I’m trying to package Nvidia’s Triton inference server and its dependencies via Nix Flakes. My current attempts are available at GitHub - SomeoneSerge/triton.nix: Building triton inference server bit by bit, via Nix

I’ll be posting updates and asking questions in this thread - I really hope for Nix community’s help. Triton’s original build is fairly complicated (with python running docker running cmake running python) and my cmake-fu and Nix-fu seem to be insufficient. I’d also be grateful to receive comments on whether I’m failing any Nix conventions/style in my derivations

  • I managed to get a successful build of microsoft/onnxruntime (no GPU, no tensorRT enabled)

  • triton-inference-server/onnxruntime_backend build succeeds but cmake install fails with

    ```
triton-onnxruntime_backend> CMake Error at cmake_install.cmake:90 (file):                                                                                                                  
triton-onnxruntime_backend>   file INSTALL cannot find "/build/source/build/onnxruntime": No such file or                                 
triton-onnxruntime_backend>   directory.
```

I had a guess that maybe `./onnxruntime` is not tracked by cmake as an output, so I tried removing `${CMAKE_CURRENT_BINARY_DIR}/onnxruntime/` [INSTALL target](https://github.com/triton-inference-server/onnxruntime_backend/blob/61c6b7479e22bce47c2295ded1dc4bfa28598c49/CMakeLists.txt#L295), but then I got a similar error for `/build/source/build/onnxruntime/bin/`
3 Likes
2

Just wanted to send some enouragement! I’m less than a day into using nix in earnest, so I can’t be much help here, but I hope you succeed!

Maybe consider starting with tensor-rt? Triton is a complex thing indeed… it wraps all of the things and does all of the things.

3

Oh hi @drewm1980 and thank you! Honestly speaking, I got fairly caught up with something else and made no progress on this issue since the last post :sweat_smile: I would love to see this done though

…starting with tensor-rt

…yeah! Just packaging the basic deployment-things in isolation from each other sounds like a reasonable start! Perhaps I find time to try it again, though I’d better still pair with someone more experienced in Nix