Thank you all for the info; I believe I have figured out the performance difference. TL;DR - I believe that the optimization flags explain the difference between the nixpkgs
Python and the one I compiled within a nix-shell
. Specifically --enable-optimizations --with-lto
when compiling Python.
I think the article about Python performance posted by @knedlsepp definitely highlighted some potential things we could do. First, while investigating the PGO & LTO optimizations I had a thought; is the libc
that Nix provides, also lacking optimizations? I did a test by compiling Python outside of a nix-shell
, but using the same GCC (and by extension the same libc
?), and the results were the same as within a nix-shell
.
Digging deeper, using something like libc-bench
(https://git.musl-libc.org/cgit/libc-bench/), and compiling that inside and outside a nix-shell
provided no insights. The performance, on average, was the same between the two; and if anything GCC 10.3 (nix) was slightly faster in some cases than GCC 7.5 (ubuntu). Therefore, libc
/ gcc
are probably not suspects in this issue.
Next, a few other bugs/issues/forum threads led me to testing with Python 3.10 and 3.8, and found that compiling Python3.8 inside nix-shell
with --enable-optimizations
did not actually use PGO or LTO by default but 3.10 did use PGO.
I erroneously assumed that LTO and PGO would be enabled with the --enable-optimizations
flag; but that is not true according to https://bugs.python.org/issue28032#msg275182 you have to specify LTO directly.
However, manually specifying ./configure --enable-optimizations --with-lto
for compilation resulted in nearly the same performance of Ubuntu’s; within 3%. Additionally, I did not investigate the PGO difference I observed with 3.8 vs 3.10, but it’s probably an environment issue or user error.
can now get similar numbers inside & outside a nix-shell
when I compile Python. I appreciate all the help thus far! I’m going to run some tests to get real world numbers, before continuing to dig deeper. We may be OK with the performance penalty that the nixpkgs.python38
incurs due to the lack of optimizations. At least we understand why!
@wkral – Ubuntu 18.04 does have Spectre & Meltdown mitigations enabled on the more recent kernels; which we are using. It indeed was about a ~20% drop in perf when we enabled the mitigations.