And now, the last tri-weekly update!
More information about the topic and goals of this project: https://discourse.nixos.org/t/tweag-fellowship-fuzzing-nix-0 .
Previous updates can be found at:
- https://discourse.nixos.org/t/tweag-fellowship-fuzzing-nix-1 ;
- https://discourse.nixos.org/t/tweag-fellowship-fuzzing-nix-2 ;
- https://discourse.nixos.org/t/tweag-fellowship-fuzzing-nix-3 .
Another three rough weeks: contributing to OSS-fuzz was my priority, and it came with its load of struggles.
As presented in the previous update, OSS-fuzz requires projects to use their Docker image.
The intention behind this requirement is to facilitate integration, by providing build tools and settings for building.
Also, no assumptions can be made regarding the runtime environment, i.e. runtime dependencies might not be available.
For that reason, they strongly advise contributors to compile fuzzing binaries statically.
We strongly recommend static linking because it just works. 1
I hoped to be able to produce a statically linked binary using libraries packaged
Sadly, a lot of
pkgsStatic failed to build properly with
clang, and I realised it was a rather unstable package set, under heavy work…
Luckily, since my last update, OSS-fuzz undertook to upgrade their base image to
Ubuntu 20.04, which made me reconsider following their recommendation.
But soon I realised that static linking does not “just work”… As it has to be compiled first.
I faced two kinds of problems there:
- Not all
nixtransitive dependencies provide statically linked libraries (for example, it seems that
krb5, required by
curl, has dropped support for static linking 2);
- Some static libraries present in the Ubuntu repositories do not provide the symbols needed by newer versions of the dependencies.
For 1. it was necessary to compile custom version of the static dependency (
curl), to not depend on a static library that does not exist anymore; And for 2. it was necessary to compile newer versions of the libraries.
And when statically compiling an example fuzzer, using the provided compiler, with the proper set of flags finally worked, the resulting binary did nothing else than straight
SEGFAULTing; And I couldn’t figure out why.
I published my work-in-progress with this approach here: https://github.com/google/oss-fuzz/pull/6338 .
However dynamic linking can work if shared objects are included in the $OUT directory and are loaded relative to ‘$ORIGIN’, the path of the binary (see the discussion of ‘$ORIGIN’ here). 1
I then took another direction: leverage
nix to manage the dependencies, but compile with shared libraries, “extract” the binary and its dependencies (transitively) from the store to the “$OUT” directory, and patch the binary and libraries’
rpath as well as the
interpreter of the binary.
As is, it did not work in practice: the produced binary
Again, I do not know why, although I noticed that
SEGFAULTs when moved out of the store.
The resulting contribution to OSS-fuzz is not yet PR-ready, but the work-in-progress can be followed here: https://github.com/google/oss-fuzz/pull/6317 ; And it is likely here that future work will take place.
The fellowship is ending today.
Sadly, contributing to OSS-fuzz
took takes much more time than it was anticipated, but I still believe this is an effort that will bring a lot of value to the community.
In the future, I intend to pursue this project, focusing on the following:
nixintegrated to OSS-fuzz;
- Share the stuff I learned along the way (documentation, blog post(s), etc.);
- Add more targets;
- Help other community members to contribute to this effort.
I am very thankful to Tweag for the fellowship opportunity: it has been of great help to get me started on that project;
Thanks @regnat for providing a listening ear to my problems, as well as your advices.
And also, thank y’all for following the updates, and for your time sharing your knowledge.