Tweag Fellowship: Fuzzing Nix #4

1

And now, the last tri-weekly update!

More information about the topic and goals of this project: https://discourse.nixos.org/t/tweag-fellowship-fuzzing-nix-0 .
Previous updates can be found at:

Progress?

Another three rough weeks: contributing to OSS-fuzz was my priority, and it came with its load of struggles.

Building and runtime environment

As presented in the previous update, OSS-fuzz requires projects to use their Docker image.
The intention behind this requirement is to facilitate integration, by providing build tools and settings for building.

Also, no assumptions can be made regarding the runtime environment, i.e. runtime dependencies might not be available.
For that reason, they strongly advise contributors to compile fuzzing binaries statically.

We strongly recommend static linking because it just works. 1

Static compilation, using nix

After a bit of exploration, I decided to completely ignore the environment they provide, and installed nix in the container to leverage the flake.nix to manage the dependencies.

I hoped to be able to produce a statically linked binary using libraries packaged nixpkgs.
Sadly, a lot of pkgsStatic failed to build properly with clang, and I realised it was a rather unstable package set, under heavy work…

Static compilation with Ubuntu packages

Luckily, since my last update, OSS-fuzz undertook to upgrade their base image to Ubuntu 20.04, which made me reconsider following their recommendation.

But soon I realised that static linking does not “just work”… As it has to be compiled first.

I faced two kinds of problems there:

  1. Not all nix transitive dependencies provide statically linked libraries (for example, it seems that krb5, required by curl, has dropped support for static linking 2);
  2. Some static libraries present in the Ubuntu repositories do not provide the symbols needed by newer versions of the dependencies.

For 1. it was necessary to compile custom version of the static dependency (curl), to not depend on a static library that does not exist anymore; And for 2. it was necessary to compile newer versions of the libraries.

And when statically compiling an example fuzzer, using the provided compiler, with the proper set of flags finally worked, the resulting binary did nothing else than straight SEGFAULTing; And I couldn’t figure out why.

I published my work-in-progress with this approach here: [WIP] Add `nix` to the projects (with Ubuntu packages) by Pamplemousse ¡ Pull Request #6338 ¡ google/oss-fuzz ¡ GitHub .

Binary with dynamic libraries, using nix

However dynamic linking can work if shared objects are included in the $OUT directory and are loaded relative to ‘$ORIGIN’, the path of the binary (see the discussion of ‘$ORIGIN’ here). 1

I then took another direction: leverage nix to manage the dependencies, but compile with shared libraries, “extract” the binary and its dependencies (transitively) from the store to the “$OUT” directory, and patch the binary and libraries’ rpath as well as the interpreter of the binary.

As is, it did not work in practice: the produced binary SEGFAULTs;
Again, I do not know why, although I noticed that ld-linux.so itself SEGFAULTs when moved out of the store.

Then, @thufschmitt advised me to try out exodus, “a tool that makes it easy to successfully relocate Linux ELF binaries from one system to another”;
And it worked!

The resulting contribution to OSS-fuzz is not yet PR-ready, but the work-in-progress can be followed here: [WIP] Add `nix` to projects (using `nix`) by Pamplemousse ¡ Pull Request #6317 ¡ google/oss-fuzz ¡ GitHub ; And it is likely here that future work will take place.

Future plans?

The fellowship is ending today.

Sadly, contributing to OSS-fuzz took takes much more time than it was anticipated, but I still believe this is an effort that will bring a lot of value to the community.

In the future, I intend to pursue this project, focusing on the following:

  • Have nix integrated to OSS-fuzz;
  • Share the stuff I learned along the way (documentation, blog post(s), etc.);
  • Add more targets;
  • Help other community members to contribute to this effort.

Afterword

I am very thankful to Tweag for the fellowship opportunity: it has been of great help to get me started on that project;
Thanks @thufschmitt for providing a listening ear to my problems, as well as your advices.

And also, thank y’all for following the updates, and for your time sharing your knowledge.

:v:

1: Fuzzer environment | OSS-Fuzz
2: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=439039#10

12 Likes