The problem is essentially that composer places a bit of extra code into the vendor directory and that may change over multiple composer versions. The reason is that PHP’s core doesn’t have a proper autoloading infrastructure and that’s basically what composer injects here.
implement tooling to deal with the situation: either ways to discover changes in vendor and update all hashes (important for me: make sure with diffoscope that we don’t have extra chnages in there).
find a different way to approach this?
Personally, I don’t have a strong opinion on this.
And to be very blunt: I absolutely don’t see myself doing this. PHP became less and less relevant for my work over the past few years and while I’m happy to keep the interpreters up-to-date, I don’t have intrinsic motivation this kind of work.
So, we’ll essentially need somebody who’d be interested in helping out with this. Feel free to reach out here, in the issue I linked above or on Matrix.
And generally speaking, changes to vendor/composer/installed.json or InstalledVersions.php are not to be excluded, or ClassLoader.php also will likely see changes eventually if PHP introduces function autoloading or built-in classmaps.
My point is… all vendor/composer/*.* files are unstable across Composer versions, and there is nothing we can do about it. The software has to evolve and with it these files have to change.
I am not sure what you can do about it except pin Composer to a certain version and once in a while upgrade all hashes when bumping the pinned version.
I wanted to explain a use case we have in Nix/Nixpkgs and ask whether Composer could support an option to better handle reproducible builds: specifically around the generation of files under vendor/bin.
In Nix, we use something called a Fixed Output Derivation (FOD) (What are Fixed Output Derivations and why use them?) to ensure deterministic, reproducible builds.
In simple terms, a FOD is a build step whose output must be bit-identical for a given set of inputs. Nix verifies this by hashing the resulting artefacts (in this case, the vendor directory). If anything changes (even a timestamp) the build is considered invalid.
Here’s how our PHP builder currently works:
Phase 1 — FOD step
Fetch the project source.
Run composer install --no-autoload.
The --no-autoload flag prevents generation of vendor/composer/* files, which can differ between Composer versions.
Compute a fixed-output hash on the resulting vendor/ directory: ensuring the dependency tree is frozen and reproducible.
Phase 2 — Final installation step
Fetch the project source.
Copy the precomputed vendor/ directory from Phase 1 into the source tree.
Run composer dump-autoload to regenerate the vendor/composer/* files.
This model yields excellent reproducibility because the hashed content in Phase 1 is static.
However, one potential remaining source of variation is the vendor/bin/* directory, which contains proxy scripts generated for package binaries.
Note: In an earlier version of the builder (version 1), we used a custom Composer plugin to create a local composer-type repository from an existing package. That local repository acted as the FOD itself. While this approach worked, it had some limitations and was eventually replaced in version 2 by the simpler composer install --no-autoload process described above.
It would be very helpful if Composer offered a flag or option to skip generation of vendor/bin/* during the initial install (Phase 1).
Then, in Phase 2 (only once the FOD hash is fixed) we could safely generate those binaries (which might vary slightly) without affecting reproducibility.
Specifically:
We already use --no-autoload to skip autoload generation.
Is there a similar flag or configuration option to tell Composer not to create the vendor/bin proxies?
If not, would you consider adding one — perhaps --no-bin-install, or a configuration key to disable binary proxy generation?
Likewise, similar to composer dump-autoload, could a command such as composer dump-bin (or equivalent) be feasible to regenerate them later?
Would you be open to:
Confirming whether such a flag or config already exists (to suppress vendor/bin/* creation during install) ?
If not, considering adding one, or accepting a proposal to support this use case ? it would help tools like Nix cleanly separate the hash-sensitive dependency phase from the more variable final packaging phase.
Thanks a lot for your time and for Composer itself!
I’d be happy to provide more technical details about our setup if that helps clarify the use case.
I have just tested it, but it doesn’t create a vendor/ directory. I guess it download stuff in the cache folder, but the flag --no-cache is set… so basically, it does nothing.
Are you suggesting to use the cache folder for the FOD instead ?
In the meantime, I updated the PR and took into account the remarks from @Seldaek.
The phase 1 of the builder (the FOD) doesn’t generate the autoload files thanks to --no-autoload AND now also delete the bin-dir (rm -rf $(composer config bin-dir)) directory.
That would make sense. As far as I know it’s only used for storing the downloaded files that would make up the binaries. Seems like exactly what you want for the FOD.
As far as I remember, your initial Nix-Composer integration was using the cache, and I advocated against that because it relies on internal behavior and may also break in some funky ways… But up to you
There is no --no-bin-install BUT you can set composer config bin-dir /dev/null (or a random dir outside of vendor dir in case /dev/null breaks, not sure) and then undo that after the first install. That should work. Or you can simply ignore vendor/bin/ from the FOD or remove that dir before computing it?
I will wait a bit before making any further changes (changes are tedious and time consuming as I have to recalculate the hash of all the packages using the builder in nixpkgs) to let other contributors share their perspectives.
As far as I can tell, we currently have 2 possible approaches:
Option 1: Use the --download-only flag, while keeping in mind that its structure may change without notice. This doesn’t fully solve the problem. It might also be worth investigating whether Composer downloads archives (ZIPs) directly from GitHub, as that could introduce another potential source of variability.
