# Fodwatch: Automation for detecting reproducibility issues in fixed-output derivations
**URL:** https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671
**Category:** Announcements
**Created:** [June 2, 2023, 6:09pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671 "2023-06-02T18:09:41Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@raphi](https://discourse.nixos.org/u/raphi)
#### Post date: [June 2, 2023, 6:09pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/1 "2023-06-02T18:09:41Z")
The goal of this project is detecting broken fixed-output derivations (for example: upstream tarballs or entire domains bitrotting away, git tags getting force pushed, fetchurl instead of fetchpatch, etc.). It does this by forcibly rebuilding all FODs in nixpkgs with allowSubstitutes=false and disabling tarballs.n.o.
Broken FODs are currently undetected because sources are also cached at cache.n.o.
Currently, 845 out of 91025 (or 0.93%) of FODs are broken for x86\_64-linux (but keep in mind that I initially ran this in May, which was a bad month for GitHub uptime). I intend to run this somewhat regularly.
[https://tapesoftware.net/fodwatch/](https://tapesoftware.net/fodwatch/)
---
### Author:  [@ryantm](https://discourse.nixos.org/u/ryantm)
#### Post date: [June 2, 2023, 7:54pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/2 "2023-06-02T19:54:18Z")
Nice website! If it isn’t too hard it would be cool to search by maintainer too.
---
### Author:  [@raphi](https://discourse.nixos.org/u/raphi)
#### Post date: [June 12, 2023, 6:45am UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/3 "2023-06-12T06:45:41Z")
That’s possible now.
---
### Author:  [@abathur](https://discourse.nixos.org/u/abathur)
#### Post date: [June 15, 2023, 2:06pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/4 "2023-06-15T14:06:50Z")
Tangential curiosity: how hard do you think it would be from ~here to trawl through past revisions to collect and deduplicate all of the FODs before checking?
(If the cache leaves S3, I’ve wondered if there’s meaningful egress savings in seeding the new cache with all of the sources that are still reachable and unchanged.)
---
### Author:  [@raphi](https://discourse.nixos.org/u/raphi)
#### Post date: [June 15, 2023, 4:15pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/5 "2023-06-15T16:15:23Z")
If I understand correctly, you want a set of all `outPaths` of all FODs, ever (and then possibly build those). The limiting factor there is how fast you can evaluate nixpkgs, as each evaluation takes 5-10 minutes ish. And you need to potentially do it for every system that gets cached (there are darwin and linux specific packages for example. Even now this project is missing some derivations because of that).
---
### Author:  [@abathur](https://discourse.nixos.org/u/abathur)
#### Post date: [June 15, 2023, 5:11pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/6 "2023-06-15T17:11:45Z")
Yes, roughly (and you’re right that there will be a tail of system-specific FODs, especially in the packages that just download existing binaries).
I could imagine reducing the eval load by writing a script that can iterate over revisions and ~accumulate revisions that don’t overlap/conflict…
That said, as I think about the logistics, and how many machine-months it could take to eval even 1/20th of the number of commits, and about how much time the git operations alone might take, and the fact that the cache may also have sources from hydra builds in other repos–I’m hoping the problem is more tractable from the cache side 🙂
---
### Author:  [@PhilipTaron](https://discourse.nixos.org/u/PhilipTaron)
#### Post date: [June 17, 2025, 12:10pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/7 "2025-06-17T12:10:44Z")
The last run was two years ago; is there appetite for another, or is there another way of watching for FOD mismatches?
This came up again on [Feat/build deno package by aMOPel · Pull Request #407434 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/pull/407434) and having a steady source of knowledge about which FODs aren’t around any more would be of great value.
---
### Author:  [@fricklerhandwerk](https://discourse.nixos.org/u/fricklerhandwerk)
#### Post date: [June 17, 2025, 3:16pm UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/8 "2025-06-17T15:16:41Z")
> [@PhilipTaron](#):
>
> or is there another way of watching for FOD mismatches?
[https://fod-oracle.org](https://fod-oracle.org) is being developed as part of [the current Sovereign Tech Fund project](https://discourse.nixos.org/t/transforming-global-software-distribution-with-nixpkgs/64989), and the plan is to deploy it into Nixpkgs CI soon™ .
---
### Author:  [@johnrichardrinehart](https://discourse.nixos.org/u/johnrichardrinehart)
#### Post date: [July 18, 2025, 1:48am UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/9 "2025-07-18T01:48:09Z")
Cool! [https://fod-oracle.org](https://fod-oracle.org) fails to load for me (DNS resolution failure, looks like). Is that expected?
---
### Author:  [@fricklerhandwerk](https://discourse.nixos.org/u/fricklerhandwerk)
#### Post date: [July 18, 2025, 9:01am UTC](https://discourse.nixos.org/t/fodwatch-automation-for-detecting-reproducibility-issues-in-fixed-output-derivations/28671/10 "2025-07-18T09:01:25Z")
The source is at [GitHub - multivac61/fod-oracle](https://github.com/multivac61/fod-oracle). We had some last-minute issues with getting the project into production, which is why unfortunately the tool can’t be shipped as intended. I’ll write up an update with details on the main thread in the next days. @multivac61 @mic92 and I received great support from @wolfgangwalther @roberth @Conni2461 to sort things out, and we’re preparing a revised implementation plan and schedule, which would also address @tobiasBora’s [concerns](https://discourse.nixos.org/t/transforming-global-software-distribution-with-nixpkgs/64989/15) about design and efficiency.