Nixpkgs failure dashboard

Announcements
1

Hi! :‌D

Finding and searching through logs on Hydra can be a bit of a friction point, so I built this to provide a simpler interface to the logs.

You may know that I’ve written a small tool to collect hydra failures into CSVs, nixpkgs-failure-notify. While this has been useful, I wanted to grab the actual build logs behind each failure, but I did not feel too good about scrapping hydra to download them periodically.

image
image688×232 15.6 KB

Instead I started building that list on my own server, as an little experiment. Turns out that having a folder with 1500 build-logs alone was not the great, so i made a very rudimentary attempt at sorting them:

image
image323×202 12.5 KB

Fast forward a few month, I wanted to make a little project to clear my mind of all the stressful exam i just passed. I thought of that list i gather that was rotting on my computer, what if i pushed that little sort-stuff.py a bit more and made it public for anyone to take a look?

And here we are:

image
image1853×949 210 KB

Now it does build a good chunk of nixpkgs (master), almost all packages tags that are exposed by recursionMode = "hydra", and builds on x86_64-linux. Note that i use substituters and a timeout of ~30s per derivation to make it manageable to build within a bit more than a day. I’m not exactly sure on which frequency i will put the job on as i got some more experimentation to do with it, probably 2 days to once a week.

It hit 105952 attribute paths in the last iteration, out of which 22131 where not building on my system, and 3852 were timeouts. After the classification and the filtering out of irrelevant errors (e.g. incompatible system, file required, …), I got down to a ~1300 failures list that seems exploitable.

There is a lot of I low-hanging fruits!

I also refined the grouping of failures, and it works better than i expected! You might also noticed that some of them are tracking issues (cmake minimum version or boost 1.89). If you have an idea for a new group, feel free to test with the grep mode, and open an pr to add it.

This is not a big project per say, but it’s nice to revise a bit my front end skill and trying to use the branding guidelines :‌o

40 Likes
2

The codebase is very focused and easy to grasp, both the Python and the React parts. I also love the simplicity of the UI that still looks appealing. The categorization seems very handy to help maintainers in picking out failing packages that fall in their area of expertise.

Thanks for sharing!

3 Likes
3

Thanks! There is still a bit of work required to make it pretty on the react side, mostly splitting that App.tsx into multiple files :stuck_out_tongue:
I want to also move the build script into a language i am more comfortable with, probably python :eyes:

1 Like
4

nevermind it’s back online :smiley:

1 Like
5

Why Hydra? You download through the Fastly CDN from S3, right? (once you know the .drv path) Example: https://cache.nixos.org/log/vwz93ribgj2krkjwh601jkrgsbghg9fr-libsecret-0.21.7.drv

I think that’s also what nix-store -l or nix log does (after an optional eval step to get the .drv path).

5 Likes
6

@vcunat OMG I DIDN’T THINK OF THAT :‌O

2 Likes
7

Good work @Sigmanificient ! I’m happy to see more tooling around this stuff popping up–I’ve got something on the way myself.

Any fun stories or annoying parts during its development?

1 Like
8

Well i definitely encountered some logs i have never seen before, that’s for sure!

error: executing SQLite statement 'insert or replace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (1, '85kh065hlyhwb1mw5garfvx9jf2pg2hq', 'samim-fonts-4.0.5', 'nar/1qsi6f986mnl944lk9mb88v46m2widq3hfkynks6ys9wkwgmqvfn.nar.xz', 'xz', 'sha256:1qsi6f986mnl944lk9mb88v46m2widq3hfkynks6ys9wkwgmqvfn', 103832, 'sha256:0mwa58g48fm9vsnx2sn9f3rimf1w0a9f1hg9g1sn32sajskcnviw', 829512, '', 'l4j99z90vvdrx2443dzhgwbz0i28jkv0-samim-fonts-4.0.5.drv', 'cache.nixos.org-1:COei06uuD0z156OZFdVwpzFFTnU8aII+YYvTI1+euay6SCDlLEITib3UiI/QBMDJ5Re4/780nOfae5xokLIjCQ==', '', 1774600455, 1)': database or disk is full, database or disk is full (in '/root/.cache/nix/binary-cache-v6.sqlite')

Then i tried to collect-garbage a bit too aggressively (by that i mean that i wasn’t really locking the other builds). I corrupted my server store and nix-store --verify --check-contents --repair was not able to fix it :sweat_smile:

checking path existence...
path '/nix/store/w5p8k5b2kaipbad1hqa72y8bc7hmpij4-lua-arbitrary-1.0.1.2.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/w5p8k5b2kaipbad1hqa72y8bc7hmpij4-lua-arbitrary-1.0.1.2.drv'

@ledettwy helped me to get it back running thanksfully

1 Like
9

Just relased 1.0.0 (first tag)

Thanks to thunze, tonybanters & Hythera (@hythera) for their prs :‌D

2 Likes