How to get a list of all URL used in nixpkgs?

Hello,

I would like to be able to download distfiles used in nixpkgs in bulk to have them locally. I can’t find how to achieve that. I tried with nix repl to find where I could get the source file of a derivation, but it doesn’t seem available.

I tried to look on the discourse with no luck…

4 Likes

There is a https://github.com/NixOS/nixpkgs/blob/441dc5d512153039f19ef198e662e4f3dbb9fd65/maintainers/scripts/all-tarballs.nix#L6 along with a script to mirror all tarballs used by nixpkgs. Maybe it’s bits will help your use case.

Otherwise they should be extractable via urls or similar:

$ nix repl .
nix-repl> gcc.cc.src.urls
[ "https://bigsearcher.com/mirrors/gcc/releases/gcc-11.3.0/gcc-11.3.0.tar.xz" ]
1 Like

the Perl scripts looks promising!

As for the repl, I looked at a few packages and they didn’t have a .src attribute, so this doesn’t seem consistent?

So far, I found a way to generate a package list with nix search . --json | jq -r 'keys|.[]' and https://github.com/msteen/nix-prefetch can be used to output the urls of each package, unfortunately nix-prefetch also download to compute the hash, without keeping the file. I’m trying to either make it only display the url OR keep the file, but it doesn’t seem to provide any of these two features.

Ok, this got better!

to get a list of available packages (you may need to strip the prefix legacyPackages.x86_64-linux.)

# in a nixpkgs directory
nix search . --json | jq -r 'keys|.[]' > /tmp/packages.txt

to get a list of urls for a given package, this include the urls for all the dependencies required to build that package

nix-instantiate --json --eval --strict "<nixpkgs/maintainers/scripts/find-tarballs.nix>" --arg expr '(import <nixpkgs>{}).firefox' | jq '.[].url'

It’s then possible to iterate over each url to download the files.

The next step is to check that each downloaded file has the correct checksum :exploding_head:

1 Like

I found a way to

  • download a file directly in the nix store
  • not download it if already exist
  • verify the checksum

this perl script is a consumer for the previous nix-instantiate command

#!/usr/bin/env nix-shell
#! nix-shell -i perl -p perlPackages.JSON perl
use strict;
use warnings;

use JSON;
use Data::Dumper;

my $json = do { local $/; <> };
my @text = decode_json($json);
my @data = @{$text[0]};

foreach my $val (@data) {
    print "Fetching $val->{name}\n";
    my $return = `nix-instantiate --eval -E 'builtins.fetchurl { url = $val->{url} ; sha256 = "$val->{hash}"; }'`;
    print $return;
}

However it fails with url using mirror://

error: unable to download 'mirror://gnome/sources/libxml2/2.9/libxml2-2.9.14.tar.xz': Unsupported protocol (1)

while fetchurl works fine with this scheme in nixpkgs :thinking: I’m a bit puzzled here

There’s a list of mirrors here that mirror:// URLs use: https://github.com/NixOS/nixpkgs/blob/e51e218db293a83cbd134740fbb0ae3e818ecd7c/pkgs/build-support/fetchurl/mirrors.nix

I found this file, but I can’t find how to make use of the mirrors in a command line.

It’s already done within nixpkgs, so there must be a way to achieve it using nix-instantiate, but I can’t figure it out :frowning:

I made a very simplistic project that work. For the mirror:// I used a very ugly workaround :grimacing:

It kinda works but for some reasons, the find-tarballs.nix code seems to miss some patches files.

I downloaded all the distfiles required for nethack and it fails to build

> nix-build --option substitute false -A nethack
these 2 derivations will be built:
  /nix/store/8yzb0i8k00a82p64zli2vv90a23hhj41-81d73ce417dda6a98e2e918e06922e68b67c53f7.patch.drv
  /nix/store/snjr06jvd25v0i9367rdparq1i38lv1l-nethack-3.6.6.drv
building '/nix/store/8yzb0i8k00a82p64zli2vv90a23hhj41-81d73ce417dda6a98e2e918e06922e68b67c53f7.patch.drv'...
error checking the existence of https://tarballs.nixos.org/sha256/sha256-PX9XtJTEE3K1yg/IwIzEIT+EZWi02gU+9msrsG9ZWQY=:
curl: (6) Could not resolve host: tarballs.nixos.org

trying https://github.com/NetHack/NetHack/commit/81d73ce417dda6a98e2e918e06922e68b67c53f7.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: github.com
Warning: Problem : timeout. Will retry in 1 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: github.com
Warning: Problem : timeout. Will retry in 2 seconds. 2 retries left.
curl: (6) Could not resolve host: github.com
Warning: Problem : timeout. Will retry in 4 seconds. 1 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: github.com
error: cannot download 81d73ce417dda6a98e2e918e06922e68b67c53f7.patch from any mirror
error: builder for '/nix/store/8yzb0i8k00a82p64zli2vv90a23hhj41-81d73ce417dda6a98e2e918e06922e68b67c53f7.patch.drv' failed with exit code 1;
       last 10 log lines:
       >   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
       >                                  Dload  Upload   Total   Spent    Left  Speed
       >   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: github.com
       > Warning: Problem : timeout. Will retry in 1 seconds. 3 retries left.
       >   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: github.com
       > Warning: Problem : timeout. Will retry in 2 seconds. 2 retries left.
       > curl: (6) Could not resolve host: github.com
       > Warning: Problem : timeout. Will retry in 4 seconds. 1 retries left.
       >   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: github.com
       > error: cannot download 81d73ce417dda6a98e2e918e06922e68b67c53f7.patch from any mirror
       For full logs, run 'nix log /nix/store/8yzb0i8k00a82p64zli2vv90a23hhj41-81d73ce417dda6a98e2e918e06922e68b67c53f7.patch.drv'.
error: 1 dependencies of derivation '/nix/store/snjr06jvd25v0i9367rdparq1i38lv1l-nethack-3.6.6.drv' failed to build

The missing file isn’t reported by find-tarballs.nix

nix-instantiate --json --eval --strict "<nixpkgs/maintainers/scripts/find-tarballs.nix>" --arg expr "(import <nixpkgs>{}).nethack" | nix-shell -p jq --run jq | grep 81d73ce417dda6a98e2e918e06922e68b67c53f7.patch

Oh! I was using some examples which were relying on an unwrapped package. I found the urls attribute then. But now, I need to find the url for patches when there are any…

I opened find-tarballs.nix not including patches · Issue #186831 · NixOS/nixpkgs · GitHub

Really nix-instantiate is underrated! It looks to me like a way to run nix code as programs, it’s just mind blowing :exploding_head:

I have been able to replace most of the Perl logic (including the ugly mirror rewrite) with just nix-instantiate calls!)

    my $url = $val->{url};
    if ($val->{f}) {
        $return = `nix-instantiate '<nixpkgs>' -A fetchpatch --argstr url "$url" --argstr name "$val->{name}" --argstr sha256 "$val->{hash}"`;
        print $return;
    } else {
        $return = `nix-instantiate '<nixpkgs>' -A fetchurl --argstr url "$url" --argstr sha256 "$val->{hash}"`;
        print $return;
    }
2 Likes

I opened find-tarballs.nix: Include patches by rapenne-s · Pull Request #188626 · NixOS/nixpkgs · GitHub to add patches to find-tarballs.nix in an usable way.

With this PR changes, you can use solene/nixpkgs-mirror-tarballs - nixpkgs-mirror-tarballs - tildegit to easily download all sources to locally build a package.

1 Like