Creating a derivation from AWS S3 files

Hello all,

I’m trying to create a derivation, the source files for which are privately hosted (currently, on AWS). I’m trying to put together a working example which looks something like this:

stdenv.mkDerivation rec {
  name = "s3-test";
  src = builtins.fetchurl {
    url = "s3://nix-s3-test/s3Test.cpp";
  };
  dontUnpack = true;
  ...
}

I’m not actually sure if this is even meant to work - I was googling a bit and couldn’t really figure out if this is a normal thing to do - but it at least gives me an error which suggests that Nix didn’t think I was out of my mind for trying:

error: AWS error fetching 's3Test.cpp': Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
(use '--show-trace' to show detailed location information)

According to the internet, this error mostly arises when you’ve got the wrong region set for AWS. This isn’t the case for my AWS CLI - ie, I successfully uploaded the file with aws s3 cp - but is there perhaps somewhere in my nix config that I should specify the region?

Thanks for any help!

1 Like

The commit that added the feature has some info: Add support for s3:// URIs · NixOS/nix@9ff9c3f · GitHub

But it seems to just confirm it should work if your CLI does since it looks in the normal places. Maybe try explicitly setting AWS_DEFAULT_REGION env var?

1 Like

Thanks, that’s helpful. I suspect the problem might be with this bit:

    auto res = make_ref<Aws::Client::ClientConfiguration>();
    res->region = Aws::Region::US_EAST_1; // FIXME: make configurable

Since I’m on the Asia northeast server.

1 Like

Actually, that looks like it’s been fixed now - download.cc seems to have been replaced by filetransfer.cc, which has

std::string region = get(params, "region").value_or(Aws::Region::US_EAST_1);
1 Like

I think this is a bug with the nix code - at the moment the process for capturing the region is (from filetransfer.cc):

auto [bucketName, key, params] = parseS3Uri(request.uri);

std::string profile = get(params, "profile").value_or("");
std::string region = get(params, "region").value_or(Aws::Region::US_EAST_1);

parseS3Uri calls splitUriAndParams(), which looks for http-style query strings and parses them with decodeQuery, but I don’t think AWS uses that kind of syntax in its URIs. It also seems that AWS URIs never specify the region - the one I am looking at is, eg, s3://nix-s3-test/s3Test.cpp.

So as far as I can see, it’s not possible for params to contain the region, and therefore it will always fall back to us-east-1 via the .value_or(Aws::Region::US_EAST_1) bit. I think the region has to come from the querying aws cli, eg via aws configure get region, rather than the URI.

I’d be happy to try a pull request if a solution for this can be found. It seems like region is a required argument for Aws::S3::S3Client::GetObject, and it will default to the user’s default region if it is not passed, as shown here: Code examples for Amazon ECR using AWS SDKs - AWS SDK Code Examples.

Defaulting to the user’s region would not be helpful in this circumstance. How about:
a) the S3 API has a GetBucketLocation function that returns the region of a particular bucket, and can be called from the bucket URI alone, though it seems like explicit permissions have to be granted for anyone apart from the bucket owner to use it; could still be tried though. And/or
b) create a fetchFromS3 fetcher that could take a region parameter?
At the very least it would be nice to have a good error message for this, AWS’s one is rather obtuse and Nix is just passing that through to me.

In the unlikely event that anyone else has found themselves confronting this rather niche conundrum (trying to make a derivation, from something hosted in an AWS S3 bucket that is not on the default us-east-1 region), it can be done by putting an http-style query at the end of the URL - like this:

  src = builtins.fetchurl {
    url = "s3://nix-s3-test/s3Test.cpp?region=ap-northeast-1";
  };

Note that that is not actually a valid s3 URI, eg if you tried
aws s3 cp s3://nix-s3-test/s3Test.cpp?region=ap-northeast-1 ./
it wouldn’t work. But it’s what Nix wants.

Not sure why this didn’t occur to me at the time.

4 Likes