NixOS & Github : Meeting Notes and Updates - January 2023

NixOS & Github Meeting

Raw notes below from the meeting with Github. Kickoff announcement - NixOS & Github - Collaboration! - Announcements - NixOS Discourse

TL;DR - We shared the Github issue that tracks all the community feedback with the Github team. Github team responses in bold at the bottom. We will meet again next months to collaborate on the Nix 20th anniversary and recurringly every quarter to discuss other potential collaborations/share feedback from the NixOS community.

Attendees: Mike McQuaid, Abigail Cabunoc Mayes, Martin Woodward, Kara Sowles, Ron Efroni (@ron), Eelco Dolstra (@edolstra)

Notes

  • Introductions + Happy New Year :sparkles:

  • Downloading packages from GitHub w/ special guest Mike McQuaid :tada:

    • Moving to users getting packages directly from GitHub
      • Will this be spammy? How can we be a good citizen.
        • Downloading nix packages directly using tarball urls directly from GitHub
    • Not sure if you’ll be in a better position to download tarball or upload to packages regularly
      • Abby / Martin: to reach out to people internally to see which we’d prefer (git systems team)
        • Otherwise, continue with whatever you’re doing now
    • Tarball from GitHub - dynamically generated the first time, then cached after that. Might also be cached when you generate a release.
    • Homebrew: moving away from full clones to tarball
  • How Nix uses GitHub - we’d love to hear from you!

  • Overview of features

    • Actions, discussions, Codespaces + others if applicable
    • GitHub Codespaces · GitHub
    • Quick call next month about 20th anniversary of Nix (20th, wow)
  • Anything else?

    • Ron can do a brain dump on GitHub Projects anytime :slight_smile:
  • Next month - Projects, codespaces, followups, and 20th anniversary

Nix Community GitHub Feedback Summary

Taken from https://github.com/NixOS/nixos-foundation/issues/25.

Feature requests:

Bugs:

Action items

  • Github - check with performance people
    • Tarball vs packages
    • Git gc
  • NixOS - Keep collecting feedback and follow up on current points
    • Share info in preparation for Nix 20th

Thank you to all the community members that collaborated on this!
Special thank you to all of those that also added/voted on issues in the tracker!

10 Likes

This is the issue: Caching `/nix/store` is ok, but unable to restore it. · Issue #749 · actions/cache · GitHub . The compression and caching works fine, but when the cache action attempts to extract it, it fails because everything in the store is read only, so the action does not have permissions to put files in the extracted folders. This can be resolved by doing the extraction with sudo (and probably the compression too).

Please see the issue for additional discussion. This issue pops up in other cases as well. There are workarounds but they are slow and should not be necessary once this change is made.

There’s a lot more to do on /nix/store and caching, but this fix should make it actually possible.

3 Likes

https://github.com/actions/toolkit/pull/1253

For the short term adrenaline rush: GitHub - divnix/nix-cache-action: Cache dependencies and build outputs in GitHub Actions - forked for use with Nix

1 Like

But what would be truly awesome (and maybe even still easier): GH Cache to expose a S3 interface.

Syncing many small remote files, over and over again.

Off the top of the stack, here’s a tiny action where fetching the dependencies takes 1m40s and the actual build only 51s. Should be easy to find more dramatic examples.

I kind of regret I brought that up, because I don’t actually see how it’s actionable. The “solution” would be persistence. If you know you’re soon going to run the same task, and you know it’s likely to use most of the same store paths, and you can afford to schedule the task to the same node, and you can afford to keep the outputs, keep them?

1 Like

Could we cache the archives of the paths instead of the paths themselves, to save round trips? What’s the size of the cached item? One store path? One item within a store path?

2 Likes

Next Meeting → NixOS Foundation Update - Github Collaboration - April 12 - Development / NixOS Foundation - NixOS Discourse