NixOS S3 Short Term Resolution!

Announcements
1

NixOS S3 Update & Short Term Resolution!

Rather thrilled to share some fantastic news to get us all going into the week (even if it’s a Tuesday)! :slightly_smiling_face:

As many of you know, we’ve been actively exploring several options to secure the future of our storage requirements following the end of LogicBlox’s generous sponsorship. Many of you have been rigorously contributing to this effort, which has been nothing short of amazing.

We’re excited to announce that the folks at AWS have committed to sponsoring all our storage costs for the next 12 months ($9,000/month)! This gesture not only addresses our immediate concern about the “short term” side of this effort - maintaining stability - but also buys us ample time to focus on sustainable long-term solutions without pressure.

Staying on AWS ticks all three of the boxes that we called out as key priorities for the short-term resolution: zero risk, as we aren’t changing anything beyond the account/payment, zero costs, and longevity (previous update).

This fast resolution (~14 days!) is yet another example of the power and passion of our community who, at a moment’s notice, once again rallied around this.
Enormous appreciation to the AWS Open Source Strategy & Marketing and OpenData teams for their commitment, to Mila Zhou and to everyone else involved in this effort, who worked at a fast pace alongside us to understand our needs and provide support. A special thank you to Bernardo Meurer Costa (@lovesegfault) who has been working closely across teams to move this forward!

Having this short-term issue resolved allows our Infra team to research and develop long-term solutions with greater focus and less urgency. We will continue to explore various options, from data optimization strategies to potential partnerships with other providers. If you’re interested in joining those efforts, please let the Infra team know or jump into the thread.

AWS Sponsorship Details & Next Steps

  • Credits of $9,000 / month have been approved. The AWS team will evaluate if these credits will be provided monthly or in a lump sum format (quarterly/yearly)
  • AWS open source credits are approved on an annual basis; we will need to reapply in June of 2024
  • Credits can’t be used for prior months - this is fine as we are within the timeframe generously provided by LogicBlox and we also have reserves if needed
  • We’ll keep the AWS team in the loop as efforts are made around the long term resolutions and cost optimizations
  • Next Step: The foundation will work with LogicBlox to migrate the payments to the NixOS account where the credits will be available

AWS Open Data Update

We’ve also been in contact with the AWS Open Data team, who are working with us in parallel to explore a potential longer-term option. The Nix binary cache is a unique corpus of compiled software with some interesting and easy to access metadata, and as such is interesting for them to store. We had our first meeting with the team this week and:

  • Shared an overview of Nix
  • Discussed our situation and the potential for growth over the next two years
  • Explored the Open Data program

The team will follow up with us for next steps.

We look forward to keeping everyone informed and involved in our progress. In the meantime, I hope folks take a moment to celebrate this great news and have it kick off a great rest of the week.

Thank you again to everyone who contributed across all the various channels and threads!

Another special thank you to @AmineChikhaoui for constantly supporting this across Nix and LogicBlox.

Please comment/add any further topics and items!

Links:

80 Likes
2

Thanks for your work!

Did we get any assurance on egress fees? That feels like the main remaining threat — that we’ll have a huge cost if we decide to leave AWS, or if they decide next year to stop sponsoring us.

14 Likes
3

We initially split the effort between what’s needed today/short term and what’s needed for the longer term.

Short term will prioritize the matters we need to handle to have a solid resolution in place for at least the next 12 months. This will allow the Infra team and the general community to explore a longer term solution with optimizations and improvements which are out of scope for the timeframe we currently have.

Egress should definitely be something taken into consideration and part of the discussion as we explore the best options for a sustainable long term options.
There’s a few items discussing/mentioning egress here but we should add it/expand → [Long Term Strategy and Priorities] Migration of S3 Bucket Payments to Foundation · Issue #86 · NixOS/foundation (github.com)

tagging @zimbatm who’s working on the Infra side as well

1 Like
4

I understood @qyliss’s question as:

What exactly is the sponsorship deal covering?

  • Is it sponsoring explicitly the storage costs, but we pay for any egress?
  • Or is the sponsorship generally $9000/month for the NixOS project, and if the storage size was reduced (e.g. via deduplication, deletion, transfer to a cheaper storage type, or other means), could the rest of the sponsorship be spent on egress?
5 Likes
5

Want me to take a gander at the team’s S3 usage and check that there isn’t an easy optimization or two lying around?

20 Likes
6

Seems like a good idea, but I have no formal role in the NixOS project. Until someone with direct access to project stuff can help you, here’s some links to get you started. @edolsta outlined the nature of what’s in S3 and its costs in a previous thread. I also found the terraform for the cache bucket.

3 Likes
7

Any help/contribution is always welcome!

2 Likes
8

I think that would be awesome !

1 Like
9

Dm’ing you to connect :slight_smile:

2 Likes
10

The way the sponsorship is coming in is as $9,000/month of credit to the NixOS Foundation AWS account to cover storage costs. We will also be recurringly staying in sync with the AWS teams as the longer term efforts make progress. So, we will have a channel to discuss these matters as things move forward.

2 Likes
11

What about after that 12 months? WE need longevity. Running after sponsors is not suitable in my opinion.
Foundation needs to fund itself.

12

The foundation will always have to rely on donations and sponsors.

And to be honest, I do not see much of a difference between a 10k recurring monetary donation of some company feeling nix was worth it, or sponsoring the service.

What I consider important though, regardless of the payee, the buckets and the contents should be “owned” by the foundation, such that we can “just” take over they bill when the sponsor leaves, without having to consider bucket ownership transfers or similar.

Or worse: what if the sponsor who “owns” the bucket gets affected by international embargos and Amazon wasn’t allowed to continue providing services to them? What would happen to that data? Not only considering actual embargos but also other political reasons, like 18 months ago, when a lot of US companies stoped delivering to Russian companies, even before official repressions have been in charge!

6 Likes
13

That’s why this is marked as a short-term solution — and a very good one, giving time to explore and implement other options with very little immediate impact.

4 Likes
14

We’ll make a more structured post about the long-term efforts that are starting to kick off. Just want to tie up everything on this thread for the short term first.
In the meantime, there’s also the long term github issue thread → [Long Term Strategy and Priorities] Migration of S3 Bucket Payments to Foundation · Issue #86 · NixOS/foundation (github.com)

3 Likes
15

Hi.
Just discovered Nix and NixOS and I love it, but seen this notes of S3, makes me wondering if is safe to switch to NixOS for now and later on.

Sorry if I misunderstood something.

16

Absolutely safe! As a community, we’re committed to full transparency - which is why you’ll see us discussing everything from S3 buckets to financials on a regular basis. It’s a key principle here to keep everything open, even when they go beyond the realm of exciting announcements.

Across the critical areas of our community, we have strategies and buffers in place that prioritize continuous/long term reliability and resilience of the Nix ecosystem.

Sharing some context on this with the S3 topic as an example:

We’ve maintained an open line of communication with the long-standing sponsors of the S3, which allowed us to receive very gracious notice (almost three months) of the need to transition the sponsorship. In addition, we’ve always included this sponsorship in our treasury/financial buffer at the NixOS Foundation. This means that even if the sponsorship were abruptly terminated, we had sufficient funds to sustain the ecosystem for several months while finding a resolution. Additionally, we have contingency plans and strategic partnerships in place that allow us to quickly respond to such situations. A clear illustration of this is our recent success in securing a 12-month resolution with AWS in just two weeks, without needing to touch any emergency funds.

I welcome any further questions or concerns you might have about our ongoing strategies to ensure that Nix remains reliable, thriving, and resilient. Don’t hesitate to ask!

Furthermore, I’ve had the pleasure of speaking with some truly outstanding individuals who have been instrumental in building and maintaining some of the world’s most remarkable open-source communities and projects. We’re keen on implementing the lessons learned from these conversations to continue improving NixOS. Your feedback on how we can better our community is always welcome!

12 Likes
17

Thank you so much for the quick answer.

Researching about this, just reading from reddit, this is only for cache, if this S3 has troubles, this will not make the distro disappear. Is this right ?

Again, sorry if I miss understood.

18

Correct. Hopefully the below can help you better understand how the cache is used, in the hypothetical scenario where that content is wiped today:

Broadly:

  • NixOS and Nixpkgs still exist, and can be fetched and updated from github. Commits and PR’s would continue, though developers will likely have some distraction.
  • Packages will be built locally if they’re not found in cache, even where no channel update was done (for example, pulling in something via a devshell).
  • The next cycle of builds can start populating the cache with fresh output, and things will get mostly back to normal soon enough.

That sounds pretty good! In reality, of course there will be some noticeable impact and pain points:

  • New users, or new installs, will be the most stuck (but the installer iso’s are kept separately and can still be used).
  • There might be some wrinkles attempting regular channel updates, users wanting new packages/fixes may need to fallback to git checkouts, but that will be a minor inconvenience compared to the full source rebuild that would follow. We have delays for channel updates from time to time when unstable tests start failing, and the end-user experience would be somewhat similar.
  • There are other community caches that may also be able to supply at least some of the build products if needed in a hurry, and users with multiple machines can copy between stores, and all kinds of other options (that are optimisations today) still exist.

So, for typical most-users most-of-the-time scenarios:

  • the cache+CDN is a (big) cpu-saver and distribution mechanism, but otherwise mostly non-critical.
  • Most of the hot usage follows a moving horizon along either the unstable or release channels.
  • The huge depth of the cache for historical data comes into play for other cases: users who have pinned versions of the distribution, or specific dependencies, for various reasons.
  • Some older cached artefacts potentially can’t be rebuilt today. Understanding that in detail, and what it means for optimising the caching strategy, is part of the longer-term project.
7 Likes
19

Thank you sooooo much for the info.

2 Likes
20

Not much beyond that to add except for huge thanks for that informative response @uep ! <3

2 Likes