NixOS Foundation's Financial Summary: A Transparent Look into 2022

TL;DR - Analyzing and making the NixOS Foundation financials transparent and also helping us understand our finances better to fund more community efforts. Creation of a general financial forecast in the works as well.

We are excited to share the financial summary for the NixOS Foundation for the first time. This goes beyond just being a detailed breakdown of our income and expenses for 2022 and is also the initial attempt at a more structured approach in being more effective with our finances.
Key Goals

  • Provide insights into our financial journey/status
  • Highlight opportunities for future growth and improvement
  • Encourage community involvement in the financial aspects
  • Help us understand our finances so that we can more confidently use them for what they are here for, the community. On a small note, I’d prefer we utilize more of the funding up to the buffer which will also mean ending the year on a lower amount (discussed below).

NixOS Foundation’s 2022 Financial Overview

Throughout 2022, the NixOS Foundation experienced a decrease in its balance from €148,035.51 to €112,785.76. This change can be attributed to total incoming funds of €80,955.28, while our total outgoing funds were €116,205.03.

Sources of Incoming Funds

During the year, the NixOS Foundation received funding from various sources, including:

  • Company Donations: €34,505.68
  • Open Collective Donations via Stripe: €48,714 (updated due to stripe 3 month matter)
  • Donations via PayPal: €1,411.09
  • Personal Donations: €3,989.9
  • Funds from NLnet Foundation for the specific programs (i.e. Summer of Nix): €140,000 total balance that was received between 2021 to 2023.

These four sources accounted for the majority of our incoming funds, contributing almost 90% of the total income.

Significant Expenses

  • Personal Payments/Wise (Summer of Nix): €-98,101.57
  • Hetzner.com: €-6,969 (updated)
  • DTH Location (NixCon expenses): €-5,640
  • Association des Élèves de l’École Normale Supérieure (NixCon expenses): €-2,500
  • Netlify: €-1,403.95

These expenses represented around 95% of our total outgoing funds during the year.

There are substantial expenses that are currently sponsored by external companies and therefore not represented in the NixOS banking logs. As these are critical we do want to keep them in mind as we calculate our financial projects and into our estimate of the buffer we want to keep in case of emergencies. These items include:

  • Storage: ~€10k/month
  • Fastly: Estimated at over €50k/month (this is hard to take into account with a buffer)
  • Buckets: ~€7k/month

Miscellaneous Expenses

In addition to the major sources of income and expenses, the NixOS Foundation also experienced several minor expenses, refunds, and miscellaneous transactions:

  • Aristo: €-354.83
  • AWS EMEA: €-102.17
  • Bank: €11.98
  • Bunq: €-771.5
  • ImprovMX: €-84.92
  • Notion Labs, Inc.: €-54.06
  • Seats2meet.com: €-411.52
  • SkillSource B.V.: €-138
  • Streamyard.com: €-322.51

Moving Forward: Reflection and Growth

Releasing this financial summary marks a significant milestone for us as we strive to hold and grow the commitment to transparency and accountability. By sharing this information, we hope to encourage community members to participate in the discussion of our financial strategies and contribute ideas for improvement.

Financial Forecast & General Budget Planning
In order to define a forecast and a buffer we took a look at multiple financial data points which included:

  • Potential unexpected expenses
  • General curve of profit/loss over the last three years
  • Expenses which today are covered externally (storage/fastly/buckets)
  • Expected large expenses based on prior years

There are a few “covered externally” expenses that are too high to prepare for such as fastly and therefore we will be working on a backup solution there. For the rest we would want to have enough funds to sustain all aspects of Nix for at least 3 months which roughly equates to consistently having 50K as the bottom bar in the bank. The reason for 3 months is that the expectation is that it will be enough time for us to get together in case of emergency to resolve the issues and get back on track (i.e. concentrated funding efforts/backup plans/expense reduction).
This will obviously change from year to year depending on donations/expenses.

Quick Final Thoughts
As we move into 2023, we will continue to focus on improving our financial transparency and strategy. We will also make it a key priority to keep exploring new opportunities to expand the scope of the funds going into NixOS, ensuring that we can all continue to support the community and further the development of the NixOS ecosystem. One of the projects in 2023 will be to expand on the Documentation funding effort and create an official yearly NixOS fundraising/sponsorship effort.

As this is the first time for an effort like this, we would appreciate any and all feedback/questions/discussions!
The next step is to conduct the same effort for the first half of 2023 (after incorporating the feedback)

44 Likes

Nice to see some numbers!

Seeing as you’ve listed sponsored amounts to keep us mindful of them, it’s missing any compute, as in hydra builders, AFAICT. With the same goal in mind, it would be interesting to see how that maps, if feasible.

10 Likes

Thanks for calling this out, looking into it now :slight_smile:

4 Likes

Is Fastly cost really €50k/month or is it mistake?

4 Likes

It’s that high, potentially even slightly higher based on the estimates we pulled. Our current resolution there would be to have a backup plan should anything happen in that area.

2 Likes

That’s really huge. Which company is sponosing this? Fastly itself?

2 Likes

Note that Fastly is “in front” of cache.nixos.org, and there is a really high volume of traffic considering how Nix and the binary cache works.

I suspect that the numbers for its usage patterns would make most people dizzy from vertigo! (Even though I don’t know them.)

4 Likes

I wonder if anyone has some traffic numbers that would explain the cost jumping from ~3500$ pre-Fastly (Cloudfront circa 2018) to 50k$ now with Fastly. That still seems like a really huge difference.

7 Likes

Let me try and see if I can pull those

3 Likes

Friendly bump for the traffic numbers, are those available in the end?

1 Like

Resurfaced the traffic numbers with the team, let me see again if we can dig them up.

stats for all NixOS and Nixpkgs releases (i.e. all the store-paths.xz files on releases.nixos.org):
Done! Found 51360806 store paths, 384085 GiB uncompressed size, 93618.1 GiB compressed size, 58.3715 GiB estimated .narinfo size

#thanks to @edolstra

8 Likes

And here are some stats from Fastly:

8 Likes

Is this per 3 hours, right? I wonder whats the peak concurrent throughput reached in GB/s :grinning_face_with_smiling_eyes:

2 Likes

Per day. Here is the 95th percentile per minute in the last 24h:

5 Likes

Some info on our AWS costs (primarily cache.nixos.org and releases.nixos.org):

  • The S3 bucket for cache.nixos.org stores 667M objects, with 107 TiB in standard storage and 318 TiB in infrequent access storage, so 425 TiB in total.

  • The S3 bucket for nix-releases (i.e. https://releases.nixos.org) is 25.1 TiB (mostly ISO images).

  • The S3 storage costs for our sponsor are about $5500 per month, after a $2300 enterprise discount.

  • The S3 transfer costs are about $900 per month, which is mostly from 29.5 TiB of traffic between S3 and Fastly for cache.nixos.org, and 2.6 TiB for releases.nixos.org. (This also appears to have a discount since the costs are $0.03 per GiB.)

  • Our Fastly traffic in April was 1508 TiB, so the CDN is amazingly effective.

  • The runtime closure of all non-beta NixOS releases (i.e. the store paths reachable from the store-paths.xz files in Channels for NixOS project(s)) is 13M store paths, 27.1 TiB. So garbage-collecting cache.nixos.org to only keep NixOS releases would shrink it to 6.3% of its current size.

    Note that this doesn’t include build-time-only dependencies like source tarballs.

  • The runtime closure of all non-beta NixOS releases and all Nixpkgs releases is 51M store paths, 91.4 TiB, so about 21% of the bucket. Unfortunately Nixpkgs has no stable releases, they’re all marked as “pre”.

    So we can reduce storage costs by almost 80% by GCing everything that isn’t reachable from the store-paths.xz files on releases.nixos.org.

17 Likes

Thank you for the insight.

Are there access statistics for these categories of paths you have outlined? There might be a category which sees next to no traffic and could therefore be GC’d without affecting users.

Yes, we do have logs for fastly, so we could use e.g. all store paths that are accessed at least N times in the last month as GC roots. That should also help to avoid deletion of frequently used build-time dependencies like source tarballs.

Yeah sounds like access based GC’ing would be less disruptive to users. There is also S3 Intelligent Tiering which can automatically move objects to less expensive storage classes, but that also has extra cost for the monitoring of objects ($0.0025 per 1,000 objects/mo).

From what I see, Nix local cache ttl is a month? So that can result in some obscure errors to users if they attempt to access a store path that was previously cached.

4 Likes

fyi - I’ve edited two items on the financial side : Open Collective Donations via Stripe & Hetzner as we received more updated numbers during the filing process for 2022 today.

2 Likes

Protocol Labs can help with a backup plan. Last year November, we launched Saturn, a community run CDN (>2500 nodes @10Gbit/s each). We could offer a competitive decentralized CDN solution at much better rates.

2 Likes