2023 Summer of Nix program goals

Development Summer of Nix
1

Hello everyone, and welcome to this year’s Summer of Nix!

My name is Valentin, and I support program organisers @mightyiam and @mat on behalf of the NixOS Foundation. My mission is primarily to facilitate communication: make sure everyone knows what’s important for whom and why, get the right people together to answer questions and solve problems, and make the results of your work visible – so you can focus on writing code.

What Summer of Nix is about

Summer of Nix is the NixOS Foundation’s contribution to the European Commission’s Next Generation Internet (NGI) initiative, which is part of the Horizon Europe program. The NixOS Foundation partners with NLnet and others within the NGI Zero consortium to help fulfil the NGI objectives of fostering technological innovation and working towards a more resilient and participatory digital infrastructure. Our goal is to assist projects funded through NGI Zero with reproducible packaging and declarative configurations using Nix, while growing the community of developers that have the necessary skills and experience to maintain these setups.

Summer of Nix org chart

Horizon Europe is a huge program with a total volume of 95.5 billion EUR. NGI has a budget of more than 300 million EUR, and is subdivided into multiple funds. Currently NGI Zero manages NGI Zero Entrust, NGI Zero Review, and NGI Zero Core, totalling at 24.5 million EUR. Part of that money – ca. 90 000 EUR for each of 2023, 2024, and 2025 – is allocated to the NixOS Foundation to provide packaging services and to strengthen the FOSS ecosystem by improving tooling and processes.

This year is the third instalment of the Summer of Nix program, and you can browse the reports from 2021 and 2022 and this year’s call for participants for details if you’re curious.

This year’s goals

Together with NLnet’s director of strategy Michiel Leenaars and project manager Ronny Lam, and based on the past two years of experience with Summer of Nix, we developed the following goals for this year’s program:

  1. Cater to software developers

    Developers build the Next Generation Internet, and developer tooling is where Nix, Nixpkgs, and NixOS already shine. We will put emphasis on robust, readable code, sane workflows, and good documentation targeted at domain experts.

  2. Focus on three major topics:

    • Open Hardware Ecosystem

      Provide out-of-the-box access to toolchains for designing libre silicon and PCBs, and get FPGAs up and running reproducibly. We want the Nix ecosystem to be the default for hardware designers, and be prepared for the activity around the European Chips Act.

    • Self-hosting and service portability

      Building software reproducibly is not enough. Being able to deploy services with minimal effort is key to fostering adoption of new web technologies that give their users more agency and control. We will develop or extend NixOS modules for web projects to enable customisable self-hosting or porting services between hosting providers.

    • Federated services

      Support projects implementing ActivityPub with convenient service configuration through NixOS. This continues efforts from past years.

  3. Increase visibility and improve discoverability

    Software projects supported through NGI Zero should be easy to find and get to work. We will set up a unified package and service module repository with continuous integration, automatic updates, and searchable documentation.

These goals are designed with current constraints and future opportunities in mind, and also address difficulties we experienced in previous years. While in the past the program went into breadth, this time we will focus on fewer, high-impact objectives and establish sustainable workflows that will carry the program over the next three years and beyond. We want to direct our attention towards software projects with far reach or great potential, where taming complex builds, dependencies, or configurations with Nix would provide the most benefit to their maintainers, contributors, or users now and in the long run.

Working in public

Summer of Nix puts learning and knowledge sharing front and center. And we know that it’s not just that code that matters, but also how it’s made. To share with everyone what we do and learn, we will post updates on activities and results in this thread. See also the NGIpkgs issue tracker for the tasks the teams are working on. Maybe you will participate next year?

Have a great Summer of Nix!

5 Likes
2

Here is a first update on what has been happening in Summer of Nix so far.

After publishing the initial announcement with @mat, @mightyiam managed participant applications and selected mob facilitators, who assembled three mobs of five people each.

Since beginning of June, @cleeyv was working on setting up a monorepo called NGIpkgs, and with assistance of @NinjaTrappeur got a few sample packages from last year to build. Until end of June, @cleeyv and me, with help by @roberth, did some debugging, and answered a few design questions around the monorepo, such as what to do with projects that are already packaged with Nix upstream or in Nixpkgs. @cleeyv proceeded with documenting workflows based on his experience from past years, and providing sample configurations.

In July, @cleeyv got Hydra to automatically build all packages in the repository, and is now working on continuous integration for each pull request. Meanwhile, I was digging through NLnet’s database to assess and select projects to work on, collected additional information, got in touch with project authors, and wrote up issues. There was also some back-and-forth between Ronny, @edolstra, @mightyiam, and me around our bureaucratic obligations towards the European Commission; I read through hundreds of pages of grant agreements, double-checked the budgets, and started sorting through existing documents and collecting notes, such that the next two rounds of Summer of Nix can be prepared more smoothly.

Two mobs started out on 2023-07-17, the third one will begin 2023-07-31. Being distributed over three time zones far apart, they will then essentially work around the clock. Because of that, we had two brief kick-off meetings, and had an opportunity to learn to know other a bit and answer some remaining questions.

@cleeyv will continue with preparing and documenting workflows for manual testing using containers and public cloud deployments, and later for automated NixOS VM tests. @mightyiam started preparations for the Summer of Nix hiring event planned for autumn.

I will post updates again in about a month.

2 Likes
3

Hi, this looks awesome.

Multiple questions:

  • will ngipkgs send their contributions to nixpkgs at some point? What’s the maintenance story also?

So if I understand well, should I expect activity for PRs like litex: init at 2023.04 by RaitoBezarius · Pull Request #237658 · NixOS/nixpkgs · GitHub and more :slight_smile: ?

Thank you again for all you do!

1 Like
4

We will choose the path of quickest turnaround. If something is already in Nixpkgs, contributions will go there. I’d like to put in place automation for version bumps, but we have to see how that fits the timeline. We don’t have funding for maintenance, but code in NGIpkgs can be moved to Nixpkgs by anyone willing to maintain it.

In principle, yes. Teams are pretty much autonomous, but I think it’s likely they will pick up these widely used packages on the way.

1 Like
5

@fricklerhandwerk Can you expand a bit on the mob programming method you’re using? IIUC it means all mob participants are in one place, and program together using one keyboard and one screen. Is this how you’re doing it, or are you doing a remote version with screen sharing, or something entirely different?

Thanks!

6

This is for @mightyiam to answer.

7

I’m curious to know what you have in mind for that. A higher-level, convenient,abstracting module to quickly setup a production ready, selfhosted database? Some convenient abstracting module to setup managed cloud ressources like terranix? Or more convenient, community-maintained bundle modules of popular self-hostable software that includes everything needed, similar to Kubernetes Helm charts?

Background: I started fiddling with NixOS to host a Rust webapp, which at minimum needs a database like Postgresql and a reverse-proxy like Caddy and an all-inclusive deployment mechanism like nixos-rebuild , nixos-anywhere, Nix disko for disk formatting, agenix for secrets handling. For self-hosting NixOS with its existing tools seems much more convenient than other popular combinations (terraform, cloud init, Ubuntu/Redhat, Ansible, Docker, Docker Swarm, Kubernetes), with totally different syntax and sometimes unreliable tooling. Still it’s a lot of Nix libraries to learn for NixOS as well.

8

I would personally imagine that what is left in that area is:

  • state rollbacks: enabling NixOS to checkpoint the right data according to local policies and offering to the user the capability to restore the state before a switch to another configuration
  • state portability: enabling NixOS to send its checkpoints to another NixOS system

Those would enable fearless migrations even when the software is broken or does mistakes (history in nixpkgs shows those happens more than we would wish for and they can delete all your data if you don’t have backups.)

I don’t think we should pursue Kubernetes Helm charts or stuff like that, this is something that already exist and is called NixOS modules. Cloud resources seems out of scope for self-hosting because it excludes on-premise hardware. I don’t really know what is self-hosted database :stuck_out_tongue: — we have PostgreSQL modules and more and they can be configured with advanced options, we even have the JIT nowadays.

9

I see. Your focus is on the data when (re)deploying once a service is already running in production, which is certainly important and should be improved.

My focus was on the setup for the first deployment. For example, i have a webapp that i want to run in production: I need a database and a hardened security setup for that database, a reverse-proxy (that takes care of TLS certificates), firewall settings etc. For the database i can either use an expensive, managed database on AWS or i can self-host using NixOS modules like services.postgresql. However, this particular module is quite-low level and i need to have a deep understanding to fiddle with its quirks and security settings. The same goes for a reverse-proxy module like services.caddy or systemd in general. Overall there’s still a lot of configuration needed (just look at Gitea), which is replicated for many other services.

So NixOS modules are a good foundation, but i think we could benefit from more production-grade option conventions, ideally declarative and minimal. I’d like to only specify that for this webservice i need a Postgresql database, self-hosted on the same server, with these compatible version ranges, and that it will be accessible from the internet; everything else could be taken care of automatically: Systemd setup, system user configurations, auto-dumps, auto-backups, ports, reverse-proxy settings, firewall ports etc.

Both help to “deploy services with minimal effort”.

10

It’s some version of https://www.remotemobprogramming.org/ .

1 Like
11

More like the third option. Here we’re supporting publicly funded open source projects, many of which revolve around federated social media, and some being rather small but promising. Having a low-effort setup is vital for ramping up adoption, which is important both to boost developers’ morale as well as users’ agency.

1 Like
12

Here’s the second progress update for Summer of Nix 2023.

The three mobs incorporated themselves as algae, moss, and fungus, and are now spreading their spores all over the new NGIpkgs monorepo as well as Nixpkgs.

Overall, participants and organisers are happy with the process. Mobs actively communicate and collaborate with project authors, and apart from the usual gripes with rough edges around Nix and Nixpkgs, we see continuous progress and working code. In particular, the mobs had to deal with Python Poetry, PHP Composer, defining NixOS service modules and deploying them to Digital Ocean, as well as NixOS VM tests. Lots of knowledge was shared, that’s for sure. There is more to come, as we also have non-trivial Rust projects on the agenda. See the issue tracker for some diligent technical reports and exchanges.

The fact that no involvement by project organisers was needed over the course of a month is testament that the combination of an initial plan and the mob format works out as intended.

More detail on what happened in the past weeks:

  • @cleeyv added continuous integration for pull requests, and helped the mobs with resolving technical issues and questions
  • The mobs refactored the repo’s flake.nix and set up automatic code formatting
  • moss mob
    • Packaged Flarum and will continue with developing a service definition once they are up to speed with NixOS tests
    • Packaged librecast and dependencies, and upstreamed them to Nixpkgs
    • Now working on openXC7 upstream with the project author
  • algae mob
  • fungus mob

To be clear, while this looks like a fairly short list, these are significant chunks of work.

Next up, @cleeyv will spend his remaining two weeks on documenting the process to deploy to cloud VMs for a smoother manual testing cycle, as well as adding documentation on how to effectively integrate NGIpkgs software into other contexts.

There is still some trouble with Darwin builds in continuous integration. Given that the services all target Linux and mobs use VMs to test them, this is not a problem for ongoing development. But that also means that it likely won’t get resolved within this year’s program time frame.

We also discussed licensing for the monorepo. A possible choice is MPL 2.0, but we’d like feedback from Nixpkgs maintainers to make sure it’s easy to upstream Summer of Nix work to Nixpkgs. If you have suggestions or feedback on that subject, please add it to the linked thread.

Providing a web front-end for the monorepo that displays all available packages and configurations in a more convenient way will almost certainly end up as a stretch goal to be covered by the remaining part of the budget for this year. For now we will focus more on substance than on appearance, simply to get more open source software into people’s hands, reproducibly.

2 Likes