2023 Summer of Nix program updates

Events Summer of Nix
1

Hello everyone, and welcome to this year’s Summer of Nix!

My name is Valentin, and I support program organisers @mightyiam and @mat on behalf of the NixOS Foundation. My mission is primarily to facilitate communication: make sure everyone knows what’s important for whom and why, get the right people together to answer questions and solve problems, and make the results of your work visible – so you can focus on writing code.

What Summer of Nix is about

Summer of Nix is the NixOS Foundation’s contribution to the European Commission’s Next Generation Internet (NGI) initiative, which is part of the Horizon Europe program. The NixOS Foundation partners with NLnet and others within the NGI Zero consortium to help fulfil the NGI objectives of fostering technological innovation and working towards a more resilient and participatory digital infrastructure. Our goal is to assist projects funded through NGI Zero with reproducible packaging and declarative configurations using Nix, while growing the community of developers that have the necessary skills and experience to maintain these setups.

Summer of Nix org chart

Horizon Europe is a huge program with a total volume of 95.5 billion EUR. NGI has a budget of more than 300 million EUR, and is subdivided into multiple funds. Currently NGI Zero manages NGI Zero Entrust, NGI Zero Review, and NGI Zero Core, totalling at 24.5 million EUR. Part of that money – ca. 90 000 EUR for each of 2023, 2024, and 2025 – is allocated to the NixOS Foundation to provide packaging services and to strengthen the FOSS ecosystem by improving tooling and processes.

This year is the third instalment of the Summer of Nix program, and you can browse the reports from 2021 and 2022 and this year’s call for participants for details if you’re curious.

This year’s goals

Together with NLnet’s director of strategy Michiel Leenaars and project manager Ronny Lam, and based on the past two years of experience with Summer of Nix, we developed the following goals for this year’s program:

  1. Cater to software developers

    Developers build the Next Generation Internet, and developer tooling is where Nix, Nixpkgs, and NixOS already shine. We will put emphasis on robust, readable code, sane workflows, and good documentation targeted at domain experts.

  2. Focus on three major topics:

    • Open Hardware Ecosystem

      Provide out-of-the-box access to toolchains for designing libre silicon and PCBs, and get FPGAs up and running reproducibly. We want the Nix ecosystem to be the default for hardware designers, and be prepared for the activity around the European Chips Act.

    • Self-hosting and service portability

      Building software reproducibly is not enough. Being able to deploy services with minimal effort is key to fostering adoption of new web technologies that give their users more agency and control. We will develop or extend NixOS modules for web projects to enable customisable self-hosting or porting services between hosting providers.

    • Federated services

      Support projects implementing ActivityPub with convenient service configuration through NixOS. This continues efforts from past years.

  3. Increase visibility and improve discoverability

    Software projects supported through NGI Zero should be easy to find and get to work. We will set up a unified package and service module repository with continuous integration, automatic updates, and searchable documentation.

These goals are designed with current constraints and future opportunities in mind, and also address difficulties we experienced in previous years. While in the past the program went into breadth, this time we will focus on fewer, high-impact objectives and establish sustainable workflows that will carry the program over the next three years and beyond. We want to direct our attention towards software projects with far reach or great potential, where taming complex builds, dependencies, or configurations with Nix would provide the most benefit to their maintainers, contributors, or users now and in the long run.

Working in public

Summer of Nix puts learning and knowledge sharing front and center. And we know that it’s not just that code that matters, but also how it’s made. To share with everyone what we do and learn, we will post updates on activities and results in this thread. See also the NGIpkgs issue tracker for the tasks the teams are working on. Maybe you will participate next year?

Have a great Summer of Nix!

6 Likes
2

Here is a first update on what has been happening in Summer of Nix so far.

After publishing the initial announcement with @mat, @mightyiam managed participant applications and selected mob facilitators, who assembled three mobs of five people each.

Since beginning of June, @cleeyv was working on setting up a monorepo called NGIpkgs, and with assistance of @picnoir got a few sample packages from last year to build. Until end of June, @cleeyv and me, with help by @roberth, did some debugging, and answered a few design questions around the monorepo, such as what to do with projects that are already packaged with Nix upstream or in Nixpkgs. @cleeyv proceeded with documenting workflows based on his experience from past years, and providing sample configurations.

In July, @cleeyv got Hydra to automatically build all packages in the repository, and is now working on continuous integration for each pull request. Meanwhile, I was digging through NLnet’s database to assess and select projects to work on, collected additional information, got in touch with project authors, and wrote up issues. There was also some back-and-forth between Ronny, @edolstra, @mightyiam, and me around our bureaucratic obligations towards the European Commission; I read through hundreds of pages of grant agreements, double-checked the budgets, and started sorting through existing documents and collecting notes, such that the next two rounds of Summer of Nix can be prepared more smoothly.

Two mobs started out on 2023-07-17, the third one will begin 2023-07-31. Being distributed over three time zones far apart, they will then essentially work around the clock. Because of that, we had two brief kick-off meetings, and had an opportunity to learn to know each other a bit and answer some remaining questions.

@cleeyv will continue with preparing and documenting workflows for manual testing using containers and public cloud deployments, and later for automated NixOS VM tests. @mightyiam started preparations for the Summer of Nix hiring event planned for autumn.

I will post updates again in about a month.

3 Likes
3

Hi, this looks awesome.

Multiple questions:

  • will ngipkgs send their contributions to nixpkgs at some point? What’s the maintenance story also?

So if I understand well, should I expect activity for PRs like litex: init at 2023.04 by RaitoBezarius · Pull Request #237658 · NixOS/nixpkgs · GitHub and more :slight_smile: ?

Thank you again for all you do!

1 Like
4

We will choose the path of quickest turnaround. If something is already in Nixpkgs, contributions will go there. I’d like to put in place automation for version bumps, but we have to see how that fits the timeline. We don’t have funding for maintenance, but code in NGIpkgs can be moved to Nixpkgs by anyone willing to maintain it.

In principle, yes. Teams are pretty much autonomous, but I think it’s likely they will pick up these widely used packages on the way.

1 Like
5

@fricklerhandwerk Can you expand a bit on the mob programming method you’re using? IIUC it means all mob participants are in one place, and program together using one keyboard and one screen. Is this how you’re doing it, or are you doing a remote version with screen sharing, or something entirely different?

Thanks!

6

This is for @mightyiam to answer.

7

I’m curious to know what you have in mind for that. A higher-level, convenient,abstracting module to quickly setup a production ready, selfhosted database? Some convenient abstracting module to setup managed cloud ressources like terranix? Or more convenient, community-maintained bundle modules of popular self-hostable software that includes everything needed, similar to Kubernetes Helm charts?

Background: I started fiddling with NixOS to host a Rust webapp, which at minimum needs a database like Postgresql and a reverse-proxy like Caddy and an all-inclusive deployment mechanism like nixos-rebuild , nixos-anywhere, Nix disko for disk formatting, agenix for secrets handling. For self-hosting NixOS with its existing tools seems much more convenient than other popular combinations (terraform, cloud init, Ubuntu/Redhat, Ansible, Docker, Docker Swarm, Kubernetes), with totally different syntax and sometimes unreliable tooling. Still it’s a lot of Nix libraries to learn for NixOS as well.

8

I would personally imagine that what is left in that area is:

  • state rollbacks: enabling NixOS to checkpoint the right data according to local policies and offering to the user the capability to restore the state before a switch to another configuration
  • state portability: enabling NixOS to send its checkpoints to another NixOS system

Those would enable fearless migrations even when the software is broken or does mistakes (history in nixpkgs shows those happens more than we would wish for and they can delete all your data if you don’t have backups.)

I don’t think we should pursue Kubernetes Helm charts or stuff like that, this is something that already exist and is called NixOS modules. Cloud resources seems out of scope for self-hosting because it excludes on-premise hardware. I don’t really know what is self-hosted database :stuck_out_tongue: — we have PostgreSQL modules and more and they can be configured with advanced options, we even have the JIT nowadays.

9

I see. Your focus is on the data when (re)deploying once a service is already running in production, which is certainly important and should be improved.

My focus was on the setup for the first deployment. For example, i have a webapp that i want to run in production: I need a database and a hardened security setup for that database, a reverse-proxy (that takes care of TLS certificates), firewall settings etc. For the database i can either use an expensive, managed database on AWS or i can self-host using NixOS modules like services.postgresql. However, this particular module is quite-low level and i need to have a deep understanding to fiddle with its quirks and security settings. The same goes for a reverse-proxy module like services.caddy or systemd in general. Overall there’s still a lot of configuration needed (just look at Gitea), which is replicated for many other services.

So NixOS modules are a good foundation, but i think we could benefit from more production-grade option conventions, ideally declarative and minimal. I’d like to only specify that for this webservice i need a Postgresql database, self-hosted on the same server, with these compatible version ranges, and that it will be accessible from the internet; everything else could be taken care of automatically: Systemd setup, system user configurations, auto-dumps, auto-backups, ports, reverse-proxy settings, firewall ports etc.

Both help to “deploy services with minimal effort”.

10

It’s some version of https://www.remotemobprogramming.org/ .

1 Like
11

More like the third option. Here we’re supporting publicly funded open source projects, many of which revolve around federated social media, and some being rather small but promising. Having a low-effort setup is vital for ramping up adoption, which is important both to boost developers’ morale as well as users’ agency.

1 Like
12

Here’s the second progress update for Summer of Nix 2023.

The three mobs incorporated themselves as algae, moss, and fungus, and are now spreading their spores all over the new NGIpkgs monorepo as well as Nixpkgs.

Overall, participants and organisers are happy with the process. Mobs actively communicate and collaborate with project authors, and apart from the usual gripes with rough edges around Nix and Nixpkgs, we see continuous progress and working code. In particular, the mobs had to deal with Python Poetry, PHP Composer, defining NixOS service modules and deploying them to Digital Ocean, as well as NixOS VM tests. Lots of knowledge was shared, that’s for sure. There is more to come, as we also have non-trivial Rust projects on the agenda. See the issue tracker for some diligent technical reports and exchanges.

The fact that no involvement by project organisers was needed over the course of a month is testament that the combination of an initial plan and the mob format works out as intended.

More detail on what happened in the past weeks:

  • @cleeyv added continuous integration for pull requests, and helped the mobs with resolving technical issues and questions
  • The mobs refactored the repo’s flake.nix and set up automatic code formatting
  • moss mob
    • Packaged Flarum and will continue with developing a service definition once they are up to speed with NixOS tests
    • Packaged librecast and dependencies, and upstreamed them to Nixpkgs
    • Now working on openXC7 upstream with the project author
  • algae mob
  • fungus mob

To be clear, while this looks like a fairly short list, these are significant chunks of work.

Next up, @cleeyv will spend his remaining two weeks on documenting the process to deploy to cloud VMs for a smoother manual testing cycle, as well as adding documentation on how to effectively integrate NGIpkgs software into other contexts.

There is still some trouble with Darwin builds in continuous integration. Given that the services all target Linux and mobs use VMs to test them, this is not a problem for ongoing development. But that also means that it likely won’t get resolved within this year’s program time frame.

We also discussed licensing for the monorepo. A possible choice is MPL 2.0, but we’d like feedback from Nixpkgs maintainers to make sure it’s easy to upstream Summer of Nix work to Nixpkgs. If you have suggestions or feedback on that subject, please add it to the linked thread.

Providing a web front-end for the monorepo that displays all available packages and configurations in a more convenient way will almost certainly end up as a stretch goal to be covered by the remaining part of the budget for this year. For now we will focus more on substance than on appearance, simply to get more open source software into people’s hands, reproducibly.

2 Likes
13

The Summer of Nix program proper is now over, and here is the second-to-last update for this year.

Since the last post, most of the larger packaging efforts came to conclusion:

  • Rosenpass will appear in the 23.11 NixOS release
  • OpenXC7 now has a reproducible development shell upstream
  • A service definition was added for Flarum
  • The package and service module for mCaptcha are operational, and the corresponding pull request about to get merged
  • Exploratory work started on packaging Libervia, but the project is under heavy rework and we will wait for it to stabilise
  • proximity_matcher_webservice, which was packaged last year, got a small fixup to unblock development

The list is fairly short: as mentioned previously, participants went for quality rather than quantity and had to deal with a variety of projects on a spectrum of complexity and maturity, which was often tricky. Nonetheless, the three mobs effectively served all but one of the actionable requests made via the NGI0 Review fund and packaged multiple projects funded through NGI0 Entrust, with the software being usable out of the box from NGIpkgs or Nixpkgs. And project authors are happy with the results:

As was planned for this year, Summer of Nix will now seamlessly transition into a continuous and more relaxed mode of operation. We’ve all met for two wrap-up calls (due to distant time zones) to exchange feedback and experiences, which was quite insightful:

  • The mob programming format proved to be extremely effective for knowledge sharing. Some started with almost no Nix experience, and were able to ramp up very quickly.
  • It was a lot of fun, but also very intense. The workload quite substantial, in part due to challenges with scheduling.
  • Working together with project authors was fruitful, and should be emphasized even more, getting them into the loop as early as possible.
  • Although we improved a lot over the past years, task curation needs more emphasis. More mature projects are much easier to package and those packages have a longer half-life.
  • The program started late this year and barely deserves the title “Summer” even for the northern hemisphere. That’s also one of the reasons why we had more software professionals than students as participants this year. We’ll put more effort into aligning the schedule with the semester breaks next time. And may consider a new name for the program…
  • Participants expressedly asked for getting prioritised access NixCon tickets, as that would be a unique opportunity to learn to know the community that otherwise appears only in the form of avatars. We’ll try to make that happen for next year. NixCon 2023 tickets were unfortunately very hard to come by.
  • Hardware requirements for mob programming and Nix development at scale should be made explicit. Running NixOS tests or large evaluations runs you into limits quickly.
  • There was also a long wishlist with regard to the Nix ecosystem:
    • Nixpkgs maintainer capacity is very constrained, and more help getting upstream changes merged would be appreciated
    • Nixpkgs documentation is still hard to find or nonexistent, and some concepts such as overlays or the module system remained mysterious to many and required an in-person introduction and working examples
    • There’s a clear need for more fine-grained, incremental builds, and better Nix language tooling

Some of the participants will stay to finish up the remaining issues, and packaging work will focus on GNU Taler, which consists of many components. As time and budget constraints allow, they will also update and migrate packages into the monorepo that were created in the past two years.

@mightyiam is working in the background to revamp the website to have a better point of contact for the next season, and is documenting organisational insights for the final report. My focus towards year’s end will be on streamlining the administrative process to ensure a quick ramp-up next year, and make a hand-over easy in case someone else will organise the program.

In the meantime, @mightyiam and me contacted companies who are relying on Nix and may be interested in hiring our program’s graduates.

If you’re representing such a company and you’re looking to expand a DevOps or software development team, check out our list of job seekers. Summer of Nix developers have collected plenty of practical experience getting complex build setups and service configurations to work reproducibly with Nix and NixOS, and demonstrated effective collaboration towards the program’s goals. If you end up hiring one of them, we’d appreciate a donation to the NixOS Foundation so we can continue packaging open source software.

(This is also posted on the job board on Discourse.)

Thanks to @mightyiam’s initiative, some of the participants recently met to record an episode of Mob Mentality with Chris Lucian and Austin Chadwick, which set to air soon. This was exciting! We’ll post a link when it’s out.

Although much delayed, we’re still planning to do a very light-weight version of last year’s lecture series. The purpose is for participants and the general audience to get in touch with developers in the Nix ecosystem, ask questions, and get some insights on what they are currently working on and how one can participate. We’ve sent out invitations to speakers and will post updates when there is more information.

Around the turn of the year I’ll post another update before returning to you with a final report on Summer of Nix 2023.

7 Likes
14

Hi everyone! The schedule for the 2023 Nix Developer Dialogues is now public, and ever-growing as we receive confirmations by show guests. Share with friends and colleagues!

3 Likes
15

The Mob Mentality episode on Summer of Nix with @albertc @augustebaum @mightyiam @lorenzleutgeb is out:

Tune in to this illuminating episode for an insider’s perspective on the Summer of Nix 2023, where mob programming becomes a catalyst for knowledge-sharing, problem-solving, and fostering a vibrant open-source community. Don’t miss out on the camaraderie and insights shared by our awesome panel. :star2:

Thanks Chris and Austin for having us!

3 Likes
16

I wrote a blog post about my experience!

4 Likes
17

Here’s the last update on Summer of Nix 2023, the post-program activities to be precise.

The 10 weeks since the last update were rather quiet, with consistent, unexciting work getting done. The focus was on addressing packaging requests by projects funded through NLnet and otherwise tying up loose ends.

@alejandrosame @enzime and @matusf made substantial progress with packaging the various dependencies of GNU Taler.

@alejandrosame migrated hyperspace to NGIpkgs.

@mightyiam and @augustebaum migrated arpa2 into NGIpkgs.

@mightyiam put some time into paying off tech debt that had accumulated.

@lorenzleutgeb helped with monorepo maintenance tasks, and got quite far with packaging Vula, but got stuck getting it to run on NixOS.

@albertc and @jleightcap started packaging LibreSOC dependencies, but didn’t manage to spend their budgeted time.

@doronbehar spontaneously helped out the developer Keyoxide with Nix packaging of GPG support tooling pysequoia and private-key-store-server.

Not all of the allocated budget was spent, as many of us had conflicting priorities — including myself, such that I did not get to re-assign tasks. I decided not to rush it due to Brook’s law, and with a recently secured budget up to including 2026, we’re in for the long(-ish) haul.

We’ll fade into the background for now, in order to close the books for 2023, evaluate learnings, write a full report, and prepare for Summer of Nix 2024.

9 Likes
18

Here’s the update to conclude Summer of Nix 2023. This entails wrapping up last year and preparing the 2024 summer program:

The next update will be on Summer of Nix 2024. Thanks to everyone who participated last year!

6 Likes