What are overlays?

[ksten]

I’ve been searching for days now on information about what overlays are, why they’re useful, and how they work, but I’m coming up largely empty.

Starting with the wiki page Overlays - NixOS Wiki

Overlays provide a method to extend and change nixpkgs. They replace constructs like packageOverride and overridePackages .

Okaay… I don’t know what those are either.

Consider a simple example of setting the default proxy in Google Chrome:

Which is doing … what, exactly? And why? And how?

The data flow around overlays, especially regarding super and self arguments can be a bit confusing if you are not familiar with how overlays work. This graph shows the data flow:

Actually, it doesn’t. I don’t know what the diagram here signifies.

Here the main package set is extended with two overlays, ext-1 and ext-2. x // y is represented by a // box with x coming in from the left and y from above.

y? x? Where are these? There’s no such symbol in the diagram. What are “x // y” and “a // box”?

As you can see, self is the same for every stage, but super comes from only the stage before. So when you define an attribute foo in the set to override it, within that overlay self.foo will be it’s version, and super.foo will be the non-overriden version. This is why you see patterns like foo = super.foo.override { … }.

Uhh…

final: prev: { f = final.firefox; }

would work, but

final: prev: { f = prev.firefox; }

would make more sense.

Why would it make more sense? What is this doing?

When writing standalone nix code, for example a shell.nix for a project, one usually starts by importing nixpkgs: let pkgs = import <nixpkgs> {} . To use an overlay in this context, replace that by:
import { overlays = [ overlay1 overlay2 ]; }

OK, so there’s some “overlays” keyword I guess? Where do “overlay1” and “overlay2” come from? And what are they?

nixpkgs.overlays = [ (self: super: / overlay goes here /) ];

OK cool, but WHAT is this overlay that goes “here”, and HOW do I define one?

Examples of overlays

Ahh good, some examples!

Overriding a version …
Adding patches …
Compilation options …
Overriding a package inside a scope …

But none of these have overlays in them! Or maybe they do and they just don’t need the overlay keyword? What are overlays again?

Here is an example of Python packages overlay. The trick is to also override python itself with packageOverrides .

I see… Actually, no I don’t. At all.

So what exactly are overlays? What are they used for? How do I make them? How do I use them? What would be some simple examples?

[jonringer]

overlays are a way to apply many changes to a package set. You can add or modify packages, and these overlays can be composed with other overlays.

final: # package set with all overlays applied, a "fixed" point
prev: # state of the package set before applying this overlay
{
  my-new-package = prev.hello;
  some-other-package = final.my-new-package;
}

Then “applying” the overlay usually looks something like:

let
  my_overlay = import ./my-overlay.nix
  pkgs = import  <nixpkgs> { overlays [ my_overlay ]; };
in pkgs.some-other-package

[NobbZ]

@ksten

y? x?

Arbitrary names. X comes from the left and Y from above. and goes into a // box.

@ksten

“x // y”

X merged with Y, X and Y beeing attribute sets and // the merge operator.

@ksten

“a // box”

A rectangle with // in it.

@ksten

final: prev: { f = final.firefox; }

Takes firefox as it is after applying all overlays, and add an attribute f that is equal to that firefox.

@ksten

final: prev: { f = prev.firefox; }

Takes firefox as it was before this overlay and adds another attribute f that is like that firefox. While firefox can be changed by other overlays, f will be like the “prev” firefox forever (unless of course some overlay changes f).

@ksten

OK, so there’s some “overlays” keyword I guess?

It is just an optional attribute in the arguments passed to the function that is returned from import <nixpkgs>.

@ksten

Where do “overlay1” and “overlay2” come from?

They are not really comming from anywhere, they are arbitrary placeholders.

@ksten

OK cool, but WHAT is this overlay that goes “here”, and HOW do I define one?

An overlay is a function that takes 2 attribute sets, the first one represents the packageset after all overlays have been applied, the second represents the packageset before the overlay has been applied ad then returns an attribute set, which is merged into the overal packageset.

You actually have some examples quoted above.

@ksten

But none of these have overlays in them!

They have… All are functions self: super: … or final: prev: ….

[knedlsepp]

Do you find the information in the wiki more useful than what is explained in the nixpkgs manual? NixOS - Nixpkgs 21.05 manual

[samuela]

I think the docs here – and in many other places – would benefit from an examples-first explanation. Sometimes it feels like the docs just throw around terminology without actually communicating the value of a concept. I find the easiest way to grok this stuff is via example.

[ksten]

Yes, that’s been my experience, and it’s why I abandoned trying to learn NixOS years ago. I’ve been at my second attempt for months now, but it’s a very tough slog to wade through the opaque documentation (which, though complete, is only really useful for people who already know how everthing works).

This time around, I’ve been taking notes and using them to put together a guide like I did for Ubuntu package maintainers in the past (also due to technically complete but unapproachable documentation).

The Nixos manual mentioned earlier is a good example:

“This chapter describes how to extend and change Nixpkgs using overlays. Overlays are used to add layers in the fixed-point used by Nixpkgs to compose the set of all packages.”

The last sentence is probably technically correct, but it tells me nothing at all. There’s no introductory explanation of what they are conceptually or why they’re useful (or maybe there is and I just can’t penetrate the jargon). And the rest of the documentation generally follows this pattern of assuming that you already know what everything is and just need a refresher.

In this particular case (overlays), the problem runs deep as I still have no idea what they are for or how to use them even after multiple explanations in the manuals, wikis, and even in this thread. They probably very neatly solve some of the very real problems I’ve been having configuring NixOS, but I can’t recognize the opportunities if I don’t know what they are (and judging by how often they are used, I’m guessing overlays are very important). All explanations end up describing a particular tree in excruciating and unnecesssary detail and jargon that a newcomer couldn’t find useful, but not how and why it fits within the forest. I need a Feinmann-style explanation…

At the moment, slogging through overly-complex or non-working example after example until I can find or cobble together something simple that I can paste into my shell and actually see work, at which point I’ll write a plain down-to-Earth explanation in the guidebook with a copy-pastable example.

[domenkozar]

Note that’s precisely the goal of https://nix.dev

I’d be happy to write a tutorial for overlays if you’d like to pair on it.

[ksten]

Hey that would be great! Msg me privately and we can set something up.

I’m just poking around in nix.dev but already I’m very impressed!

[NobbZ]

The TL’DR of overlays is, they “patch” nixpkgs, without touching its sources, be it to change packages or add new ones.

In my opinion, this concept is not as important as everyone pretends. They are nice if you want to change some library to be compiled with debugging or optimisation globally for each and every program you use, but agree to pay the cost of compiling everything that depends on them from source. Overlays are also a perfect backdor for modules that do not provide a package option. You can replace/modify their used package that way.

For all other intends and purposes I think it is much cleaner to just use overrides or new packages “inplace”.

You can get around overlays for quite a long time. Personally the only overlay I use is the emacs overlay, but only because they are not providing a packages output in their flakes directly.

[azazel75]

@ksten

At the moment, slogging through overly-complex or non-working example after example until I can find or cobble together something simple that I can paste into my shell and actually see work, at which point I’ll write a plain down-to-Earth explanation in the guidebook with a copy-pastable example

But, to be frank, overlays aren’t really stuff for beginners (due to the sometime “alien” concepts like “fixed-point”) and you seem to have an hard time to understand a text that writes the x // y expression where the // operator is described in the basic Nix documentation… Maybe you skimmed past some of the basic docs? . Anyway, the first time I heard about overlays the video from Nicolas B. Pierron at NixCon 2017 and the The DOs and DON’Ts of nixpkgs overlays post from Christian Kauhaus helped very much in getting a grasp of them, just maybe start by reading the post.

[Pamplemousse]

overlays aren’t really stuff for beginners

Yet, wanting to “patch” a package is something anyone might be needing to do. When a beginner wants to do that, the documentation will eventually point them at overlays, and I don’t think we should put a “BEGINNERS, GO AWAY” sign over there.

“alien” concepts like “fixed-point”

Interestingly, the first occurence of the word “fixed-point” in the “Overlay” chapter of the nixpkgs manual…
Also, I thought we had a page with terminology - or glossary, or vocabulary; I just spent 5 minutes searching for it without success (which I think is a problem in itself), so I don’t know if “fixed-point” is defined over there…

Maybe you skimmed past some of the basic docs?

This kind of condescending comment makes me a bit upset…
See previous point, and also @ksten comment: there is a LOT of documentation, too much to ingest, especially for people who might not know exactly what they are looking for.
I am always surprised that we seem to expect beginners to read and know: 1) the nixpkgs and nixos manual, 2) all the nix-pills, 3) Eelco’s thesis etc.

[azazel75]

@Pamplemousse

See previous point, and also @ksten comment: there is a LOT of documentation, too much to ingest, especially for people who might not know exactly what they are looking for.

Of course, that doesn’t change the fact that those operators are the basis on which most of the other stuff is built, that’s the reason they are used to explain more complex concepts. Nix isn’t JSON or YAML, it’s a language with most the bells and whistles of other programming languages. So if you want to divert from the basic configuration.nix tweaking it seems to me that knowing its syntax, data types, operators and builtin functions is key to understand every other concept and feature. It think that’s true for every language. Is it too much? I don’t think so. Maybe the problem is to find your way into it, but It think it cannot be a way around it. (but we don’t have to agree)

[justinas]

Do you know other programming languages, such as Python? Analogies are always imprecise, especially considering that Nix is quite an unusual language (functional, lazy). However, take a look at this and see if it clears some things up. It is an approximation of what is happening when you:

1. Define overlays (functions taking nix “pkgs” set and returning what you’d like to add to it)
2. Apply overlays (call all of these functions to get the final result).

from dataclasses import dataclass
from pprint import pprint

# Dummy structure, represents a nixpkgs derivation
@dataclass
class Derivation:
    pname: str
    version: str

    # Make a copy of this derivation, replacing some of its attributes
    def override(self, pname=None, version=None):
        # only override the attributes that are provided,
        # keep originals otherwise
        if pname is None:
            pname = self.pname
        if version is None:
            version = self.version
        return Derivation(pname, version)

# The original "pkgs" from the nixpkgs.
# An attribute set (dictionary/hashmap) of available packages.
pkgs = {
    "firefox": Derivation(pname="firefox", version="91.0")
}

# Introduce a new derivation, which is just an alias of firefox,
# without changing anything about the derivation itself.
def add_firefox_alias(prev):
    return {
        "f": prev["firefox"]
    }

# Slightly modify the existing derivation and add it to the package set
# as its own thing.
def add_newer_firefox(prev):
    return {
        "firefox-92": prev["firefox"].override(version="92.0")
    }

# Add a completely new package to the set
def add_chromium(prev):
    return {
        "chromium": Derivation(pname="chromium", version="92.0")
    }

# Apply each overlay to arrive at the final set.
def apply_overlays(pkgs, overlays):
    current = pkgs
    for o in overlays:
        current.update(o(current))
    return current

pkgs_with_overlays = apply_overlays(pkgs, [add_firefox_alias, add_newer_firefox, add_chromium])
pprint(pkgs_with_overlays)

(I intentionally get rid of “self/final” argument and only use “super/prev”, as the former is less interesting at this time. Not to be confused with Python’s super and self (this in other languages), both of which are used in classes… I use prev here for that reason).

I also think that knowing the Nix language primitives such as // or function definitions is super important. These are orthogonal to nixpkgs and overlays, but important. Same goes for override, which is again not directly tied to overlays, but often used together.

[mausch]

FWIW I’m not a nix expert but I’ve been using NixOS for about a year and a half, manage several machines on various operating systems with Nix, have contributed several PRs to nixpkgs, and still haven’t needed overlays at all.

I agree with NobbZ that it’s generally cleaner to use overrides in-place.

I also was puzzled by overlays for a long time when I started learning nix. From reading blog posts, etc I got the impression that previous “override” mechanisms were somehow being deprecated in favour of overlays. I get that sometimes overlays are necessary but I really think this concept should be more downplayed in general.

[toraritte]

I found @layus’s post Nix overlays: the fixpoint and the (over)layer cake highly enlightening, and it may have been shared on NixOS Discourse, but I found it by googling “Nix overlays”, and opening ca. 27 tabs… It also takes a shot at visualizing overlays, even explaining the Nixpkgs fix-point. (The TikZ sources for the illustrations are also available on GitHub.)

it’s a very tough slog to wade through the opaque documentation (which, though complete, is only really useful for people who already know how everthing works).

This is such a spot on observation that I just had to highlight it because the Nix* manuals are indeed not for beginners (not to ones like me anyway).

With that said, I would also like to express my gratitude here for all the contributors to the official and non-official docs, because they mostly do this in their precious free time, and the work is done with the best of intentions. Thank you!

[brianmcgee]

it’s a very tough slog to wade through the opaque documentation (which, though complete, is only really useful for people who already know how everything works).

This perfectly summarises my experience so far. Often times I’ve been spinning my wheels trying to get something to work and yes, I probably skipped over something basic when trying to ingest the NixOS manual that would have helped.

Learning by example has instead been how I’ve learned. Just reading through other people’s setups has been the most informative. It has allowed me to see in context how to solve common problems and given me a starting point for how I should organise things.