On backport requests, and failing backports

samueldr · June 8, 2026, 8:04pm

Hi everyone!

While doing prep work for the LTS offering for my employer, I started feeling like I was noticing a pattern with backports and PRs with the security label. It seemed like changes that were intended to be backported often enough ended-up not being backported.

The problem(?)

A feeling’s not enough. I can’t just go with the vibe of how something feels and call it a day. So I took some time to see if I could figure out something robust to disprove what I was feeling.

To make a short story long, I coerced GitHub PR search into showing me likely candidates. This query is not by itself sufficient, since it only shows where backports happened to fail. Sometimes it’s because there’s nothing wrong, and some results are irrelevant since they’ve been handled.

What bothered me was for those where there was no follow-up.

There are two classes of such PRs with no follow-ups:

The backport failure was noticed, something was done, and there was no cross-referencing.
The backport failure was missed, and nothing was done.

The former (no cross-referencing) while annoying, is not a big deal. The necessary work has being done.

The latter (when nothing was done) could be an issue. So I took some time and went through the data to figure out if it even is an issue.

The data

I used gh as a quick and dirty way to make GitHub API queries, and jq to transform to CSV.

 # NOTE: the query was refined over time, this is only an example.
 $ gh search prs --json "id,number,title,state,updatedAt,url" \
    --limit 100 --repo "NixOS/nixpkgs" \
    --label '1.severity: security' \
    --sort 'updated' --order asc \
    '"Backport failed for"' "25.11, because" \
    'updated:>=2026-05-01' \
    | jq . \
    | tee updates.json
 # The previous unneeded `jq .` is used to pretty-print the output.

 $ < updates.json \
    jq -r '.[] | [.number, .title, .updatedAt, .url] | @csv' \
    > failed.csv

This data was then imported in a (private) Grist document. The nice thing with importing in that Grist document is that it can effectively UPSERT on columns matching, so tracking over time was not painful.

I don’t have the freshest data, since I only have a snapshot of the state on the 12^th of May handy, so take this with a grain of salt. On the other hand also consider that this is somewhat representative of a large chunk of time for a supported stable branch, 163 out of 212 days or 163 out of 181 days until the following release.

Out of 137 PRs with failed backports and the security label:

106 had a related merged backport.
7 were obsoleted or were entirely irrelevant.
4 had gotten a backport PR.
Leaving 20 packages with no follow-up.

These numbers don’t take into account the recently merged PRs. For example, at that time 3 of those PRs without a follow-up had their latest update less than 24 hours prior.

I think this is better than the gut feeling I initially had. The main confounding factor initially was the sheer amount of PRs with failed backports, and an unfortunate sampling either with no follow-up, or lack of communication with follow-ups.

What’s been done

It would be rude to only come here and tell you all about these observations.

Instead, I took some time and thought about what could be done to help either prevent this from happening (impossible) or equip the contributors with the tools to manage these failed backports.

So I came up with a small tweak to the existing backport action, which now also adds a label to track failed backports.

The main reason I’m posting here and now is to make it known what the label is for, now that it had some time to cook and show its pertinence. This new label (8.has: failed backport) is used as a signal that there is some work to do with the PR still.

Before removing the label, either:

A backport needs to be made and/or cross-referenced.
A rationale for not needing a backport needs to be added.

As usual with all things Nixpkgs, there is no obligation, no requirement to do anything with this label.

But unusually with a big project such as a distribution, there is a way to achieve “inbox-zero” with this label.

The proposed workflow:

Search for sort:updated-asc is:pr label:"8.has: failed backport"
Open the first result, which is the one that was updated the least recently.
Check what has happened in the PR since the label was added.
- If there is a rationale for not backporting or a cross-reference already, remove the label.
- If the backport target branch is EOL, add a comment stating so, and remove the label.
Search for related work. Search for the package name through all PRs: is:pr ${...} in:title. (The in:title search operator is helpful, but not mandatory.)
- If there is an existing backport of the package (for that version or newer), add a comment on either PR for cross-referencing the backport PR, and remove the label.
- If there are further updates to the package, take note of failures or discussions surrounding backports in those, and report back.
Handle the backport. Report back in the PR.
- If feasible, open the backport PR, cross-reference and remove the label.
- If unneeded or irrelevant (for example different upstream stable branch followed) report so, remove the label.
  - For security updates, a backport is unneeded or irrelevant if the security issue does not apply to the version in the stable branch.
- If there are issues or it is non-trivial, either:
  - keep the label and sum up your observations in a new comment.
  - open a draft PR highlighting the problems, cross-reference, and remove the label.

While doing all that, keep in mind that if there are newer PRs targeting the unstable branch for the same package, make sure that any failed backport for the same package up to the newest one is handled at the same time. That is, if hello: 2.0.1 -> 2.0.2 has the failed backport label, and there are PRs for hello: 2.0.2 -> 2.0.3 and hello: 2.0.3 -> 2.0.4, backport work should make sure to take all those into account, and properly cross-reference.

Also be mindful about the overlap in stable branch support periods during the month where both the new-current-stable, and the now-old-stable are supported. There can and will be cases with a failed backport to only one of the two branches.

And note that this can be made more manageable by scoping it with a topic, for example with the security label.

Early 26.05 stats

For PRs with the backport release-26.05 and security label (limited to the date I write this post), there is currently:

81 automated backport requests
8 PRs with the failed backport label
Of these 7 have been successfully made for 26.05, and only failed for 25.11.

Though this is not representative of the whole support cycle for the stable branch. This is not long after the branch-off, when backports are generally more trivial.

The only conclusion that can be made is: backports are trivial right after branch-off ¯\_(ツ)_/¯.

Closing words

Finally, I want to make it clear: this is not a big problem that should scare people off of the stable branch. While there could be some security fixes that it would have benefited from, if not from that lack of execution, it’s not a systematic failure. The few unhandled failed backports are not making NixOS and Nixpkgs stable specially more insecure. The large majority, if not all, of the issues are unlikely to have any real impact. The impactful issues are generally handled with more care, and followed-through to the end.

It is also quite unlikely that any “secure” or “stable” NixOS fork or non-fork offering will magically solve all of that. And it would be rude not to work upstream in Nixpkgs if they would.

Do not panic. Just remember: if you’re a contributor or maintainer, please keep an eye on for that new label on those PRs you have made, reviewed, or merged :).

DISCLAIMER: This post is written by a human.

DISCLOSURE: At the time this is posted, I am an employee at Cyberus Technology. While this is written on the clock, these are my own words and my message.

TLATER · June 9, 2026, 7:03am

That’s some really nice science, thank you.

I’ve heard grumblings about stable being insecure on account of backports never being applied lately, but this post demonstrates that those concerns are largely false.

Knowing this, and having data to back it up, is very helpful. I’d been reconsidering my employers’ approach to trusting nixpkgs, though some due diligence in chasing CVEs in packaged software is definitely still required, if only to help with maintenance.

sliedes · June 14, 2026, 9:40pm

Some thoughts from someone outside enough that you should sanity-check everything I say:

How does this handle the case where there is another security update needing a backport before the first one is backported? What prevents a security update that doesn’t need to be backported from hiding an older one that needs to? I don’t know if this would be feasible in GitHub, but it feels like the safest tag would be something like “needs a backport of PR 1234”, not “needs a backport of something”.
This is way more of a nix-in-general thought than about this workflow which you explain well and doesn’t seem overly complicated, just one more thing to remember: This is all pretty involved and subtle administrativia with likely evolving rules that are not always well documented and understood. I wonder if there’s any possibility to make things like this a bit more assisted so that instead of knowing when to add or remove labels, we’d e.g. see some bot adding a comment saying “this may need a backport; if it doesn’t, respond with foo and the rationale” etc, and it would take care of (and likely explain!) the administrative moves (tag adding/removal, possibly opening a different issue, …). That would presumably lighten the cognitive load, make less involved people like me way less scared of touching labels, document the intent, and allow changing the process without reeducating everyone. I wish to emphasize that the workflow you describe is in my opinion not even an important offender here

samueldr · June 15, 2026, 1:50pm

Hi! Thanks for the questions. Those might be some that other people also wonder, so they’re useful.

First, reading the questions, I think one detail that might have been missed is that what this post is about, here, is a small refinement around the existing workflow. And you’ll see in the answers to your questions how that matters.

The new has: failed backport label serves as a correct way to search for failures in the automated backports process.

Let me rephrase: “How does this handle the case where there is another ~~security~~ update needing a backport before the first one is backported”. This is the most likely reason for which an automated backport fails.

If it had been “another security backport” that needed a backport, and wasn’t backported, the same answer applies: the backport needs manual intervention.

In other words, the usual backporting workflow applies, but with a bit more cherry-picking involved.

And as far as the automated backport workflow is concerned, nothing changes.

I’m not sure exactly what situation this could be. The only one I can see is, if a given package is on a previous stable usptream branch (e.g. foo-1.2.3) on the current stable Nixpkgs branch, while on unstable it is on the next one (e.g. foo-2.0.2).

This is the second-most likely situation where the automated backporting workflow will fail.

In that case, a security fix for 2.0.2 may not be applicable to 1.2.3, but 1.2.3 may still be susceptible to a given vulnerability.

Nothing prevents that. But that’s actually out of scope for this label and automated backport workflow. This is a case where a manual intervention is needed.

With that said, the new has: failed backport label will be applied to a PR if the automated workflow fails. And that’s one signal that can be used to highlight these issues.

As for what happens usually? Maintainers and contributors would somehow notice the issue (the new label may help) and have to figure out if there is an issue to be solved and, if so, how it is solved.

I’m not sure to what this label would be applied to.

But in any way, creating one label per (security) PR wouldn’t work out, and I don’t think it would be useful. There is no PR created for an automated backport that failed, so there is nothing to apply that label to anyway.

The fact that there is a backport failure generally implies that there is another change that needs to be backported.

It would also be hard to handle. Figuring out which changes need to be backported to get a security fix outside of trivial cherry-picks can be non-trivial. So the label would need to be manually created, and by that time, you’re outside of automated workflows.

If it was possible (and safe) to automate this, there would be no need for a label: the automated backport could combine those required changes.

Anyway it’s much safer for the relatively low amount of such situations to have someone involved in analyzing why this trivial automation failed. It is likely enough that there are be security implications from the failure.

Paraphrasing a bit,

There may be some things that need to be improved, but given the tooling available (GitHub) there isn’t much more that can be done in term of automation that I can see off the top of my head. But I’m also not an expert in that specific topic, so anyone feel free to prove me wrong.

The main reason I think so is because there’s not much available for attaching structured data into the GitHub issues or PRs, so that makes it hard to properly attach information between different PRs and issues. So that leaves mainly comments and labels on a single PR. Labels need to be re-usable between different issues and PRs, so nothing much can be personalized to specific issues.

That leaves comments, and as far as the security workflow goes, I don’t know if there’s any useful automation here.

With that said, I have been and currently am thinking about that problem, but have nothing in mind yet. More to come whenever I have had some thoughts on the topic.

Hope these helped. Feel free to ask for more clarifications, especially if you think I misunderstood one of your points!