How should we handle software created with LLMs?

Well, given that Ghostscript’s -dSAFER is not likely to ever have been safe against malicious inputs but we only mark it insecure when someone bothers to report a specific exploit, sure insecure is not a super reliable marker.

Not sure we have a chance to maintain a useful exposure-weighted contamination list in the case of absence of any public legible data, given the precedents of other metadata.

(And yes, given the absence of a policy, I have by now merged at least one LLM-generated PR to Nixpkgs where the submitter did understand the feedback and did make sure it is applied — and obviously have seen PRs with which submitter was not able to do requested changes properly so they went nowhere)

2 Likes

One place where I’ve used Claude Code when contributing to nixpkgs is to create the VM test in this PR: nixos/postgresql: don't perform postgresql-setup when in standby mode by bouk · Pull Request #469863 · NixOS/nixpkgs · GitHub

I changed the module and had it create the NixOS regression test (it iterated on a couple different versions until it worked). Without an AI coding tool this PR would’ve probably landed without the test, but this is obviously better and was still low-effort. I (and the PR reviewer) still reviewed the test of course.

It’s now much easier to expand the testing we’re doing, which will reduce bugs in NixOS.

2 Likes

First LLM bot contributing to nixpkgs?

That’s appears to be just a dashboard for the nix-community update bot.

2 Likes

A dashboard that the bot appears to have made for itself: GitHub - MiniHarinn/r-ryantm-orbit: Static dashboard for r-ryantm / nixpkgs-update logs. ⚡ · GitHub

Weird bot-like behaviour: hello: trivial description tweak (EYES-bug repro) by MiniHarinn · Pull Request #6 · MiniHarinn/nixpkgs-ci-test · GitHub

“docs: drop AI-agent process artifacts”

That’s just normal CI debugging behavior for humans. That repo is literally for testing out CI, when testing out CI (which is something I try to avoid due to this) I do lots of stuff like that.

Using LLM coding tools does not make you a bot. In fact, both nixpkgs-flow and r-ryantm-orbit demonstrate that he is most likely NOT a bot, the nixpkgs-flow repo has “This site is vibecoded. Functionality over code quality, please no judgment. I am not proud of this, but it works.”, And in r-ryantm-orbit’s README: “Made with :heart: by MiniHarinn and (a little less of) his Codex”. His nixpkgs work feels human to me.

2 Likes

Hi, @yaleyoc893, I am here to clarify. I will have to say I that am not a bot/fully-automated-account ;-; (I don’t know why it have gone this far just from my side project but nvm)

First:

This is just a project that display nixpkgs-update logs for @r-ryantm sorted and categorized by possible outcome status. This project aimed to help nixpkgs maintainers to see if @r-ryantm has failed to automatically update and open pr for your package(s). This is never meant for myself from the first place, even tho it is somewhat vibecoded, but that’s definitely does not indicate that I am a bot/fully-automated-account.

This one is interesting because I was debugging something in our CI, see my pr here: workflows/{review,reviewed}: hand off review metadata via artifact by MiniHarinn · Pull Request #520522 · NixOS/nixpkgs · GitHub

3 Likes

Yes. The endless bickering over it is pointless. Having core maintainers is more important than ever. At some point they’ll just be training (unknowingly) the bot that takes over nixpkgs (hopefully) and we get https://www.youtube.com/watch?v=kHUZCVl28lg