Frustration mounting, I steam like a kettle,
With the current direction Nix’s AI policy has settled.
In response to AI commits, I ask we show,
the extent to which this community says “NO”.
I’m the type of minor contributor who tends to live under a rock, less in touch with the community than sending PRs, reading chat and issues — this is probably only the second time I’ve actually talked to someone outside of a GitHub PR. So I often miss most things until after they’ve already happened in the NixOS space.
Honestly, I’d rather not pick a side if things get heated, but the core team has already made their decision and the policy looks pretty okay to me. Let’s move on.
“Let’s move on”, I find amiss,
The point of this post, is orthogonal to this.
The judgement made gives me grief,
and I raise this point to display my belief.
If you do not respect my right to air my frustrations,
I shall in turn dismiss your negation.
AI commits are a risk intrinsically,
So I am not satisfied with “a committee approves of AI wittingly”.
Banning AI entirely is too unenforceable. At least with the given policy you can test someone’s understanding of the code by talking to them. It means something in practical terms.
AI isn’t the problem. Slop is the problem. I see no reason to use those tools myself, but I acknowledge that some workflows involving such tools can produce results of sufficient quality. Banning them entirely seems to me like an irrational, emotion-driven overreaction.
I’ve yet to see that in the last decade, unless you intentionally lower the standards so that slop passes.
I’m far less certain there are any such workflows that are actually gaining productivity from the tools while producing results of sufficient quality, but certainly the involvement of the tools doesn’t inherently make the result slop.
Also, it’s worthwhile to leave room for the future, given that this field is still rapidly changing.
Given the upstream contributor base a full rejection is almost definitely impossible, even besides the fact that bans are unenforceable. This policy is probably the most sensible compromise possible.
I think it’s more important to see where lix goes.
Lix has always seemed like the fork that takes a step back and thinks critically instead of doing whatever commercial interests pay for. They’re probably the people who are more likely to see value in the creative process and its impact on how you think about and design software.
The fact that you need to go a little out of your way to find it also means that your average slop kiddie won’t try to contribute to it, so a ban on trust basis is likely more feasible.
Also, frankly, unless you contribute to nix itself, probably not your place to complain. If you do contribute to nix (or want to help out more actively) and you’re upset about this policy, see if you can pick up some issues on the lix repo.
This may be slightly tangential, but is is very useful to see polls like this as a member of the SC. So from my perspective, insofar as the positions made here are actually civil and productive, it’s at least useful to me.
Your poetic skills aside, and with understanding that you’re frustrated, your strongest position here will likely not be “the community completely agrees”, since that’s not the case.
On the other hand, there is apparently at least 34% that do agree with the maximalist position, which is a substantial amount.
I think that when I see the other voices in this thread, they reflect some of the same concerns e.g. the Steering Committee has had when working on policy around LLMs.
For me, one of the most interesting open questions to the segment of the community that’s interested in a blanket ban is: How would one actually do this in practice? It’s by contradiction near impossible to positively prove something is made by an LLM, and the counter to that, it’s very easy to accuse others of using an LLM.
For me at least, the risk seems to easily become that novice contributors, specially ones that don’t know other people in the project, would easily be accused of using AI. I could easily see this becoming a reintroduction of the problems we’ve previously had with overly nitpicky and unresolvable reviews.
And what I personally landed on when I thought about my own reservations with LLM code wasn’t in itself that it was create by an LLM, but the quality of the code itself. And unlike “whether or not something is made by an LLM”, I can actually see it possible that one can review "whether or not the quality of a PR meets the bar for a given project.
That’s already one of the goals of code reveiws in the first place.
So perhaps, a final question is what examples are of verified failure modes that has been observed here are, where has LLM generated code actually lead to introduction of harmful code into a NixOS project?
With that, I think it would be much easier to advocate the interests of the 34% that are looking for a stronger stance here.
In the same vein as Christina’s post, I’d be interested in teasing out the different factors at play here.
I’ve recently written a number of AI-assisted PRs for Nixpkgs, as well as a couple for Nix and nix-index. I use AI to try to understand the surrounding context for a change more fully, and to gather URLs I can put in code comments and my PR descriptions to try to make my changes as easy to review as possible.
In my view, one huge benefit of the current AI policy is that it incentivizes disclosure of AI usage. This is good for everyone. So I’d be curious what additional benefits we could hope to gain by making the policy more strict, at the cost of disincentivizing disclosure. In my view, making a strong argument for that would pretty much need to include concrete examples of AI being “intrinsically” harmful regardless of who wields it. I humbly submit my contributions linked above to be criticized for this purpose.
While I understand this view, the final product is not the only aspect that’s worth considering. The old saying goes - code is a liability, not an asset. I care more about someone having thought about issues and having them in their mind than about the text they wrote.
There are plenty of gorgeous haskell codebases with a bus factor of 1 because nobody else understands it.
LLMs put us in this beautiful world where the bus factor edges ever closer to 0. We can keep going on about individuals using them responsibly, but the science shows (including studies by anthropic, who have incentive to be dissenting voices) what we all know deep down - the net effect of even limited use is a reduction in individual performance and understanding.
It’s not just the lack of understanding of the actual commits you’re sharing - it’s understanding the problem space and wider exploration of what you’re working on. Getting easy answers surrenders all of the learning and experience you’d normally get. This is why a “do you understand this code” litmus test isn’t sufficient in my opinion - by definition, the use of an LLM has led to you not understanding everything you otherwise would have.
I don’t care for nix having all the features today. I’d much rather live in a world where we have 20 people adding the features sustainably next year, and retain the ability to do so responsibly and intelligently. LLMs will never solve the difficult issues, the training data does not contain silver bullets. We still need people to be able to read and understand the code, the risk I see is that that disappears.
I also have concerns about the actual quality of the artifacts generated, and people’s willingness to review qualitatively if nobody actually cares anymore, but that is secondary and much less objectively provable.
I have no doubts that we will just “move on”, at least until it becomes much more expensive to use these tools so carelessly, but since you asked what people who’d prefer we didn’t do that object to, the above is it for me.
Nonetheless, I do think at least for security review a zero-use policy would be irresponsible. While fuzzers perform similarly, LLMs are much easier to use, and the effects are clear.
I also understand that commercial interests are much less interested in waiting a bit longer for features to have a more sustainable project. This is why I ultimately see zero hope for sensible engineering in the long term. It’d be nice if more community-driven FOSS projects remain, though, and it’d be bloody nice if nix was one of them - doesn’t look likely, though.
A valid concern you have placed in light,
Is whether a commit will be shunned if not “quite right”.
False accusations are not my desire,
And (given my name) no witch-hunt is required.
Disclosure helps visibility, that much is true,
But “humans in the loop” will bring more AI into view.
For if an AI coder saw they could commit openly,
It would encourage them to commit more, and brazenly.
You also ask plainly what risks now exist,
To which I break verse, and respond via non-exhaustive list:
There is more, but my eyes feel like lead,
So I shall nap, please consider what was said!
To perhaps make my long answer short and sweet, my thinking is that quality includes notions such as not making things overly complicated, making them easy to review, having clear explanations of what you’re doing and why it matters, and the code quality itself. And all of those, I honestly wouldn’t assume that an LLM could produce without strong involvement from a human in the loop.
In that sense, I’d hope quality would gate all those, and if not, I’d ask for more quality, since I’d equally want this for the non-llm contributions. Of course, with understanding for new users having to learn.
Yeah, I think that is true, but trying to assert this would mean much more strict review than we currently have (at least in nixpkgs, I look at nix far less). You say so yourself - people don’t like nitpicking, and a lot of the quality gates here will feel like nitpicks.
If more senior contributors start adopting LLMs as well, or buckle under the increased load, I seriously doubt there will be capacity for this. There already isn’t sufficient capacity for just human submissions.
Anecdotally, even heavy contributors seem to have issues with this - I think here is an example of someone being happy enough to be done with the work to not consider the why very much, only being caught because it happened to go to discourse: [RFC] Nix function that overrides a scope, with automatic inheritance propagation
Arguably this is an example of the happy path, and my entire argument is very slippery slope, but there is plenty of evidence of other communities (especially commercial companies) slipping down that slope.
I don’t disagree with any of these, but I’d rather say these are all risks we are already taking on:
To be clear, none of that is a “gotcha”, it’s more of an actual brain dump of how I see this in total. I think that the risks you’re showing are all valid. I also think most are not unique. But I also see that LLMs manage to package a lot of bad things into a neat package, and that it is a problem.
With all that. I’m still unable to see an actual policy that simply says “LLMs aren’t allowed” that would be effective at solving these problems. Or would at most solve some of those problems, partially, while opening up other massive risks.
That’s how I end up landing on the problem that’s a through line, unique, and SOLVABLE at our current structural scale and influence being quality. That said, I can see arguements for us endorsing this being bad by association, I can see inherent moral arguments about “this is just really harmful in so many ways”, and the legal aspect is also one I always personally felt was a pretty deep cut, as someone that is very over the top about wanting clean lines for my licensing and my codebases (and who was deeply burned e.g. by GitHub deciding my private repos could be trained on, by the fact that a lot of code I’ve written for free has trained billion dollar companies without any sort of kickback after decades of copyright law being harmful to consumers but ignorable the moment it was profitable).
Maybe I’m just having a too pragmatist stance on all this?
I think I agree with you, but I really want to just give context on this line. I really honestly think that there is a big different between nitpicks and quality gates, and that that is an extremely important part for us to get right, regardless of how hard the line we set for LLMs are.
There is to me an obvious spectrum from purely stylistic things that e.g. a formatter could catch to structural malfunction or improper architecture. The hardest middle for me is the quality gate for when and when not someone would have an obligation to e.g. consider something of a quality worth a review.
My working mental model for that was simply that we’re all volunteers here, and so, we should all be able to say, even if it isn’t a falsifyable claim, that a PR is too big, isn’t described well enough, or is too far reaching or similar, that we personally just wont review it.
If someone else wants to review it, that review is on them, and as the person that potentially approves it, they’re partially on the hook for introducing bad quality code with the author of the PR.
It vaguely covers most practical problems I’ve found with LLM nowadays in open source (and is how I’ve found a good working model at my dayjob to clear the review queue while enforcing a required bar), but it’s very much a working model.
But I’ve also found when talking, specifically with NixPkgs maintainers that one of the things they dreaded the most about LLMs was feeling obligated to review gigantic slop PRs. And whatever policy we land on, that’s one of the things that’s really important to me to make sure doesn’t happen. That no one feels obligated to review hard to review PRs, and that no one can act entitled to a review, specially if it’s of a poor quality. No policy should burn out others as a downstream effect of their own lack of care.
I said I’d nap, I promise I will,
But if I dont comment just one thing I think my brains will spill!
You said that we use GitHub already,
But I would love to challenge that too, when more steady. 
To your other points, their persuasion descends,
The presence of one failing does not addressing another, prevent.
We can start with AI because it is an easy win,
Then discuss this NixCon matter and the associated energy sin.
Rather than preemptively surrendering, on a basis of guilt,
I propose we act with purpose, and pride of what has been built.
(You are not alone in wanting to challenge that, and it’s also a case of one battle at a time, but enjoy your nap!)
I didn’t read the full thread, but I’d like to defend myself a little bit. It would be false to state I “did not consider the why very much” - I opened the discourse thread precisely to discuss the “why”, and the AI produced code was so complex which is what made me ask the why. Note also the following PRs that came after the fruitful discussion on that discourse thread: Pull requests · NixOS/nixpkgs · GitHub . All of these PRs were written completely by hand.
I’m largely wearing my non-pragmatist hat in this thread. It’s ok to be a balancing factor. That’s the point of discussion.
I’m very happy that you’re engaging faithfully with it and address the concerns, and acknowledge that they have merit. It’s nice that you even engage with the less directly technical points that we usually brush aside since, y’know, who likes politics?
I dread this future we’re building. It’s depressing to see that nobody cares to understand the world. My passion for the software world has completely dried up. I know some LLM will vaccuum up any code I write and sell it back to me in an NVIDIA-powered surveillance box that my landlord forces me to install, and I will have paid for the pleasure because my employer forces me to. Meanwhile the people I used to collaborate with in my free time don’t even care about their own work enough to do it anymore. Let alone all the other long-term problems the world has not been coping with that the LLM craze intensifies.
At the end of the day I’m submitting to the powers that be. I’d love a world in which we could all stop and think for a second, but that’s not the world we live in. “Pragmatically”, money speaks. But I appreciate that just for this moment someone is at least acknowledging that this sucks.
To be honest, Nix* is a bit of misguided hope there: binary cache is not done optimally but even done right it would cost significant amounts to run, so the nice infra many in the community want means some acknolwedgement of corporate interests across the ecosystem.
Like, I am personally fine with the project actively asking Amazon for sponsoring the cache (dependence is sad, but it’s a lot of work to fix, and I am not going to do that work, so oh well). Even if I would prefer a legal regime under which the definition of manslaughter and enforcement practices would lead to Amazon being found guilty of manslaughter as a corporation. (I don’t think that sponsoring Nix actively helps Amazon get away with any of the things I would prefer it being hit hard for) But in a sense, it’s that bad.
Way too many people consider misguided at least one technical decision in Nix pre-ChatGPT-service-launch, though.
Opening a discussion and bringing something to play with the design implications sounds good to me, even if the prototype is not checked carefully for quality.
I think we are inching away from nitpicking, and never do it enough; so here we will err in the correct direction — if people try to read and sincerely fail, they will most probably ask for explanations and improvements.
Are we talking about problems with modern LLM deployments in general, or about problems that are relevant to a significant number of proposed Nix* patches? I agree that some exceptional cases need care, but the typical case is «I ask LLM to do some work for me, based on public data and my preferences, then I will look at the work and publish it as a proposed patch». Data privacy issues for that sound significantly less important than for median use case of hosted LLMs.
It’s not just the question of GitHub having worse issues from data privacy and identity control and stuff (which it does have), it’s the question of the policy covering contributions submitted for review in the open. Sure, some internal workflow details can leak, but that’s the user’s problem (and LLM use for code navigation without using LLM code output, which is unregulatable from enforceability point of view, will still leak comparably).