Call to ban AI commits

This thread is getting uncomfortably heated.

It is okay to believe that no output of generative AI has any value.

It is okay to believe that some outputs of generative AI do have value.

It is okay, though I advise caution, to use the term ‘slop’ to refer to undesirable AI output. People who hold the first belief will naturally use ‘slop’ to refer to all AI output. If people who hold that belief and people who hold the other one wish to have a productive exchange with each other, I advise not using the word at all.

If you aren’t interested in having a productive exchange, please refrain from having an unproductive one.

It is not okay to attack each other. Criticize beliefs all day, but when you find yourself getting frustrated with the opposing point of view, please take a break to cool off.

30 Likes

The most morally opposed or outraged should not be given undue weight. Their votes should not count more than others just because of how strongly they feel.

9 Likes

I care relatively little about most aspects of this topic*, and I would prefer the votes of the morally involved (in both directions) to have higher weight than mine.

(* Just in case anyone cares about me personally, I have lots of opinions about AI; it’s just that they don’t determine anything about what’s at issue here.)

I think this largely already occurs. If someone subjectively does not feel that they have a strong opinion, they are less likely to vote or answer a poll, thus providing a mechanism for weighting the results of said voluntary vote/poll.

2 Likes

Thank you @rhendric, for keeping things grounded.


(I’ll be speaking as myself, not for the entire SC, and I deemed it worth clarifying my stance.)

The overall topic is a useful one and there are many echoes here to what was discussed during the development of the AI policies. I see these threads as raw input to understand or encounter new viewpoints. So far, I can say these viewpoints have been heard and were considered.

@cafkafk laid out a reasonable framing of the various subtopics; I agree with much of what was said about the resulting pragmatic outcomes. I don’t see a full ban or direct strong stance against AI to have a reasonable chance to become policy at this time. That is not to say I’m dismissive of that perspective, but I do not believe the Nix ecosystem is the appropriate venue to take such a strong stance or to be the test bed of that approach. There plenty of other institutions that are designed around the goal of pausing AI usage, are much better equipped to host such debates, and are more effective at driving toward those stated objectives.

There are serious risks. Many projects have become overrun by how cheap it is to waste people’s time with AI. Some have decide to be open source, but closed contribution. Some project a hostility to AI which may have the desired effect. Other have gone with full adoption - jumping into the deep end with steadfast courage. I’m inclined to have a general preference for human contribution and interactions.

On the other hand, the quality of AI output is a rapidly changing. Whereas two years ago I actively discouraged the use of generative AI for Nix related things due to poor quality, I’ve come to see the quality improved. Starting with code-completion, then boilerplate generation, then analysis/investigations, maintenance work by an agent, and a growing scope of tasks that help me do my job effectively. So pivoting back to the Nix community itself, I start to consider Automation over process and toil and ask if there are ways to leverage AI in ways that have less downsides? Rather than as a generator, but as a watchdog to flag suspicious commits? To automate tasks too fuzzy to directly script? Should we provide agents.md or other well-known artifacts so that when people do inevitably use AI, it can be given constraints that WE decide on, leading to fewer inadvertent violations of our policies? I’ve wondered what a parallel “ai-nixpkgs” would accomplish? Could it autobump and manage the package set, flagging upcoming critical breakages and PoC’ing security fixes to make the work on nixpkgs easier?

I voted for the status quo in the poll above because I also approved the current stated policy; it is an attempt to balance the risk + damage to the ecosystem with the potential benefits. We have yet to see how all of this plays out, so I suspect our policies will shift over time. Let your SC reps and various teams know if there are substantial changes that should spark a reconsideration of the policy.

11 Likes

This ignores the all-you-can-eat buffet of reasons why GenAI technology is awful and just focuses on the quality issues. Even if those were completely fixed, GenAI is still fundamentally horrible.

It’s still a regurgitation machine taking in massive amount of human outputs and outputting something completely soulless. It strips the humanity from everything.

There are still all of the environmental issues. All of the social issues. Skill atrophy. Ties with fascism and eugenics. So, so many other things.

The quality issues are frankly irrelevant when talking about the way more serious issues.

11 Likes

I hear where you’re coming from, but Nix isn’t an organization designed to fix any of those things. Paraphrasing the values listed on the governance page, it’s designed to create and maintain free, purely-functional software deployments. Maintainers’ concern should be with whether commits make good, necessary contributions to Nix projects, not who wrote them. I’m not even an gen-AI apologist, and I have never “vibe-coded”, but to ban all gen-AI assisted commits over concerns on which there is not a general consensus would alienate a large part of our community, violating the principals of open source, open contribution, software.

I reiterate that I would not consider myself a gen-AI fan– I also have moral quandaries about the development, current use, and future of gen-AI, but when we’re expecting people to contribute their own time and effort to Nix projects, banning them from responsibly using tools that can significantly reduce their workloads seems cruel. I voted for the “human in the loop” approach because it feels like the most reasonable compromise between those hesitant to accept AI-generated code and the all-in do-everything-with-AI people. No one’s going to be completely happy, but this approach enables contributors to save some of their own time, while also keeping the burden of making sure their code is good on them instead of reviewers. I wouldn’t be against something like a yellow/red card system for people with poorly written, clearly unreviewed gen-AI PR’s, but banning all gen-AI assisted PRs outright probably puts us on the wrong side of software history.

9 Likes

Briefly I return, to give some corrective:

I am concerned with a new and prevalent perspective.

The idea being that it isn’t our concern,

Of how Nix is made (or used) in turn.

This is erroneous, and for reasons plain,

As the responsibility could only be ours to fulfil, or abstain.

Therefore the idea that it is not our task,

Is factually wrong, despite how policies might this mask.

To be more curt, so the message is clear,

It is our job to discuss these effects, now and here.

The dismissal otherwise, I will ignore,

As it is bad-hearted (and rhetorically a bore :stuck_out_tongue:).

Just to break verse, vomit non-withstanding,

Here is a summary to complement understanding:

There has been a lot of comment about the scope and salience of Nix’s responsibility to even moderate these things, or its feasibility. I propose this is a bad framing of the issue.

First, the matter of whether something needs to be moderated should come before whether it is easy to moderate. This is hopefully obvious (not every crime can be prevented, but that does not make the act non-criminal). So let’s drop that for now, as it really has no space here.

Second, there is an idea presented above that this impinges on the openness of open source. To that I remind you the GPL also impinges on your rights to open source software, for the benefit of the project. Indeed, the intent to ban AI is based on this same logic. So away with this too! :slight_smile:

Thirdly, there is the idea (only approximated, not articulated) that the list I provided before is a list of particulars. In response to this, I latched on to the NixCon example before.

To my best estimation, none of my points have actually been challenged, just compared to other issues. This is not a refutation.

To see this is not a refutation we can note that, for example, Nix has ties to military companies. But the fact that Nix has ties to the military industrial complex should not give us the licence to be laissez-faire about adopting another questionable industry link. The morally conscientious approach is instead to keep stock and “not get worse”, in other words. This same rubric can be applied to all responses of this type that I have read.

Fourthly, there was some comment about how this affects access to obscuring text, and that me being against consentingly adding text to bots is in turn against good OpSec. This is not the case. Good OpSec changes dependent on the environment you are in: so again, it is a secondary concern.

Lastly, and possibly most interestingly, there was a practical conversation about how this would even be done. I had sort of neglected this thread (apologies) but I think the serious answer is that it will change over time. Several other projects are taking a hardline stance against AI commits (Gentoo and Zig come to mind, for example), and I fully expect the process to be worked out amongst the myriad communities that are all taking hardline stances against this stuff.

In particular, I think it would operate in a manner similar to what was suggested, in that its less of a “mechanistic rule” and more of “the right to reject PRs if they look AI”.

There was a response above that this would be a regression from the current policy in that it would incentivise bad actors to just not disclose that they are using AI. Which is almost true: except bad actors wont necessarily disclose with the current policy either. And so we are back to the idea that the policy is not actually practical to begin with, but is more of a statement about what the community expects.

Deepest apologies, the above para’s were vile,

I think I will be sleeping with a bucket for a while.

I hope that this summary helps contextualise my measures,

And that there is peace with no need for displeasure.

5 Likes

I have a question, why don’t we do these two things at the same time?

I mean, nixpkgs can set up two rules at the same time: one is for contributors submitting PRs, telling them not to use AI; and the other is for contributors reviewing PRs, telling them not to block it just because the PR ‘looks like vibe code’.

The logic is like this:

Based on my understanding of this thread, we’ve at least reached some consensus:

  1. AI code have caused a lot of trouble, like low-quality code and some ethical issues, but mainly it’s the low-quality code.
  2. AI code is really hard to tell apart, and nixpkgs reviewers don’t have an effective way to distinguish AI code.

At the same time, I also want to point out some facts that weren’t mentioned in the thread but are obvious on their own:

  1. AI code just makes low-quality code problems worse, but they’re actually two different things.

Based on these points, I assert:

  1. According to 1 → Ideally, banning AI code would be good for the health of nixpkgs.
  2. According to 2 → No one has the ability to tell if a PR uses AI code, so no one should have the power to block a PR just because of ‘using AI code’.
  3. According to 3 → Raising the code quality requirements for PRs is unquestionably a good thing, and we should do it even without the AI issue.

Based on assertions 4, 5, and 6, it’s not hard to see that carrying out the corresponding measures doesn’t conflict; they can be done at the same time:

  1. According to 4 and 5 → I suggest that nixpkgs set both of the rules mentioned earlier, requiring that AI code not be used for PRs, but in reality, reviewers don’t actually check whether it’s AI code.
  2. According to 6 → I obviously agree that we need to raise the code quality standards for PRs, and filtering out most AI-generated code is a nice side effect of that.

I think doing this can bring some benefits:

  1. According to 7 → Reviewers don’t need to put in extra effort, because there’s no real requirement to check whether the PRs was generated by AI, and in fact, it’s not really possible right now.
  2. According to 7 → Nixpkgs has successfully avoided AI ethics issues. If a contributor decides to deceive the community and use AI, then that’s now their own moral problem, and they can’t excuse themselves by saying, ‘nixpkgs didn’t forbid AI code!’
  3. According to 10 → Going further, does the community really need a contributor who deceives others to use AI-generated code? A user who uses AI to generate code and still meets the code quality standards at least has enough communication and learning skills. If they feel guilty, it will push them to stop submitting AI-generated code; if not, then the community doesn’t need a user with such low morals.
  4. According to 8 → This policy can already deal with over 90% of low-quality AI-generated code. So I think 9 is reasonable, and nixpkgs doesn’t need to use a more aggressive standard to review AI-generated code.
  5. Additionally, I think a clear rule like ‘no AI-generated code’ handles some toxic rare community situations better than a vague standard like ‘humans in the loop’.
Explain 13

This is my personal experience. Maybe this situation is somewhat rare, but it’s definitely disgusting enough.

Vibe code users are a very broad group; I agree that there are senior programmers among them who can ensure code quality, but the vast majority are beginners.

Vibe code users also usually have unusually high efficiency and enthusiasm, and they submit a lot of code, regardless of quality, which 100% increases the burden on reviewers. There are 10k+ PRs pending in nixpkgs, and the burden on nixpkgs is already serious enough.

Sometimes, some vibe code users who lack programming skills just act as messengers between reviewers and agents, which makes communication extremely difficult; in rarer cases, some idiots with no understanding of programming submit really terrible code, refuse reasonable modification suggestions, and attack people throughout the community. In the past, these people would be kept outside the ‘programming ability’ barrier, but now these people are everywhere……

If the community clearly doesn’t encourage AI-generated code, at least it can quickly ban those toxic users.

1 Like

I see that the Nix Homeowner’s Association is still in session.

Have we made any progress on the backlog? It seems odd to worry about industry issues from 2023–that we can’t do anything about–when we still have open issues in nixpkgs from a decade prior–that we can.

Over a third of our packages are unmaintained. We need all the help we can get. LLMs give us some resources to help fix that. Arguing about LLMs takes away from the attention economy that would otherwise be used to support the actual work.

Many of these posts arguing one way or the other are longer than the commands that would be required to actually update or fix or test these packages.

Y’all, this is embarrassing.

8 Likes

I see that the Nix Homeowner’s Association is still in session.

Have we made any progress on the backlog? It seems odd to worry about industry issues from 2023–that we can’t do anything about–when we still have open issues in nixpkgs from a decade prior–that we can.

Over a third of our packages are unmaintained. We need all the help we can get. LLMs give us some resources to help fix that. Arguing about LLMs takes away from the attention economy that would otherwise be used to support the actual work.

Many of these posts arguing one way or the other are longer than the commands that would be required to actually update or fix or test these packages.

Y’all, this is embarrassing.

Yes, if LLMs were the right tool to fix the backlog then the discussion about banning them would be ridiculous.

You’re not demonstrating how enlightened and pragmatic you are by simply taking that opinion as unarguable fact and pointing fingers because other people are wasting time.

Or to adopt your style of arguing:

Have we made any progress on the backlog? It seems odd to even argue about LLMs when their harmfulness is proven and we already know what we can do about them: An absolute ban.

How can people try to defend their use when we still have open issues from a decade prior?

We need all the help we can get and supporting these harmful machines alienates the principled and talented people that actually care about code quality.

Arguing against the obviously right thing of banning them takes more time than simply accepting the ban and letting the LLM-Sycophants lie about their use. If they’d be found out, we can kick them out after the fact, if they’re not, their deceit still lead to code that at least appeared to be acceptable.

Y’all, this is embarassing.

I’m sure you’re very convinced and the matter can now be put to rest. Everybody is better off.

14 Likes

Don’t you worry, I am aware,

Of how poorly updated some packages fare.

I intend to participate in maintenance,

But I still care about this topic’s sustenance.

In a sentence, “why not both”,

Neglecting issues does not spur growth.

6 Likes

@Grunty I am sorry to bring this, but as a non-english native, following your answer has been a big difficulty that let me out of the thread. The structure you are using, that i’ve not been exposed to, is a non negligible barrier for comprehension. In fact, i have to read them multiple time to extract their meaning, and it makes me feel like i’m 5 years old starting to read :sweat_smile:.

If i understand properly, this is way to express yourself that is deep tied to you, but in a space like this, i hope you would be willing to make your argument accessible to most.

14 Likes

Bonjour Sigmanificent, je m’appelle Grunty,

Je connais parler français ici!

Mon argument, en anglais (sans vers), est simplement,

Voir ci-dessus, maintenant: Call to ban AI commits - #69 by Grunty

4 Likes

Since the original question calls for the ban of all AI commits and I have only seen one person tangentially mention it before, what about local LLMs? What if someone uses an quantized open source model on their own hardware?

Most of the stated concerns with LLM apply to the provider more than to the actual models themselves after all.

Sure local LLMs are less powerful than the top frontier models, but from what I have read we are at or close to the point where they might form a viable alternative for scoped yet non-trivial work. The concern with quality is one which is more prominent due to the constraints with consumer hardware, although setting up such a workflow is not as easy as buying a subscription service, resulting in a higher barrier of entry. It seems less likely to me that someone who tinkered with something like this is producing thousands of slop commits and PRs.

Lastly, a bit of a hope of mine, is that a ban of non-local AI might higher the demand of open source or at least open weight models, resulting in more industry interest in creating such models. Training an LLM on open source code and then publishing its architecture and weights is, at least for me, in line with the philosophy of Free Software, although I am not sure how others see it so I am interested in anyone else’s viewpoint on this.

3 Likes

There are techniques to make models do less crap.

They work much faster than us if they have a good guidance and can deal much more efficienly with some annoying iteration cycles we have to do. W

We can take some lower hanging fruits so even a person that just toss a problem to an AI can give a decent output.

4 Likes

My perspective on that matter: I oppose a prohibition on using LLM. LLM is a very diverse subject with many different way to use, and, even thought I sparingly use it, I wouldn’t want someone to prohibit LLM uses when it does not significantly impact the end result (as i believe that everything that happen in a local Git repository is irrelevant and does not need to be moderated, given it is 1. Never shared online and 2. How someone prefer to organise their work is their personal choice).

I would also like to share how good are LLM at spotting bugs (at times). I can point an LLM to some code that do not work, and it sometimes figure out what is wrong very quickly (and other times get it wrong entirelly, which lose me time, thought it is usually in an obvious looking way. And then sometimes looks obviously wrong but end up actually being correct after manual debugging). I foundly remember the day it spotted an incorrect < instead of <= (or was it the inverse?) in a for loop in C some year ago.

I think any use like that, where you take hints from an LLM but no LLM output ends up directly or literally-retyped in the final commit, would not be covered by definitions of AI commits that have reasonable chances of adoptions in a policy.

2 Likes

I think any use like that, where you take hints from an LLM but no LLM output ends up directly or literally-retyped in the final commit, would not be covered by definitions of AI commits that have reasonable chances of adoptions in a policy.

This would still violate the spirit of a potential “absolutely no AI usage allowed” policy and discourage this kind of usage in benign contributors. Similarly such policy would also discourage any potential contribution from someone who uses AI in any capacity.

This brings me to another point, that I think is important to discuss. While this thread has greatly reasoned about the merits of LLM written code or lack thereof and whether or not it would be good for the future of nix to allow such PRs, ultimately nix can take the position of disallowing AI commits. Regardless of whether or not LLMs are actually productive or not. That is however not the case with Security.

To my knowledge, LLMs are responsible for the recent flood of CVEs and do worthwhile work in that field. If the maintainer of curl, who famously shut down their bug bounty program and whose project was among the first to be targeted by slop prs now uses them extensively and has this to say:

We have not seen any AI so far report a vulnerability that would somehow be of a novel kind or something totally new. They do not reinvent the field in that way, but they do dig up more issues than any other tools did before.

[…]

The AI tools will improve further and the researchers can find new and different ways to prompt the existing AIs to make them find more.

We have not reached the end of this yet.

I hope we can keep getting more curl scans done with Mythos and other AIs, over and over until they truly stop finding new problems.[1]

It seems like at least in this area, LLMs provide actual value. The problem here is, that while we can prohibit use of LLMs for contributors, malicious actors looking for vulnerabilities obviously won’t abide by this and use whatever tool at their disposal, which will include LLMs. A potential policy banning AI applies to AI assisted research and can have a noticeable negative impact to the security of nix projects.

[1] Mythos finds a curl vulnerability | daniel.haxx.se


  1. Footnotes ↩︎

3 Likes