I do use LLMs daily, only in browser chatbot form, to help me achieve what I want to code. I think it’s a great tool.
However as was pointed out on the agents.md PR, adding such a file will inevitably drown the already constrained and few maintainers with low quality PRs, which will result in even longer lead times to get “good” PRs processed.
Also I’ve gone through a rather awkward experience of contributing to a project where the author lets Claude handle 90% of issues and PRs. I found it was unpleasant not to know if I was talking with the human or the LLM or both.
I think the flipside (argh, this guy, arguing both sides) is that any label needs not to be taken as an opportunity to flame people who followed the social contract of labeling their commits.
My personal view on it is that people can choose their level of engagement, but should be able to assume respectful treatment when they decide to put a PR up that uses AI help.
More opinions
Personally, I’ve not used it to generate code in nixpkgs so far because all the things I’ve added to nixpkgs have made it seem somewhat like overkill, and there’s probably some mental overhead to adding it to the review process in the things I engage with. Also… really, a coding agent bumping a hash? We have update scripts for a reason!
This is very reasonable. The first thing I do when I go to review a package is check meta.lib.licenses because, as a personal rule, I don’t work on unfree software*. Of course, I’m not going to say anything about it, I just move on with my day. I’m sure many reviewers have their own rules for various types of software or packages that they do/don’t review for their own reasons. I would even say it’s a much healthier system than assigning unwilling reviewers code that they don’t want to be working on.
*I did, may the FSF forgive me, recently PR a change to the package for a propietary game server that I run for friends.
Another interesting side effect of the more “open” models (well, not the company with open in their name) is that they may naturally be a little more OSS-friendly (I guess, depending on what went into them too, but the weights are a start), and also could be useful for certain kinds of simple fuzzy tasks as they get better.
Anyway, I think there’s something to be said for the label being used as a positive filter for those interested in the capabilities of how these models actually perform, too.
The problem with an LLM-assisted label is that it might get filtered by nearly all reviewers, just like the stale label, subjecting the PR to the void. This in turn creates an incentive to hide LLM use
This argument applies to every possible discouragement for LLM use, and what can one say in response? Yes, that’s a possibility, but no, those of us who don’t want to review LLM-based PRs should not give up on discouraging people from submitting them.
Turn it on its head: if I tell you I don’t want to review any LLM-based PRs, and you send me one anyway and try to claim you wrote it by hand, what do you think my reaction is going to be when I catch you in your lie? In other words, shouldn’t the well-known negative social consequences of being caught in a lie be enough of a disincentive for lying on this particular subject?
(And I will catch you. I get enough of this crap at work, it’s not hard to tell when someone is being the mouthpiece of a LLM.)
personally i’d prefer a label or a checkbox i can filter for when i review, i don’t want to deal with the the llm slop nor the anti-social behavior attached to it either. The problem is that this risks incentivizing nondisclosure, making it ineffective. Furthermore it also risks fragmenting the community in two halves that don’t interact nor even check each other’s work. Yet another schism is the last thing the nix community needs
Any policy beyond “do whatever you want” in some form encourages lying. Hell, all laws do that. If countries had no laws there would be no need to lie about being lawful I understand where this argument comes from, but I think it’s a bit misguided.
Yet another schism is the last thing the nix community needs
While I hate to be the doomer, the schism is here, it has been here for a while. It needs a resolution whatever that resolution might be…
Fragmenting Nixpkgs work into merely two halves sounds like an unprecedented and improbable feat of consolidation to me! If you look who ever touches any specific file, you see that «checking each other’s work» is rarely about an at least double-digit number of contributors.
I have recently merged a PR marked as LLM-assisted where an update needed one extra line of a fix though! (which looks totally reasonable as long as it works etc.)
Honestly, this might even be within grasp of local LLMs now, although that specific PR was assisted by one of the oligopoly ones.
And honestly, maybe this diff size in lines and files could be a useful signal sometimes — using the model to find a few lines that happen to work is not that dissimilar to «I have no idea what I am doing but this seems to run» review mode one sometimes needs to apply; +200-220 LLM refactoring… the submitter should better bring really good understanding and explanations.
One underrated usage is to summarize what’s going on in another project’s crazy build system so maintainers can understand it better. There’s not much distance between that and just having it write the one line change you could have written anyway once you spelunked through all those files yourself, but we also have the commit trailer now…
I’m not a contributor and I’m a bit annoyed at the idea being able to give my opinion on something that has no direct impact on me.
But if I may try to provide an argument that I haven’t read yet and hopefully succeeding at keeping my personal position private…
I have the impression that there’s more to lose by allowing LLM usage than to gain, that LLM-interolants are more likely to stop contributing on principle than the pro-LLM would, and that benefits from LLMs would not be completely left on the table even with a strong anti-LLM policy.
While I would not want anyone to make decisions based on fear or threats, is the tradeoff worth it?
If project decisions are made based on what will make the fewest people leave, then decision making is ceded to the most-intolerant. The anti-AI contingent is very loud. That gives the impression that they are more numerous than they actually are.
I don’t think any amount of polls or arguments in this forum are going to give the decision makers an accurate idea of how the broader contributor base will react to future decisions on the AI policy, including the decision to leave it alone.
(Disclosure I guess: I think the current AI policy is fine, and it will be a few years before the AI/LLM landscape has changed enough for there to be a need to revisit it. I don’t think an AGENTS.md file should be added to the nixpkgs repository.)
They declared the following item as the most important recommendation when LLMs are concerned in FOSS projects:
The FOSS community should support, not just tolerate, those who outright reject LLM-gen-AI systems. There are many intersecting ethical and moral issues regarding these systems, many of which are not currently fully understood. Anyone who chooses to avoid them deserves our support and assistance.
From my point of view and my interpreting of the above, among my understanding of other arguments I saw when it comes to these agent files (Example in Fish shell), I think this “outright reject” does extend to these agents files which are only useful when combined with LLM-gen-AI systems.
Judging from the above, my very personal take is: only a strictly defined policy, not any agent files.
I am on the side, that LLMs are a useful tool for programming. In the case of Nixpkgs I have used Claude Code to research, how to build DLLs for wine/proton. An aspect that is poorly documented. With Claude Code it was a matter of minutes, without it, it would have taken me hours.
And for that matter, I am on holiday right now, I would have gladly spent the few hours with you figuring out how to build DLLs and fix any issues we may have ran into. It sounds like a fun project. Alas we didn’t get to do that sadly… Next time you’re not sure how to do something, please come to #users:nixos.org on Matrix or if you’re specifically doing wine stuff, reach out on Wine’s IRC room, im sure there are many lovely people there that would have helped you out.
@MagicRB In a sense LLMs are an expert at my disposal. In regards to Nix I am a bit isolated. I do not know any users
outside of this discourse. Maybe next time, I will ask for help here.
This is probably the main issue I have with LLMs! They present an ‘easy’ way forward while actually isolating you. The community is actually very helpful once you get over the fear of asking questions :). I, and many others, would love to spend time helping people understand the things we are passinate about, in the same way things were explained to us once! Please don’t hesitate to reach out next time.
We definitely do not want anyone isolated. Please join Matrix and reach out to me directly, I’m @magic_rb:redalder.org on Matrix or magic_rb.01 on signal.
P.S. this applies to anyone that needs help with Nix, wants to talk about Nix, or just get a new friend.
I don’t like it, but I think we are moving in a direction (generally, not just nixpgks) where there needs to be a two class citizen type of system, where people with a proven track record are given some leeway with using agents, while at the same time preventing people that don’t know what they’re doing from doing “drive-by” pull requests that just burden reviewers.
I understand where this argument comes from, but I think it’s a bit misguided.