Can we explicitly ban LLM-generated PR descriptions, commits?

Development
1

Today I came across these walls of text:

The first PR description is 804 words according to https://wordcounter.net . The second is 660. The commit messages total 550 and 1027 words respectively. The PRs have single comments that are over 150 words long.

I don’t think it’s a wild take to assume that this contributor has not read or understood either of these PRs in their entirety. To be honest, I doubt they’ve read the code. If someone left a review, I think the review would go straight to the LLM and the response would be fed back into the GitHub UI.

While these PRs may be in compliance with the letter of our AI/Automation Policy given that there’s an Assisted-by footer, I strongly believe that they are in violation of the spirit of the policy and are inherently extractive contributions.

Obviously, the community is very divided on the current AI policy. There are folks that would prefer a total ban, folks that want a middle ground of sorts (likely inline with our current policy), and I’m sure there are some folks that would prefer a free for all. Given that the Steering Committee may hold a referendum on the matter soon, I don’t think this is the place to have this discussion.

The discussion I would like to have is banning the usage of any and all LLM-generated output in anything but code. Including, but not limited to: code comments, commit messages, PR titles, PR descriptions, PR comments, responses to reviews, comments on issues, and whatever else. We can amend the current policy, so please, let’s do it. We’re not beholden to the current for all of eternity.

Not only does this prevent situations like the above two PRs (which I imagine will probably never be reviewed or merged), but it requires that contributors actually understand the code that they are submitting for review and keeps the human in the loop.

LLMs allow for people to submit PRs that they don’t understand, which places the burden of understanding (and all of the effort) exclusively on the side of the reviewer. By definition, this is an extractive contribution. This also erodes any and all trust the reviewer has in the contributor or their capabilities.

By requiring that contributors use their own brain and words to write code comments, commits, PR metadata, etc. we can reduce the burden on our reviewers and ensure higher-quality contributions to Nixpkgs.

There are other notable open source projects with similar policies. Namely:

54 Likes
2

The pull request template literally starts:

<!--
^ Please summarise the changes you have done and explain why they are necessary here ^

For package updates please link to a changelog or describe changes, this helps your fellow maintainers discover breaking updates.
For new packages please briefly describe the package or provide a link to its homepage.
-->

Yeah, the idea is that it’s a summary. In other words, a short description of the change that describes why you feel the PR is worth merging. (Ironically, LLMs tend to be excellent at summarization, but people still use it to submit walls of text for some reason).

11 Likes
3

I don’t think it’s a wild take to assume that this contributor has not read or understood either of these PRs in their entirety. To be honest, I doubt they’ve read the code.

As the author, I fully understand both. Happy to answer any questions you may have on their substance or merit.

I’ve actually read every line carefully and done multiple passes on the implementation.

As for the summary, the first paragraph is a TL;DR, and there are additional paragraphs of detail, LLM-drafted (as disclosed), that I included because I believed they would be helpful to a reviewer of a change of such scope (the number of packages affected).

11 Likes
4

Thanks for commenting. I’m sorry if you feel singled out or attacked, because that’s not the point of the post or my intention (changing policy is).

I hope you can see how it’s hard for me, the reviewer, to trust any of those statements? If the prose was human written and the code LLM-generated, to some degree of certainty I can trust that you understand it at least enough to write the commit messages, PR description, etc.

But just with you telling me “I understand it. I promise. I worked on it a lot. I promise.” doesn’t really give me much to go with. Both of these things could still be true, but the PR could be really difficult to review simply because of all the excess text. IMO brevity is a virtue. I hope you can understand where I’m coming from! Thank you for responsibly disclosing.

11 Likes
5

Understand you’re making the case for a policy change and that your intention wasn’t intended to be directed at me personally.

As I can only speak to the text I specifically included in those two contributions (which were nontrivial to prepare and test, with or without a coding agent), I tried to apply the standard of including text I believed would be of use to the reviewer. I reviewed every sentence of every paragraph, and they specifically cover the WHY the changes are structured the way they are, which the reviewer would need to recover from testing/tracing (something I assume is relatively expensive for reviewers) rather than something readily apparent from the implementation alone, or to defend the implementation choice versus alternatives (since it’s also work to ask a contributor why an alternative approach was not taken). In preparing it, I TRIED to include those details that a reviewer familiar with the code would find useful, and was fully conscious that it was double the typical PR body length; however, given the many thousands of packages affected by the change, it didn’t strike me as unreasonable.

If you have particular feedback on the details I included that you would rather I omit, I would welcome any specific feedback. Ultimately, I tried hard to optimize for the reviewer, but the balance of brevity at the cost of omission is admittedly subjective.

7 Likes
6

FWIW, my suggestion would be something like a human summary of the change up front and putting what the LLM outputs under the fold with a “this is what Opus says” or something. I appreciate the TL;DR, but maybe what’s making it hard to read is that it needs to be broken up a little (so people who want to engage with the toplevel human written piece can do so, and those who want the gory details can also do so).

4 Likes
7

All this is really about is feeding maintainers with the amount of information they’re ready to process at a given time. How do you know? Talk with them first.

You won’t go to your city council and say “Look, I’ve written this 500 page dissertation on how to build a subway system, please review”. You’ll ask “What do you think, should we build a subway system?” and go from there.

18 Likes
8

Something like an Architecture Decision Record is good for the top of issues, probably:

In the context of <use case/user story>, facing <concern> I decided for <option> to achieve <quality>, accepting <downside>.

That may help to communicate to others that you understand the change going on, especially where it’s ambiguous. I believe these ambiguous situations can happen quite often as we are a global project.

9

When i review, I frequently run into stuff like this:

Completely hallucinated PR templates are draining to come across.

1 Like
10

I must say I agree here that the PR description fees like a wall of text.

A specific criticism I’d add is there is no apparent structure to the paragraphs. Maybe it’s a limitation of my brain but I need some clear apparent narrative first, then more details. I tried reading through the paragraphs but it feels like arguments go all over the place, with no clear direction. I don’t understand the relation between the start and end of a sentence nor the relation between paragraphs. Nothing sticks in my mind.

A first step would be to add headings but this is not enough. I’m sorry I don’t have more ideas here, I’m certainly lacking vocabulary to explain what I feel is wrong.

The best analogy I could give is the difference with documentation that do or do not follow the https://diataxis.fr/ principle. Those that do not feel hard to grasp, have too much information, have not a clear audience while those that do are clear, to the point, and with no extra fluff. Maybe there are similar principles for text in general? My usual principal guidance for writing is that a text is perfect not when there’s nothing to add to it anymore, but when there’s nothing that can be removed from it.

Hopefully this will help a little.

6 Likes
11

(Deleted, not considered helpful after rereading)

12

this could be solved more automated with a GHA that checks if the PR template is strictly followed ( this would mean it would turns from a guideline to a requirement ), maybe make a comment and assign it a “invalid” label. Would be a easy very first thing that could be done.

2 Likes
13

You just need to ban people using walls of text.

Disclaimer, the text below was generated by an LLM for informational purposes only:

A ban on LLM-generated PR text only works if we can tell model prose from human prose. We can’t: humans and detectors both fail on lightly edited text, so the only signal left is length and structure.

Edit: To clarify, which text would you rather read in a PR message?

3 Likes
14

This is incorrect, and dangerously so. LLMs, when asked to summarize, are liable to erase aspects of the original that are far from the average of the training corpus – which are exactly the aspects that a human would make a point of highlighting. In some cases this can cause the LLM “summary” to make the exact opposite claim from what was intended. https://aclanthology.org/2020.eval4nlp-1.1.pdf has some good examples and is a decent starting point if you want to dig deeper into the literature on this topic (it has been quite extensively studied and appears to be a hard limitation of LLMs in general).

If it were up to me I would ban LLM-assisted contributions of any kind, so I endorse this proposal as a step in the right direction. Re the “I bet you can’t tell” argument, I refer you to my previous remarks on the subject.

10 Likes
15

sorry, i clarified my intention with the comment i made

16

I’ve been trying not to say this out loud, but oh well :index_pointing_up: :upside_down_face:.

:+1: I’d much rather review a PR without description and comments than than one with unedited generated comments. More generally, speaking for myself, I need to feel that I’m having a conversation with the contributor and we’re both making an effort to get on the same page, otherwise why waste the time…

Precisely why I’m more and more convinced that, for the sake of both LLM-using and LLM-avoiding contributors, we simply need to ban all “LLM-looking” content:

We’ll only need to learn, just like with the “moderation” efforts, that rejection (a “moderation action”) has nothing to do with “shame” or “judgement”, and is ephemeral…

6 Likes
17

This should be trivially closeable under we should instantly close PRs that don't contain the automation/ai and contribution checkboxs · Issue #551976 · NixOS/nixpkgs · GitHub. PRs that wildly violate policy are not acceptable. As mentioned this can be automated.

I don’t think that’s overall what we are discussing here, though (cases where the policy is technically followed but flouts the intent of reducing extractive contributions).

4 Likes
18

The issue with this is that autistic and neurodivergent people like myself may get caught in folks’ LLM filters for sounding “too much like LLMs” or similar. I haven’t been caught up in this but I’m aware of some friends who have, along with various anecdotes I’ve seen around the web, such as Welcome to Reddit or Welcome to Reddit, along with research papers like The misclassification of autistic writing as AI-generated, which I recently came across and notes that “posts from the likely-autistic corpus were significantly more likely to be flagged as AI by this detection model”(Note this is, as far as I’m aware, a GPT-2-era classifier, but I believe the point still stands.)

So I would not want to do this at all, as it presents a barrier to entry that I have to change my writing style to appease a detector that frankly doesn’t work at all. And to be clear, I say this as someone who is in general extremely anti-LLM.

14 Likes
19

If it’s likely to be misclassified, is it probable it might also be hard to read to other people? If it is, maybe we just edit the text together during the review? People being autist is just expected in Nixpkgs…

4 Likes
20

Yeah, I’d have more trouble naming someone who isn’t.

Either way, if you’re writing a 2000-word essay to explain your commit, (on average) nobody wants to read that. SO I would hope no one is doing all that manually.

I’m going to ignore that entirely because that’s a misclassification by AI-based tools which we know to be garbage at identifying AI vs non-AI text anyway. Same for the reddit threads. I’ve never seen anyone propose using such tools in nixpkgs.

I think anyone who actually encounters LLM text on a daily basis can tell the difference (and if people are using LLMs so much that they sound like an LLM, well…)

2 Likes