Policy on vibe coded nix tools?

What im doing is calling out people’s bullshit:

  1. White washing with “computer assisted”. Its not computer assisted it’s written by an LLM. If they stated clearly it was LLM written, id have ignored it.
  2. Saying “i know people have issues with LLMs, but i hope you can ignore those issues and look at my project anyway” is either ignorance or malice.
  3. As for the comment you replied to, its clearly not in good faith and so im just flinging it back.

And as for my input, which i think ive given, but nonetheless, here we go again:

We either ban all LLM assisted/written code from inclusion in NixOS projects and discourse, which is unrealistic as i know prolific contributors unfortunately already use LLMs.

Or we require that all LLM code be clearly labeled and all projects under the NixOS umbrella have clear policies on whether they accept LLM slop or not. If someone fails to disclose LLM involvement, they get a warning, ban the next time. Yes people will lie like they always do, but thats not an argument to fold.

6 Likes

I guess my point is calling it „slop“ whenever you can is not „calling it out“ it is discrediting it. You could have for example asked for a more precise wording but you didn’t if you felt misled (which I did not).

Edit: you were quite fast at editing so this is not relevant anymore: Why not engage in the discussion here (with concrete proposals)?

3 Likes

I did feel misled by “computer assisted” but i make a habit now of chexking for indicators in the repo like claude co-authorship, claude.md or the style of commit messages.

For why i chose not to post here, well with the computer assisted one i felt obliged to warn users about the use of an LLM as the author didnt do a good enough job in my opinion. With the “ignore” one i felt like i had to say something for a reason i didnt know yet, the post really rubbed me the wrong way. (Which ive by now figured out). If i posted here, people could have been misled and/or there would be no response to the “ignore” which i find very problematic.

sorry about the edits, i wasnt expecting you to reply so quick and wanted to clarify as much as possible.

And as for calling it slop, sure, written by an LLM, but its definitely not “vibecoded” or “computer assisted”. I feel like slop describes the reality of the code quality written by these LLMs with accuracy other descriptors cant.

If we get a policy of clearly labed LLM-written code, ill just quietly ignore all of it. I only got one life and one brain, the battle against LLMs is already lost, but what we should get at least is transparency.

(Done with the edits…)

8 Likes

I would also really appreciate AI-assisted tooling to be clearly labeled. It’s happened many times now that i see a cool new tool, and check the commit log to find everything co-authored by claude. I would really like to be able to easily focus my efforts on human authored tools. If there is nothing wrong with using AI tools, surely there is nothing wrong with explicitly disclosing their use…

(edit: grammar :clown_face: )

7 Likes

But feelings are quite personal. What feels right to you might feel discrediting to others. Why can’t we stick to something less personal that includes feelings but rather use terms like LLM generated/assisted.

I think the hardest thing is getting the nuances between asking an LLM for solving specific problems (like you would do with Stackoverflow) and having an agent do all of the work and things in between where one does the design and the LLM does only the coding? I personally think it is really hard to draw the line somewhere but I still would like to know in advance for tools so I can choose whether I want to have a look at it or not.

Yeah then we essentially agree, im just not the type to be very nice :slight_smile: . Personally i draw the line at “no LLM use whatsoever” especially no proprietary SaaS ones, if some LLMs that are trained only legally obtained data with all attribution preserved, appear, i might consider its use. Then again such an LLM is probably not worth my time, nor will it be legally able to produce any meaningful code, since the copyright of all the data should transfer to the output. GPL producing LLMs are a bit restrictive. (I am aware that the EU and the US legal systems have a different outlook on this, idgaf)

9 Likes

Slightly off topic but one thing I’ve learned is that even open training data models will breach TOS (Longpre, Mahari, Lee, Lund, et al).
The only text model I’ve heard of that allegedly does not do this is K3LM, and it’s quite small (3.7b).

5 Likes

@MagicRB please stop shitposting every discourse post that contains llm generated content. Just move along and think before you post.

This is quickly getting similar to the RTFM posts of other platforms. I think such behaviour is toxic and bad for the community. Even if a tool was vibe coded and even if the code quality is bad, someone took the time to think about the problem and try to solve an itch. There is a human on the other end…

5 Likes

I’ve explained my behavior and no i will not quietly move along when the human tries to hide behind “computer assisted” intentionally or not, and when the human asks me to ignore all the moral issues of LLM use. As ive stated before, posts which are clearly labeled in the subject as “LLM” i wont even open, but by not doing that, youre inviting me to waste my time reading your post, or worse, looking at the code/commits to just go “not this shit again”, thats inconsiderate. Having to always do this for every, single, damn, project i come upon is getting extremely annoying and until people start clearly labeling LLM involvement, i will not shut up. All you gotta do, is add [LLM] in the post title or project description and ill quietly move on, but until then…

(Ill save you the speach on why ive grown to despise the term “vibecoded”, im tired of this discussion too. Start labeling LLM posts and everyone will be happy)

8 Likes

You can expect to be eternally angry if you get upset every time LLMs are involved and not labelled the proper (according to you) way so that “you do not waste your time”.

1 Like

eternally angry

Ill add it to my other things which are not worth doing by that logic then. Goes right next to fighting for software freedom and privacy. Looking at a project only to discover its LLM contaminated is a waste of time. Trying to save myself time from doing that, is definitely not a waste of time.

proper (according to you)

I guess asking for people to state that they used an LLM is too much these days.

7 Likes

We’ve managed this with licenses. I look for a LICENSE/COPYING file and i immediately know if i should ignore the project or not. Can we do that with LLMs too? An LLM file? Claude.md is that already, but not by convention, but by accident. Do that and we can part ways disagreeing, stop wasting my time by making me spend it on discovering if your project was written using an LLM thanks

5 Likes

There is https://agents.md/

1 Like

That is also a good one, another is seeing claude in the commits, or the commit message format. Maybe i ought to build a tool which does this itself since people seem to not want to do it

1 Like

Please take this somewhere else. I want this topic to be a place for folks express their thoughts and feelings around LLM content in forum posts. Not to litigate individual forum posts or behavior of specific individuals.

I get the feeling everyone’s said their piece here. We’re all welcome to talk about AI usage more broadly, and things can and will get a little heated. But I think we should cool down or take this elsewhere. This isn’t about shutting down conflict, but rather leaving room for others to talk without being afraid of being antagonized.

4 Likes

Im out of this thread again, i think ive made my view point clear. Dont have anything else to say, if you want discuss with me, youre always welcome to contact me in some other way privately. :waving_hand:

3 Likes

That being said if you want to keep talking about potential LLM fingerprints I see no issue with that. The conversation had already shifted by the time I got my post out.

1 Like

So, as the author (after a fashion, one supposes…) of one of these tools, I do want to point out a few things:

  • There is a big difference between a tool that just gets yeeted up there and a tool which undergoes a bunch of testing and tweaking.
  • It’s annoying to go to the trouble of being honest about this code’s origins when people dogpile and say shitty things solely due to that fact. I consider myself an honest person, but there’s nothing stopping anybody from just…not mentioning in their repo or commits that an LLM helped.
  • Seeing people complain about attribution and hallucinations is rich when I’ve gotten feedback on PRs specifically to remove comments explaining where some code or idea came from. Bit of a double-standard…either we care about provenance, or we don’t.
  • It’s really annoying to take time out of my life and spend it on tooling that solves a real need only to have the first comment be snark.

Let’s get some facts out there:

We have something like 8.6K packages that are broken.

We have 47K packages that are orphans.

We have 89K packages without tests, and 64K without automated update scripts. That’s right around half or more of nixpkgs.

We need all the help we can get.

P.S. –

I would appreciate bug reports, since even with a lot of human testing sometimes I do see behavior that might suggest an issue. For example, it may be possible that my script is miscounting packages maintained.

(And yes, the tongue-in-cheek directory name is what it is because I do like keeping my LLM stuff clearly delineated in my own work patterns when it is significantly LLM-generated…even when I do a bunch of testing and workshopping. Again, that’s just being organized–and computer-assisted is a lot to type.)

5 Likes

Thank you for your input! That certainly looks like there may be some miscounting! Although I would add that someone’s contributions to Nix are far more than the sum of their nixpkgs contributions. Writing tools, writing documentation, helping people on the forum, etc., are all important. Many people contribute to the Nix ecosystem without maintaining packages.

With that said, I think there’s a lot of conversation to be had about using AI (at least while credits are still heavily subsidized) to ensure quality and pick up sone slack in open source. It’s not exactly clear cut. For example, widespread AI usage decreases engagement with open source. Still, from a pure productivity point of view, there are certainly some smart areas where AI could be utilized.

I personally don’t agree with that line of thinking, although I’ve fallen into it many times: where I think it goes wrong is that I fell into a framework that only considers productivity in my attempts to simplify and grapple with the implications of AI. Every time I start thinking like this I catch myself because the thought experiment of “let’s only look at things with an eye for productivity/efficiency or quality” deliberately ignores all the other factors. Productivity to what end? Does coding with AI bring me joy? How does AI affect the broader community? What about economic, societal, and environmental implications? What of worker solidarity?

And to what extent should the NixOS discourse care about any of these? I don’t know. But it sounds like finding a way to self tag without framing it as a brand of shame is agreable to most people here. Maybe just a simple “This project was LLM Assisted: ${details}” at the bottom of a post? Something folks can find by scrolling to the bottom and glancing, but that isn’t a scarlet letter in the title of every post. I of course would like to see disclaimers in titles but I can see how that might feel demeaning to authors of projects. Maybe some simple changes to the forum rules is all that’s needed? I think if we take the wikipedia approach and flag AI usage liberally that will reduce a lot of questions around how somethibg is judged AI generated. Used Super Maven to complete code? Flag it as AI and put that in the details.

2 Likes

uh huh. whatever. all the trash we had before it was trained on was …. gold? whatever. I think if a “core contributor” says some patch was made with assisted help from the computer we’re gonna reject it? or whomever will reject it? Maybe? I dunno. Nature of software dev is changing :person_shrugging: thankfully it’s mostly still all UNIX-like (but the youth of the future will have something better than UNIX :prayer_beads: )