"Let's be mainstream"

Just came across an old talk by Elm’s creator, Evan Czaplicki, reasoning about UX, and mass adoption, and how to pitch things. First of all it is just a good talk, rather pleasing to listen to. But more importantly, it brings up points and examples that transfer well to the work that Nix community is doing and are generally good to keep in mind:

E.g. “Gradual Learning”

…you can actually get something going in very little time without getting hit with a ton of concepts

…he didn’t have to read a tutorial, he didn’t have to read a paper, he just looked at the pattern and added a thing in

E.g. pitchlines that “connect” with users:

“a javascript library for building user interfaces” - good, I know javascript, and I want to build a user interface

vs “an advanced purely-functional programming language” - do I want advanced, do I want pure-functional?

E.g. on competing with large and well-financed teams:

The way that you can be competitive is to design things that just can’t be done in these languages at all

…just like no one can, by-design, tame caching as good as Nix

That’s it, I’ll just leave it here

7 Likes

Excellent and concise resume!

This line of thinking motivated to start working on:

(were I’d like to add some sort of per language templating feature for exactly this.)

2 Likes

Cool, is this to provide tooling for working with nixpkgs repo? I figured to do nix develop github:blaggacao/nixpkgs-devshell but I’m not sure what follows. Oh and please do elaborate on templating if you would, I didn’t quite catch that

You just prompted me to add an (initial version of a) README. Thank you!

To the original point about doing something that otherwise can’t be done, I sometimes think the major payoff for Nix is taking the complexity past the point where in other situations you’d have to compromise or be unable to manage it. But this is not the best pitch for a newcomer.

As I evangelize, the features initial uses seem to love are things like setting up nix-shell/common dev environments for projects that JustWork. Or being able to reuse caches for that project.

The initial problems they run into are invariably: installing on OSX is finicky. How do I add some other random python package? Why can’t I pip install?

One thing that is a positive sign: I more often see random projects that have a “.nix” lying around. We can perhaps do a better job supporting and encouraging this, pointing it out, and providing assistance.

8 Likes

Hello tomberek,

I personally think that a lot of nixos users get to the project

  • because they broke the OS (like ubuntu / arch) too often

→ they depend on a system that works (and cannot be broken too easily)

it is about control and transparency

  • pkgs
  • services
  • licenses
  • tooling like nix-tree
  • flexibility like overrides

What you describe are ideas of starting with nix on “day 1”:

on day 2:

it is about configuration of nixos options/services (more than installing nixos packages)

  • user could discover mach-nix for python pypi/conda-packages :slight_smile:

and it’s about specific versions of packages/pinning and workflow automation

  • to not get stuck within the power nixos provides by creating pkg template for a lot of project packages [and “only” changing version and sha] - which is fine for “one main project” but not for adhoc / poc work

and build & deployment (ci/cd) …


I saw quite same people making a fast start in running nixos but crashed full-speed into the wall of limitation of " boilerplating"/recompiling very little change not in nixpkgs repo commit

  • working with py pkgs other then “requests” in their python env (like pytorch) → but even with “normal” pypi pkgs in pkgs.python38.withPackages is causing too much overhead/cost (time, cpu) - e.g. when having a (Data Science) environment and adding or removing any package without to have all other pinned (what would be nonsense when not in production)

In case of python ecosystem in nixos we need to change the not acceptable workaround of pkgs.python38.withPackages to a proper and (py-package and python interpreter independent) environment build

  • or the python community will never adopt nixos in mainstream

Re: “day 2”

I recently engaged almost full time with great fun and pretty nice emotional rewards @nrdxp and @Pacman99 on improving the day 2 story (once a user wants to start using NixOS)

I’m going to confess here, that while I have pretty strong involvement, I’m still not using NixOS for my personal use (“I’m still at breakfeast on day 2”). I think that makes me a good fit and element in that trirumvirat to hold deer the interests of my fellow day 2 users.

We put great care in creating a strong contributing and sharing oriented model, which means contribution and sharing should be as easy as it could get, even for a day 2 person, with one (1) recommended and well thought through way of doing things (a concept we applied to all sorts of things from overlays, to exports, to backports, to patches, etc.).

Although bringing in more mainstream users seems like a great thing. The flipside is there’s more expectations of having refined user experiences. I think this is eventually achievable, but the nixpkgs community really does need to setup up some proper backing for maintaining the nixpkgs that we have currently. There should really be a fulltime individual (or better yet, team) that is able to at least address the current avalanche of issues and PRs that we currently have.

EDIT: One of the nice bi-products of having a purely volunteer organization has been the emphasis on quality. There’s a high premium on making process not painful for individuals investing their free time into a project. This has caused the odd duality of having more non-unique packages than Arch+AUR, but not having the manpower to deal with the upkeep of 80k+ packages.

12 Likes

@jonringer , do you think it’s possible that at least a portion of the gigantic number of PR’s could be addressed by whipping up a delegation strategy for BORS? I feel like empowering maintaners to merge their own work once it’s done would be a solid motivating factor. It’s kind of a bummer when you put some serious effort into a PR, only to watch it stagnate.

The good thing about BORS is that we could delegate the merge command to the PR author, but still require up to date review by community members to actually allow the merge. Of course, there is still the problem of finding trustworthy indivduals to perform the reviews, but at least those reviewers don’t necessarily need to have full commit access, making things slightly simpler.

2 Likes

I think this could work for simple packages. Nixpkgs gets a little more complicated because you often need to touch files like all-packages.nix which can greatly affect other packages. But I think something like ofborg + bors could be useful.

From a security perspective though, it would still be hard to delegate just because of the potential of malice, such as a user changing the upstream to malicious source. But maybe we could have a bors operation which verifies that the upstream source is similar. Unfortunately, I’m not familiar with another package manager that has a similar paradigm. The AUR has a voting system, and most of the “ownership of verification” relies on the user, not the maintainer to be aware of risks.

1 Like

Automerging PRs by a bot is a phenomenally bad idea.

4 Likes

I disagree. We are 2021 and should be able to design systems that can ensure a equvialent or even superior outcome compared to our current non-scalable model comparable to the image of a public servant putting on the last stamp (if person and ink happen not to be in short supply).

If we regard our local maximum as the non-plus-ultra in system design / community workflows, in my opinion that’s a complete capitulation before the challenge that lies in front of us.

At least, please, let’s not stifle attempts or motions to tackle that challenge well in advance of fully acknowledging their benefits and at least have tested out things in a prototype and learnt from it.

again, bors wouldn’t allow this unless an up to date review is in place. This means that if the author changes or addes code after the review, the bors merge would fail.

1 Like

That’s a concern indeed. More importantly, is merging PRs really a problem for nixpkgs? I’m not a maintainer and I don’t know

Just as a direction for thought though, perhaps automation wouldn’t sound as such a terrible scheme, if we had means of verifying:

  1. If upstream’s signature keys haven’t changed since last derivation (the way, e.g., google play releases developer-signed APKs, I think)
  2. perhaps that the PR has been approved by a maintainer responsible for the changed derivation
  3. the change is local, e.g. limited to certain derivations, no spurious effects

For that we’d need to accompany every fetch* with the URL of the author’s signature for the respective file, and we’d need a way to reliably determine the scope of change that PR introduces.

If a PR meets all listed conditions, I don’t see why it wouldn’t be eligible for an unmanaged merge?

For example, if a PR just updates a revision of, say, colmap, which, being a “leaf node”, doesn’t trigger a re-build of any other derivation in the closure - that’s local, and we probably only need to make sure colmap still refers to https://github.com/colmap/colmap and is still signed with a key we’ve already used. If the PR, however, additionally introduces modifies some scripts or wrappers, etc - the bot could lock it until a human maintainer reviews it. Then, if there actually was a large number of "update rev" PRs queuing before merging, those could be “fast-forwarded”, and maintainers would only spend their time on actual novel code

I think right now there are no facilities in nixpkgs to verify GPG signatures of src.

The Nix maintainer list allows adding GPG public key fingerprints, here is mine

However, these information are unused right now:

On top of that, maybe 100 people out of over 1500 maintainers have their fingerprints listed here. If automerging is that desirable, maybe it could be enabled under the conditions:

  • Only version and src attributes of a derivation were changed and nothing else.
  • Commit is signed with a valid key of one of the listed maintainers.
  • PR has one approval of someone with write access.

However, that still has the approval bottleneck which an automerge probably wanted to avoid in the first place. Also, the signature would be checked as part of CI, requiring that the CI is not vulnerable to a forged signature attack. This reliance on CI is exactly what hit Homebrew in the guts.

That’s probably too strict to be a useful criterion.