Attendees: @edolstra @roberth @thufschmitt @fricklerhandwerk @ericson2314
Agenda
- Discuss architecture, responsibilities, delegation
Update on Sovereign Tech Fund grant applications
-
@fricklerhandwerk talked to a bunch of people
- A few interested, amongst the Tvix people in particular (for RFC 134)
- will start working on a project proposal for implementing RFC 134
Architecture, responsibilities, delegation
Results, summary by @fricklerhandwerk:
- We want to split maintainer teams into API (back-end) and UI (front-end)
- Each team should also have someone specialised on documentation
- This is ambitious but would help us with finding better focus
- We have to balance interests in different interface types as they serve different purposes:
- “covariant” interfaces to build around or reuse
- examples: store API and protocols, Nix language
- concern: developer experience for third-party applications
- “contravariant” interfaces to hook into
- examples:
nix run,nix bundle, flakes - concern: user experience for a wider audience
- examples:
- “covariant” interfaces to build around or reuse
- Fostering independent innovation (grow the developer community and ecosystem) and providing a powerful, uniform interface (grow the user base and increase immediate utility) are related but to some extent conflicting goals
- A more diverse and competitive ecosystem will likely give rise to more functionality and better products, but also make navigation harder and increase decision fatigue
- This is also a resource consideration: We of course want both, but have limited capacity to actually do things
- There are governance issues involved. One has to answer questions what makes an official project, how to make those and others more discoverable, and how to ensure interoperability within the ecosystem. This is not something Nix maintainers can or want to do alone.
- We haven’t reached a conclusion on how to deal with that
Complete discussion
-
@ericson2314 proposed deciding if we should split out responsibility for some of the layers in Nix, such as what is currently
nix shellandnix develop- a CLI provided by Nixpkgs would know about all the details of
pythonPackagesand other langauge-specific environments, being able to integrate with all the nasty stuff
- a CLI provided by Nixpkgs would know about all the details of
-
@edolstra: what’s the advantage to the user?
- @ericson2314: it would allow for a degree of integration we would never dare attempt in Nix
-
@edolstra: currently you’d have a
flakes.nixand run anixcommand. with that you’d have this Nix expression and would have to figure out how to get another tool- what sounded better is what we discussed some months ago was @roberth’s proposal with attributes for hooks
- @ericson2314: we can have both, and one can depend on the other
-
@roberth:
nix-shell -palso lacks cross-compilation support
-
@thufschmitt: to be able to discuss it more effectively would be having someone implement a POC
- @ericson2314: totally agree. Nixpkgs doesn’t need to ask for permission, they can just do their thing and we can still see how that goes
-
@thufschmitt: take the example of
nix-output-monitor, it got some traction because it is arguably better for some people- we could draw some inspiration from that to redesign Nix output
-
@edolstra: we don’t have to give anyone permission, but we can still incorporate good ideas or make them official outside of Nix itself
- but don’t want Nix being a poorly integrated patchwork of third-party tools
- @fricklerhandwerk: we currently are in that situation
- @thufschmitt: this requires figuring out clearly what integration or making tools official should mean
- but don’t want Nix being a poorly integrated patchwork of third-party tools
-
@edolstra: what concretely would integration mean? take for example
nix-output-monitor- should it be reimplemented in Nix, should it be included in the installer?
- @ericson2314: don’t care much what things are called. it’s fine to have a batteries-included product, but there also should be independent building blocks – those don’t have to be promoted to the same extent
-
@fricklerhandwerk: also agree on having a batteries-included system
- (some considerations on providing policy and mechanisms, such as rules for being official and options to bundle tools)
-
@thufschmitt: vendoring Nix requires a much simpler installation
- @ericson2314: that is possible
- @thufschmitt: but still some work to be done
-
@thufschmitt: example of Nix bundling third-party binaries is
nix bundle - @fricklerhandwerk: we probably also want the option for users to ramp up from just installing a binary for running ephemaral programs on any system up to fully declarative system configuration, as a first class user story
-
@ericson2314: what I care about is having stable interfaces without having to constrain ourselves
- also very interested in Tvix implementing this, because more people being able to do things without our permission allows more innovation overall
-
@thufschmitt: one major question is how we make Nix interface with third-party things, such as we discussed with
nix-shell- where do you put the boundary between internal and external?
- @ericson2314: we should find natural cleavage points, such as the store and the language
-
@thufschmitt: to ask provocatively: how would splitting the store help with
nix-shell? what would it help with- @ericson2314: it would enable lots of DevOps use cases outside of the Nix implementation, and we’ve barely scratched the surface there
- @thufschmitt: to take the Kernel example (mono vs. micro), maybe this is similar to mono vs. polyrepo?
-
@fricklerhandwerk: we seem to be revolving around the question who is to provide a (possibly unified) user interface
- UI in my experience is the hardest and most expensive part in software development
- from an economic perspective, it may be sensible to shift that burden on third party implementors with other incentive structures and more resources
- we could recommend others to compete over shiny user interfaces and harvest some of that energy for core maintenance. it’s likely more profitable but also more expensive
- example: devenv.sh bypasses the entire CLI debate
-
@thufschmitt: agree devenv is a much better interface to development shells that Nix offers, but it has a fairly narrow user base, arguably because it’s not Nix itself
- if we allowed for UI competition, we’d have to watch out for users to have a central touchpoint
-
@thufschmitt: agree devenv is a much better interface to development shells that Nix offers, but it has a fairly narrow user base, arguably because it’s not Nix itself
- example: devenv.sh bypasses the entire CLI debate
-
@thufschmitt: if we wanted a uniform interface for all official tools, we’d have to coordinate it
- @fricklerhandwerk: or we could leave that to competition, and rather coordinate on the composable APIs
- UI in my experience is the hardest and most expensive part in software development
-
@ericson2314: For the adoption by the largest organizations, having a easy-to-learn user interface is probably less effective than providing industry-standard protocols
- example: Docker and OCI
- user interaces would be only a vehicle to build momentum up to the point where industry adoption is a serious question
- organisational users won’t want 6 people defining the rules for everyone, but would rather prefer modular components for them to mix and match
-
@thufschmitt: disagree on the Docker example, OCI is essentially Docker but slightly more standardised
-
@ericson2314: right, Docker emerged and had to manage competition with standardisation
- we don’t want adversarial competition. Docker was a company, Nix is not
- but we may want to have some of that energy, hopefully through friendly competition
-
@ericson2314: right, Docker emerged and had to manage competition with standardisation
-
@thufschmitt: we would still need a default user interface for people to start out with
-
@fricklerhandwerk: @ericson2314 proposed auto-generating the CLI from stable, say, C bindings
- @ericson2314: yes, if we had great internal architecture, that would be fairly straightofward
- @roberth: our CLI is different than e.g. IPFS because we have multiple layers wrapped in one
-
@fricklerhandwerk: @ericson2314 proposed auto-generating the CLI from stable, say, C bindings
-
@ericson2314: we could have both: composable building blocks (maintainers’ work) and a unified interface (competing third-parties) by blessing (a) particular UI implementation(s) as official
-
@thufschmitt: but that would still be many use cases with possibly different tools involved
- @fricklerhandwerk: yes, but we can’t really know all the use cases
-
@thufschmitt: but that would still be many use cases with possibly different tools involved
-
@roberth: we could provide an interface for implementors to hook into
nixcommands, so it remains the driver, for examplenix developlooking for particular hooks in attributes provided by derivations-
nix runis a great interface that builds on top unchanging Unix conventions -
@ericson2314: wonder if anything could be build on top of
nix run -
@thufschmitt: similar to
make, which for many has devolved from a build system to a uniform interace to run programs with./configure; make; make install
-
-
@thufschmitt: another example is
nix fmt, which could essentially benix runon some external formatter-
@fricklerhandwerk: if
nix runis the universal interface that just runs binaries, who defines the conventions around it? does Nix even have to do it? most of the action is happening in Nixpkgs-
@thufschmitt: for most stuff having a convention within Nix is not that useful
- for example, having some
nix lintwould probably not that useful - but there is some value in defining particular use cases centrally
- for example, having some
-
@fricklerhandwerk: question of understanding: why does Nix have to define the convention?
- @ericson2314: I’d say brand inertia. Most marketing says Nix rather that Nixpkgs and NixOS
-
@thufschmitt: yes, we’d probably not have that discussion if
nix developwere not a thing
- @thufschmitt: Nixpkgs has much less leverage over other offers, as opposed to Nix
-
@edolstra: if every project goes off and defines their own user interface we can throw away the past years of UX work
- the goal was to provide a uniform interface where you can use Nix expressions from any context conveniently
-
@thufschmitt: for most stuff having a convention within Nix is not that useful
-
@fricklerhandwerk: if
-
@ericson2314: would be good to have short-term strategy to delegating specific use cases such as
nix develop(as with the hooks example) and a long-term one where Nix is essentially invisible infrastructure-
@roberth: agreed, we need to do both the low-level and the high-level interfaces
- we may want two Nix teams, one for
libstoreandlibexpr, one for the CLI and possiby flakes and figuring out how to use Nix- @fricklerhandwerk: the second is what the documentation team is already doing by necessity of having to figure out what to show people, but without any ambition to change anything
-
@ericson2314: yes, Nix overall as a tool for people, and Nix as a set of building blocks is a tool for developers
- ideally both teams would have documentation specialist attached to them (reference and learning journey each)
- we may want two Nix teams, one for
-
@roberth: agreed, we need to do both the low-level and the high-level interfaces
-
@edolstra: a big advantage of distributed development projects is that they don’t have to convince anyone to make progress, such as with devenv
- also adding more people to a late project will only make it later
- with Nix you have to convince a lot of people, so while moving things out of Nix would make that go faster, getting things official will make those projects jump through all the bureaucratic hoops
-
@fricklerhandwerk: yes, we’re lacking governance processes, even an idea of what it entails to become official. who would be the group of people who could make these decisions? certainly not Nix maintainers alone; we don’t even know all the possible use cases
- @edolstra: if we don’t who does then?
- @ericson2314: this has to be crowd sourced
- @thufschmitt: this brings us back to encouraging people to experiment. that would outsource lots of design work out of our responsibility
-
@fricklerhandwerk: yes, we’re lacking governance processes, even an idea of what it entails to become official. who would be the group of people who could make these decisions? certainly not Nix maintainers alone; we don’t even know all the possible use cases
-
@ericson2314: a large problem is with communication. nixos.org stopped making a distinction between Nix, Nixpkgs, and NixOS
-
@edolstra: this was discussed in the marketing team at length, the distinction was consfusing to people
- would be good though if we had one website for Nix, and a separate one for NixOS
-
@edolstra: it would be okay to e.g. link to devenv, but than it risks it being perceived as blessed and people expecting official support
-
@fricklerhandwerk: we can communicate it clearly enough, but the interesting question is how would we bless something? how would implementors know if they’re eligible? Nix has a lot of brand power that would help drive adoption
-
@edolstra: we’d have to pick winners and losers, also this is not for Nix maintainers to decide
- having third party projects is great, as one can experiment with things, such as zero-to-nix recommending flakes.
-
@edolstra: we’d have to pick winners and losers, also this is not for Nix maintainers to decide
-
@fricklerhandwerk: we can communicate it clearly enough, but the interesting question is how would we bless something? how would implementors know if they’re eligible? Nix has a lot of brand power that would help drive adoption
-
@edolstra: this was discussed in the marketing team at length, the distinction was consfusing to people
-
@thufschmitt:
- how can we encourage people to write tools to seamlessly integrate with Nix? (to reduce maintainer workload)
- how can we make those well-integrated tools feel native? (to foster a unified experience)
-
@ericson2314: we’re assuming that Nix has the best brand and that going through Nix is the default. this is a very limited view
- if you think you have a minor product, you can hook into Nix. but if you have a way bigger deal, Nix should be able to be subservient
- @thufschmitt: yes, flox is a good example integrating Nix
- @ericson2314: they’re shelling out to Nix, and that’s a terrible way to interface with Nix, they should not have to do that
- if you think you have a minor product, you can hook into Nix. but if you have a way bigger deal, Nix should be able to be subservient
-
@ericson2314: we’re assuming that Nix has the best brand and that going through Nix is the default. this is a very limited view
- collection of general approaches satisifying these conditions:
- provide a “contravariant” interface of hooking into Nix, offering some agreeable common convenience interfaces (such as already done with
nix run) - polish and promote our “covariant” interfaces and make them usable more independently:
- daemon protocol should be documented
- libraries in multiple languages with key data structures and protocol implemention, so new projects can get up and running quickly
- make the outer UI layers more of a community-wide project and provide a way to make a third-party project official in the Nix ecosystem
- adds more cooks to the kitchen, but we also already follow the RFC process to some degree
-
@thufschmitt: good example is
nix replwhich originally lived a life of its own and got integrated into Nix-
@edolstra: the reason of not initially including it because it was an experiment and introduced a dependency on
readline
-
@edolstra: the reason of not initially including it because it was an experiment and introduced a dependency on
- provide a “contravariant” interface of hooking into Nix, offering some agreeable common convenience interfaces (such as already done with
- agreement: we want two maintainer teams, one focusing on components and one for user experience, who can work largely independently
- we acknowledge this is ambitious though