We think Nix is one of the few technologies left in the world that’s extremely powerful but still too difficult for many engineers to use. flox is trying to change that—to get the world building with Nix. We think it’s too good to be anything less than the de facto standard for packaging software.
Sign up for the closed beta to use it and help us shape its future. It’s free and available for all Linux distros, including NixOS. (MacOS support is coming soon). We’d love to hear what you think of it and see what you use it for.
What is flox?
flox is a platform for installing, building, and deploying Nix packages. Our users think it makes Nix simpler for individuals and more scalable for teams. At a 30,000 foot view, flox’s main technical innovations are that it:
- Provides all users shared access to a unified Nix store, managed by a single Nix daemon and backed by a farm of powerful build hosts
- Distributes the contents of the shared Nix store to all endpoints on-demand with local caching and automatic background garbage collection
- Instantly synchronizes Nix profiles globally and provides web and command line interfaces for managing them
- Exposes a new command line interface that makes writing Nix expressions easier for novices
- Creates a new framework for organizing Nix expressions which facilitates continuous integration
And don’t worry: flox accomplishes all of this without interfering with your local Nix store and daemon on NixOS.
Why did you build flox?
The histories of many visionary technologies include periods where they’re only adopted by the best engineers solving the hardest problems. Over time, the core technology evolves, and follow-on projects tear down the barriers to adoption less sophisticated users faced:
- Docker and Kubernetes made containerization ubiquitous by simplifying build and orchestration, but modern containers trace their heritage to FreeBSD Jails released 20 years ago.
- Linux and macOS put Unix in the hands of almost every developer in the world by providing beautiful user interfaces, but Ken Thompson and Dennis Ritchie built Unix Version 1 back in 1970.
We think that today Nix is somewhere between these extremes. The community is growing faster than ever, and Nixpkgs has become one of the most active open source projects of all time. But Nix adoption is still much lower than Kubernetes and Docker adoption. More importantly, it’s much lower than it ought to be.
That’s why we built flox at the D.E. Shaw Group—to solve the problems our developers faced adopting Nix at scale. Now, with help from @grahamc and @Infinisil, we’re sharing flox with other developers to address the challenges they face adopting Nix too. Ultimately, we want to see more people contributing to Nix, Nixpkgs, and NixOS and using the awesome tools you’ve built on top of them.
flox is currently in a very early stage beta (expect bugs!). We plan to continue improving the tooling, documentation, and performance, but we didn’t want to wait any longer before sharing it with you. We want to see what creative applications you find for it and what you think we should build next! Eventually, we may build a stand-alone business around flox. But for now, we’re just looking for feedback.
Why should I care if I’m already using Nix?
Our early beta testers have highlighted 3 main ways flox enhances Nix:
- Improved collaboration experience
- Simplified continuous integration and continuous deployment for Nixpkgs (CI/CD)
- Faster development cycles
Improved collaboration experience
We designed flox from the ground up specifically to facilitate collaboration. When evangelizing Nix, we identified three main barriers to collaboration:
- Everyone on a project needed to run their own Nix daemon and manage their own store, which introduced reproducibility problems (e.g. the same path could contain different bits) and frustrating overhead sharing/deploying build artifacts
- Profiles were isolated on individual machines, which made them difficult to use as shared development environments and deployment environments
- Users didn’t know where to put their Nix expressions or find other users’ expressions outside of Nixpkgs, which led to Nix expression sprawl across GitHub
The first problem motivated us to build the flox Storehouse. The Storehouse is a centralized fault-tolerant set of machines that houses a globally available Nix store and runs a shared Nix daemon, which flox relays all Nix commands to.
With this design, users don’t need to install Nix everywhere they want to use it. When a machine traverses a /nix/store path, flox automatically downloads the required packages from the central store and caches them locally until they’re not needed anymore. This gives all our users local Nix performance with no configuration/maintenance overhead.
This also eliminates Nix daemon version control and reproducibility problems. Everyone is using the same daemon and receiving artifacts from the same central store. That means /nix/store/11zj...kfr2-foo is bit-for-bit identical across all developer machines and servers.
To address profile isolation, we built a new profile manager that instantly synchronizes profile generations and contents across machines, along with a CLI and web app for interacting with it.
This allows flox profiles to act as portable development and deployment environments, which can be accessed using the same paths anywhere. Just by using the same profile, multiple developers can keep their tools and packages in-sync to iterate and collaborate more easily. And sys admins can update everyone’s tools with a single command.
Global profiles also makes deploying, updating, and rolling back packages across multiple machines transactional and atomic. All it takes is a single click in the web app or a single floxpm upgrade command.
To address Nix expression sprawl, we developed a unified framework for organizing Nix expressions in git servers, which we refer to as the “Nixexprs framework”.
The framework uses orgs/users on a git server as the first-order unit of organization. It arranges the Nix expressions for each org’s packages in a repo named “nixexprs”, which is patterned after Nixpkgs itself. Each Nixexprs repo serves as a runbook for Hydra and produces a channel of packages based on its contents.
We’ve found this system has made it easier for developers to start and join new projects, transfer knowledge about projects to other devs, and collaborate on expressions.
Faster Development cycles
We love Nix, but we wanted to make it faster. So flox adds a few performance-enhancing features:
Powerful remote build farm: Instead of building derivations on your laptop, flox builds them in a powerful remote build farm. Builds complete faster, and your machine stays available for development and other local workflows.
Lazy loading: Unlike Nix, flox loads dependencies lazily at runtime (by /nix/store subdirectory). That means flox only loads what you use, when you use it, no matter what languages you’re using. For complex packages with sprawling dependency trees, this can lead to large reductions in memory footprint.
Negative lookup caching: flox serves /nix/store with a file system that knows its contents are immutable. This allows flox to cache all negative lookups–path traversals that ultimately don’t find the requested files–and avoid re-executing them unnecessarily. This produces significant runtime speed increases, especially for languages known to execute many negative lookups (e.g. Python).
Simplified CI/CD for Nix packages
Together, our Nixexprs framework and global profiles allow users to rebuild Nix packages with a simple git push and deploy them anywhere with floxpm upgrade:
Our profiles have automated upgrade schedules by default, so the second command isn’t even necessary.
How I can I get my hands on it?
-
Sign up for the private beta - flox is a work-in-progress, so expect some bugs. But don’t shy away from trying to break it.
- Request access to our Discourse – Our users post bugs, questions, and feature requests there.
- Email barry@floxdev.com if you’re interested receiving an enterprise demo or private instance.
We’re looking forward to hearing what you think of flox and making it better for you.
Where can I learn more?
@zimbatm is interviewing me and @limeytexan about flox on Nix Friday on February 5th at 16:00 UTC. We’ll demo a few features and chat about why we built flox, the implementation, and what we think the most exciting uses are for individual devs and teams. Tune in to ask questions and learn more!
- The flox team
It might be too groundbreaking to promote the greater good out-of-tree on the long term.