Pylightnix the Nix-style data management in Python

Announcements
1

Let me announce a new version of Pylightnix, the Nix-like data storage library in pure Python.

Since most of the readers know what Nix is, I’d like to highlight the differences between it and Pylightnix:

  1. The scale. Pylightnix is a library to be used in applications, the software package management is not among its primary goals.
  2. Pylightnix uses Python programs where Nix uses Nix expressions.
  3. In Pylightnix, we require users to explicitly specify the managed part of a derivation. We expect it to be a JSON-friendly Python object (an arbitrary structure made of dicts,lists,prims,but for example not tuples). Only these objects are involved in the dependency analysis.
  4. Pylightnix supports non-deterministic “builds” resulting in multiple concurrent realizations to co-exist in storage. We allow users to filter them by calling a special type of callbacks called Matchers. We hope this feature will be useful, for example, in machine learning or other scientific applications.
  5. In Pylightnix, objects are not tied to the location of the storage in the filesystem.
  6. We support “promised” artifacts declared in the expression. The interpreter checks if the promises are fulfilled and may throw an error earlier if it finds they are not.

Pylightnix remains quite compact (cloc returned 2450). It is written in a non-idiomatic procedural Python, requires no Python dependencies (but does optionally depend on wget, curl and atool), employs mypy and hypothesis for testing. The recent changes include:

  • Support for with current_storage/with current_registry contexts, allowing to write a more Python-looking spaghetti code.
  • Implemented “spack”/“sunpack” which roughly resemble nix-copy-closure into zip-archives.
  • An mklens tool for dot-style accessing the artifacts.

BR,
Sergey

(The project was previously discussed here Pylightnix - A lightweight nix-like DSL in Python for ML)

3 Likes
2

Nice!
Does this mean now we can use Python to write a derivation instead of Nix?

3

It’s a separate implementation of the ideas that make Nix what it is. The project should probably not contain “nix” in its name because it gives the impression that it’s compatible with the nix ecosystem.

1 Like
4

Yes we can, but that would be a pylightnix derivations with some domain-specific meaning, probably not related to the package management.

1 Like
5

Regarding the name, I picked it thinking that “nix” would become a well-known name for the specific set of design ideas. I think that the name is long enough to highlight the distance from the original Nix. Please let me know if you think that confusion increases.

6

The initial reaction from pvonmoradi shows the confusion that it generates. pylightnix looks great, and has the potential of becoming its own thing.

Sorry for reacting only on the name, the project looks interesting in its own regard. It’s great to see the nix ideas being explored independently.

1 Like