The overhead of PnP, the patches it requires, and the need to run Yarn makes this a lot slower, and carries the same cache non-determinism issues I mentioned before.
Ah - I meant using the same approach of intercepting the requires/resolves to point node directly at the files in the store instead of having to compose node_modules.
That would make it trivial to skip around in commits and always having the correct dependencies.
Have you tried this tool out yet?
Give it a shot, I think it does almost exactly what you’re describing. It’s good to see it implemented and how it plays out for your project’s needs. For your use case it might be fine; but we had issues using workspaces and composing builds together - and unfortunately it did not appear to “share cached tarballs between projects” as described ( again, the underlying non-determinism in Yarn’s hash keys is the problem, there was nothing the authors of this tool could realistically do about that ).
My initial approach with my tools was hacking parts of this, node2nix, npmlock2nix, and pacote together. I’ve accrued a large graveyard of approaches that extended these tools, and they were all essential references. And honestly, if you only care about building packages that are published in a registry - these tools totally cover you. My use case is for managing a large collection of unpublished modules, so we really needed “full coverage” for acting as a drop in replacement for Yarn or NPM to avoid hitting gaps/edge cases not handled in existing tools.
1 Like
How dream2nix framework fill the gap here?
1 Like
dream2nix looks interesting for my use case, I’ll give it a shot. After that the more complex yarn-plugin-nixify might indeed do the trick.
Second on dream2nix. I’ve used it extensively to package extremely complex yarn-based monorepos with workspaces and a variety of other off use cases, with close to 100% success rate with nodejs-based packages. The nodejs ecosystem is pretty much solved for dream2nix. It has support for both pure (package-lock.json, yarn.lock) and impure translations and provides many paths to solve various problems, such as package overrides, dependency injections, building without devDependencies (so build from the outside with nix instead), etc.
1 Like
I don’t see any indication on the project’s github that this is related to Node.js.it looks like another Flake, Niv, etc project management CLI tool.
I’ll dig into their docs, but if they have really solved Node.js in Nix they might want to put that front and center in the readme.
I dug deep to find every Node.js+Nix framework I could a few months ago. It would be a shame if I dumped all this effort into a new tool because an existing solution had a vague readme 
Edit: yup. I found the nodejs part of the repo. It’s less granular than mine and has way more abstraction but we have nearly identical routines for all the parts that matter… Feels bad. Spent literally months working through all of the nasty edge cases, writing tree walkers and closure resolvers. Pain.
2 Likes
Argh 
this is really a problem with the nix ecosystem. I hadn’t heard of dream2nix before this thread. The project’s packages are even in the default flake registry :-/
1 Like
Just to let you know, derogatory comments about whole classes of developers are not welcome here.
1 Like
I feel a bit better. Dream2nix looks fantastic, it handles workspaces well. The documentation leaves a lot to be desired, no Node.js docs now, and the existing examples contained deprecated routines, but it’s a WIP so I can’t fault them for that.
I think it’s biggest strength is how it organizes projects and the interfaces it’s created for builders, fetchers, input parsers, and input “discoverers”.
I think my best path forward is to plug my builders and my registry fetcher into their API as a dream2nix extension. This would save me the headache of defining my own abstractions, which is a time consuming process.
Building a known project isn’t actually that hard, but providing abstractions to allow overrides, dealing with exceptional packages that need special treatment, and giving users the ability to configure their build is basically covered by their API which is pretty dope.
With my builders that completely replace NPM, this thing is gonna fly 
1 Like
Yes! I’m glad you’ve taken a deep look at it. The flaws are pretty much what you pointed out, but it’s come a long way in a short time. Development chat is on matrix at #dream2nix:nixos.org
2 Likes
Do you have anything for dream2nix yet? I’m hitting the problem that it doesn’t resolve peerDependencies: Track nodejs progress · Issue #22 · nix-community/dream2nix · GitHub
Yeah I have it working on a real project, I just need to yank my routines and move them to my library.
Do you care about peerDependenciesMeta for optionals? I haven’t handled those yet because they are allowed to have dependency cycles which is a mess to deal with.
@Growpotkin I have a somewhat dirty PR up at nodejs: Use more pure symlink builds by wmertens · Pull Request #195 · nix-community/dream2nix · GitHub - it handles cyclic dependencies and simply includes all optionals for now.
1 Like
I have a dream2nix PR for new build system for nodejs as well. It could be merged anytime soon.
But well it does not yet resolve peerDependency conflicts. To resolve such problems your first need to use pnpm with the pnpm lockfile and then a build system that is capable of producing the pnpm node-modules folder. (I had such a thing about 80% ready but would need to invest more time on it
I kind of forgot about my PR :-/ The problem is that it’s quite different, it builds everything with symlinks which is fine if you put cyclic deps together. I should look at it again.
Currently, we have the idea, that dream2nix could be more modular. I think of well-defined interfaces for atomic parts, such as dependency resolution. I didn’t entirely look into your approach yet. The yarn-plug-n-play solution is quite interesting also with using symlinks because we can omit creating the node_modules folder. (If I memorize correctly) With the evolving dream2nix architecture that logic could be encapsulated into a module that you or the user of dream2nix can provide to configure the build system.
Can I contact you about the plug-and-play system? Maybe we can provide a good module for it and leverage the power of the nix store.
+1 on more dream2nix API for frameworks.
floco was basically rewritten from scratch to use modules so that we could integrate with d2n but haven’t managed to work out exactly how external modules and frameworks are supposed to plug in without forking and extending or merging my framework into their codebase entirely.
1 Like
Right, so my PR:
- reads the v3 package.lock format and gets all the peerDeps, optionals etc
- changes the cyclic dep calculation so it groups cycles instead of the current snipping cycles (which is incorrect). This could be made to happen only for npm
- changes the module build so each cycle group is copied together, their deps are symlinked and then the deps’ build scripts are run
- makes it easy to generate a node_modules with or without devDeps
This way there’s lots of reuse, and devShells are trivial.
I got stuck on Node package-lock v2 by wmertens · Pull Request #246 · nix-community/dream2nix · GitHub
1 Like
@phaer and me have been working on testing the grounds for a new integration API based on drv-parts. That way the whole integration story should become a lot simpler. I’ll ping you once we got a first draft of it merged. I’m really looking forward to incorporating your work on floco.
6 Likes