Installing Plone with Nix?

azazel75 · January 13, 2019, 3:40pm

Has anyone ever tried to install Plone (and Zope) with Nix and has a recipe to use at least as starting point?

azazel75 · January 16, 2019, 2:41pm

For anyone coming here, the discussion on the Plone’s community site has been more fruitful.

joehealy · March 5, 2021, 5:49am

I’ve used your repo to get a development shell (metapensiero / plone-nix · GitLab) I had to change the commit it was point at, presumably because of a rebase. I had to change e5b34577bb7fc3782362d659b56ff1fea2ef7f16 to e62703057add368d6ad7fc231955f0628171cb6c. Is that the right commit to point to?

azazel75 · March 23, 2021, 2:26pm

Hi @joehealy, sorry for the late response. It has been some time since I worked on this repo, can you tell me where this commit e5b34577bb7fc3782362d659b56ff1fea2ef7f16 is mentioned?

joehealy · March 23, 2021, 11:43pm

Thanks for the reply.

In the docs on this page: metapensiero / plone-nix · GitLab under “How to create a Plone development environment”. I’ve been using the latest commits more recently having figured out how to modify the examples to match your latest changes.

However I’m running into quite a few issues installing things (collective.celery most recently) and am thinking of moving back to a more traditional buildout install. Is the wider plone world not yet ready for a nix approach?

azazel75 · March 24, 2021, 12:58am

What issues did you stumbled into? want to share some?
Python packaging and nix do have few problems alone, mostly due to the packaging of packages with architecture specific bits. Some other problems often arise from the fact that some people still need to learn how to specify the dependencies (e.g. testing dependencies listed among the others) and that setuptools required much effort before reaching maturity. On top of that a tool like zc.buildout that wants to do a bit what pip does and then some product/system configuration doesn’t help. Plone with its hundred or like dependencies means that’s inevitable to have issues when you want to install it in the same environment (python/nix env) with another medium sized framework with its own dependencies that may interfere with Plone’s.

Also please bear in mind that mach-nix work is split into two major components, there the “Nix package deriver” that runs when you use it, but this part uses the dependency tree calculated by a crawler at a specific point in time (it downloads this db from a git repo). This in turn means that if what you want to install is newer than the pin it will not be found… I pin that db here default.nix · master · metapensiero / plone-nix · GitLab

joehealy · March 24, 2021, 2:22am

thanks for the above - I’d love to share

I’ve mostly (sort of) figured the above out. I last used plone heavily about 5 or 6 years ago, before plone 5 (4.3.2 from memory). Since then I’ve been using nix for haskell with a tiny bit of local python packages. So I’m new to mach nix.

I’ve figured out the pinning of the crawler part and updated that to a more recent revision.

I’m now wondering if there are inherent incompatibilities between the packages I’m wanting to use at the revision of the crawler I’ve picked or if somehow they don’t work nicely on nix.

my next planned step is to just get the buildout environment and try running buildout by itself, then if that still fails an ubuntu or similar vm to see if I reproduce the issue.

In terms of the direct issue I find, when using:

pypiDataRev = "1ba4b301d347c90197d99c9e476d6dc230c94dd3";
pypiDataSha256 = "1pkgjj8kdhz0vy6rklkca9hbciyxng05d8hiwpfn6naj3d2yspaq";

I can add collective.documentviewer to the “moreRequirements” and it works well, however so processing can occur in the background, I want to add collective.celery so the pdf processing occurs in the background. When I do so, the environment builds, but I get errors when trying to run instance fg:

 File "/nix/store/z1ivb7r03dr6gyns1vpry2srywmr4lq4-plone-5.2.3-pending-skeleton/parts/instance/etc/site.zcml", line 16.2-16.23
 File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/Products/CMFPlone/configure.zcml", line 110.2-114.8
 File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/documentviewer/configure.zcml", line 11.2-11.34
 AttributeError: can't delete attribute

At this point, I’m not to sure how to debug directly, so trying a few options as described above.

My preference is to resolve this and for this to all be controlled via nix - anything else these days feels a little shaky.

joehealy · March 24, 2021, 2:38am

For reference, this is the shell.nix file I am using: debian Pastezone

azazel75 · March 24, 2021, 12:18pm

hi Joe, I run your shell.nix and I stumbled on your same error, which with
some more context is the following:

  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/documentviewer/async_utils.py", line 40, in <module>
    @task()
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/celery/__init__.py", line 37, in decorator
    return getCelery().task(base=AfterCommitTask, **task_kw)(new_func)
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/celery/utils.py", line 98, in getCelery
    _local.celery = _getCelery()
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/celery/utils.py", line 88, in _getCelery
    del(celery.backend)
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/celery/local.py", line 161, in __delattr__
    delattr(self._get_current_object(), name)
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/documentviewer/upgrades.zcml", line 6.2-12.50
    File "/nix/store/s1aw12vgmv4w1005yh3yl47mzln9y1xr-plone-5.2.3-pending-skeleton/parts/instance/etc/site.zcml", line 16.2-16.23
    File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/Products/CMFPlone/configure.zcml", line 110.2-114.8
    File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/documentviewer/configure.zcml", line 11.2-11.34
    AttributeError: can't delete attribute

From this my opinion is that some celery configuration is missing. The story
here is that when plone is starting up, some celery task is scheduled, probably
after reading the collective/documentviewer/upgrades.zcml file, between the
line 6 and 12:

  <gs:upgradeStep
    title="Convert all valid unconverted files document viewer"
    description="if async installed, we queue up all processes"
    source="*"
    destination="upgrade-all"
    handler=".upgrades.convert_all"
    profile="collective.documentviewer:default" />

All runs until a callable is decorated with @task (from celery) to become a
task (in collective/documentviewer/async_utils.py). Decorators do run custom
code when the code context is first evaluated (in this case the module), takes a
callable and returns another callable. During this phase the line
del(celery.backend) and this is the beginning of the end: that backend
attribute is missing and I suppose that this is due to a missing celery
configuration, but I’m no celery expert. What I know is that surely it needs
some configuration for the queues and the results.

Looking at the collective.celery package repo, I see that there is some
configuration to prepare, You can find more here:
https://github.com/collective/collective.celery/blob/497e81ddc82694ff2388e0b44672a936f44d750e/docs/README.rst .

I’ve tried something like:

$ pcelery worker /nix/store/s1aw12vgmv4w1005yh3yl47mzln9y1xr-plone-5.2.3-pending-skeleton/parts/instance/etc/zope.conf 
Traceback (most recent call last):
  File "/nix/store/771bdvayqd8axcncslj3i55n8ikmx9pm-python3.7-collective.celery-1.1.4/bin/.pcelery-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/collective/celery/scripts/ccelery.py", line 42, in main
    from Zope2.Startup.run import configure
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/zope/deferredimport/deferredmodule.py", line 70, in __getattr__
    v = get.get()
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/zope/deferredimport/deferredmodule.py", line 54, in get
    return super(DeferredAndDeprecated, self).get()
  File "/nix/store/805sa391yav0ncm5cxd1k62vpzpf4nyg-python3-3.7.6-env/lib/python3.7/site-packages/zope/deferredimport/deferredmodule.py", line 37, in get
    v = __import__(module, *self._import_chicken)
ModuleNotFoundError: No module named 'ZServer'

because in plone-nix environment the zope.conf is not local but is packaged,
but as you can see I stumbled on another issue which is the missing ZServer
module, which is missing because latest Plone/Zope combo use a more generic
WSGI pipe to publish the stuff, and the ZServer isn’t used anymore.

Another issue may be that the plone.celery dependency on the celery package
is defined as celery>=4 but recently celery released a new major release (the
5.x) which a friend of mine uses and he tells me that especially “startup” phase
changed much. Checking the celery version I’ve got 5.05 version:

[nix-shell:~/wip/metapensiero/joe]$ celery --version
5.0.5 (singularity)

I think you will have some work ahead if really is required for plone.documentviewer
and collective.celery to work with the latest Plone releases, even in an
non-nix environment.

joehealy · March 24, 2021, 10:47pm

Wow! Thank you so much for your efforts. I really appreciate the help.

I had assumed that celery would add something in to give me a form to configure it through the management interface.

I was wondering how all this worked, being more familiar with the old z server approach (well zeo cluster and then front end workers…).

Thankyou so much for your help again - looks like I’m back on with nix and finding a new solution to viewing pdfs - probably custom views with an iframe or embedded object. Also laying out your process has given me some tips on debugging this.

I’ll keep an eye on celery and see how this evolves over time.

Thanks

joehealy · March 30, 2021, 5:05am

For reference, there appears to be a known issue with celery (as @azazel75 kindly worked out above) Doesn't work with plone 5.2.4-py3 (but works with plone 5.2.1-py3) · Issue #17 · collective/collective.celery · GitHub. I think the issue may be closed prematurely, but I’ll give applying that patch a go and report here and on the issue.