Allow installation of dxtbx with read-only conda_base/

[ndevenish]

Raised as an issue in 2021-07-09 Dials Core meeting, because dxtbx is installed as a "real" python package now, it fails when the base python environment is read-only, and there is no virtualenv/conda environment layering to get a writable "top layer" environment.

This patch should fix this by checking if the site-packages/ folder is read-only, and instead "installs" into a dxtbx subfolder inside the build directory - this won't generate "console_scripts" executables, but it will allow the normal enumeration of e.g. the format entry points.

It then appends the dxtbx src/ folder path to the global environment PYTHONPATH variable so that it is written into the libtbx.python et al dispatchers.

Then, a Minor Roadblock Was Hit

Unfortunately, due to a bug or otherwise unintended combination of features, the libtbx.env object that you can access during a libtbx_refresh.py is not the environment being constructed.

Although libtbx.env_config.environment.refresh() assigns the variable with libtbx.env = self (just after pickling itself to libtbx_env), the first time import libtbx.load_env is run the libtbx.env object is replaced with a new unpickled version. libtbx/libtbx_refresh.py unfortunately imports this, guaranteeing that the object being constructed in subsequent libtbx_refresh.py files is never the one being constructed, thus changes to it will never be serialized to disk (and subsequent refresh runs).

To work around this, it walks up the stack to find the env_config.py/refresh function, and pulls out the currently configured environment via the locals dictionary.

Dispatchers

Now, normally, for scripts generated via "console_scripts" entry-points (and others), the "libtbx.dispatcher.script" entry-point is used to specify scripts that are re-exported as an inner libtbx dispatcher by libtbx.env_config.generate_entry_point_dispatchers(). In this case, since the base environment is read-only, the original scripts are never created - and the generate_entry_point_dispatchers() function ignored any entry points that don't have an entry in the path. To work around this, for the case of read-only base environments, the libtbx.env_config.module.extra_command_line_locations field is filled out dynamically, which means that the dispatchers are properly created on subsequent runs of libtbx.refresh.

[codecov]

Codecov Report

Merging #413 (8bbd9eb) into main (c4abbb3) will decrease coverage by 0.72%.
The diff coverage is 0.00%.

❗ Current head 8bbd9eb differs from pull request most recent head bce0abc. Consider uploading reports for the commit bce0abc to get more accurate results

@@            Coverage Diff             @@
##             main     #413      +/-   ##
==========================================
- Coverage   64.65%   63.93%   -0.73%     
==========================================
  Files         177      177              
  Lines       15263    15060     -203     
  Branches     2066     2076      +10     
==========================================
- Hits         9869     9628     -241     
- Misses       4851     4888      +37     
- Partials      543      544       +1     

[phyy-nx]

Hi @ndevenish, I tried this on main first by logging in to machine as a different user and trying to use a readable but not writable conda_base. I got this error:

Processing: "/net/dials/raid1/builder/modules/iota/libtbx_refresh.py"
running develop
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/dev/shm/aaron/conda_base/lib/python3.7/site-packages/test-easy-install-47159.write-test'

This iota problem is known, but I don't have a workaround besides hacking the code (which is what I've done in the past). Is there a flag set up in iota to skip this step? Otherwise I'll put in the iota hack and try these changes again.

[graeme-winter]

Hi @ndevenish, I tried this on main first by logging in to machine as a different user and trying to use a readable but not writable conda_base. I got this error:

Processing: "/net/dials/raid1/builder/modules/iota/libtbx_refresh.py"
running develop
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/dev/shm/aaron/conda_base/lib/python3.7/site-packages/test-easy-install-47159.write-test'

This iota problem is known, but I don't have a workaround besides hacking the code (which is what I've done in the past). Is there a flag set up in iota to skip this step? Otherwise I'll put in the iota hack and try these changes again.

Just sayin' here - iota != dxtbx -> I would not expect issues relating to iota installation to be fixed in a dxtbx issue (you know this, I know) - if it is a known problem (which it appears to be) and has no crosstalk with this issue best raised elsewhere?

It would seem a good idea to allow the changes in here to be merged since they do make the situation better, I gather?

[phyy-nx]

Sure, no question, and that’s what I intended. I’m just wondering if there’s a known workaround without hacking iota. I figured maybe Nick ran a similar test that I did?

…

-Aaron

On Aug 10, 2021, at 11:00 PM, Graeme Winter ***@***.***> wrote:  Hi @ndevenish, I tried this on main first by logging in to machine as a different user and trying to use a readable but not writable conda_base. I got this error: Processing: "/net/dials/raid1/builder/modules/iota/libtbx_refresh.py" running develop error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 13] Permission denied: '/dev/shm/aaron/conda_base/lib/python3.7/site-packages/test-easy-install-47159.write-test' This iota problem is known, but I don't have a workaround besides hacking the code (which is what I've done in the past). Is there a flag set up in iota to skip this step? Otherwise I'll put in the iota hack and try these changes again. Just sayin' here - iota != dxtbx -> I would not expect issues relating to iota installation to be fixed in a dxtbx issue (you know this, I know) - if it is a known problem (which it appears to be) and has no crosstalk with this issue best raised elsewhere? It would seem a good idea to allow the changes in here to be merged since they do make the situation better, I gather? — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub, or unsubscribe.

[ndevenish]

I don't have a workaround - DIALS bootstrap doesn't install iota, so never ran into this issue (I tested by chmod -R a-w conda_base/. IIRC iota also installs as a "real" python package, so would expect that to also fall over.

XIA2 also fails with the same issue so needs to be removed from bootstrap (or the _install_xia2_setup(); _show_xia2_version() part in libtbx_refresh.py commented out). If this solution is acceptable, my proposal is that we add this to xia2 also. I'm reluctant to put this code/process somewhere central as it's really a dirty hack, but it might make sense if used in several places.

Have you looked into building a real python environment e.g. a virtualenv with --system-site-packages (or whatever conda's equivalent for python is) to solve this readonly issue? If you can't do that and can't write to the python environment anywhere, then setting PYTHONPATH on the dispatchers is really the only way to solve this; previous solutions relied on the packages already being on sys.path and the egg-infos being put somewhere else already added to the sys.path, which caused problems with pip e.g. "More than one .egg-info directory found in ....".

[phyy-nx]

Looks good. Thanks.

[phyy-nx]

I disabled iota and ran the build and it looks like it worked.

So I did use virtualenv at the EuXFEL and it worked great. I was given a specific (read only) gcc9.3 build of python3 + openmpi 3.1.6 + an mpi-enabled hdf5 library (1.10.6) and without this python I couldn't use MPI on the Maxwell cluster. I used a virtual environment + pip to install cctbx dependencies and then ran our build.

In fact, it's one of the subjects of my IUCr talk this Friday night at midnight PST (part of the Saturday workshop), if it's ok to advertise a bit :)

Thanks again for the patch.

[ndevenish]

If virtualenv is working well for you, then this change is somewhat surplus to requirements?

[phyy-nx]

Nah, I like it. If I have a read only conda base or even a read only virtualenv it will be useful.

…

On Wed, Aug 11, 2021 at 8:38 AM Nicholas Devenish ***@***.***> wrote: If virtualenv is working well for you, then this change is somewhat surplus to requirements? — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#413 (comment)>, or unsubscribe </~https://github.com/notifications/unsubscribe-auth/ADGY5STTCDXBE77F3Q6QXHLT4KKQXANCNFSM5BWOKHQA> .

[ndevenish]

Okay then!