2018-08-07 Incompatible Qt Binaries

Summary

In a the first trial deployment of a new environment at LIX, from PyQt5 import QtCore segfaulted. There was a qt binary from conda-forge in the nsls2-tag channel. It is binary incompatible with the pyqt from the anaconda channel. Having picked that version of qt pinned a bunch of other compiled dependencies, we can not install any other version of qt.

Timeline

The offending files were removed from the conda servers. We re-generated the enviroment yml files (using the same metapackages).

The broken environments were removed from the two beamlines where they had been deployed:

ansible -i production beamlines -a "rm -rf /opt/conda_envs/collection-2018-3.0" --limit=16-ID -kKb
ansible -i production beamlines -a "rm -rf /opt/conda_envs/analysis-2018-3.0" --limit=16-ID -kKb
ansible -i production beamlines -a "rm -rf /opt/conda_envs/analysis-2018-3.0" --limit=08-BM -kKb
ansible -i production beamlines -a "rm -rf /opt/conda_envs/collection-2018-3.0" --limit=08-BM -kKb

The new environments were pushed out in the standard way.

Lessons Learned

What went well

  1. Our acceptance test process led us to catch the problem before we walked away from the beamline.

  2. We tested at LIX and TES before a wide rollout.

What went wrong

  1. We didn’t have clarity that conda-forge binaries containing any native extensions should not be copied into nsls2-tag.

Where we got lucky

Action items

These are only sample subheadings. Every action item should have a GitHub issue (even a small skeleton of one) attached to it, so these do not get forgotten.

Process improvements

  1. {{ summary }} [link to github issue]

  2. {{ summary }} [link to github issue]

Documentation improvements

  1. {{ summary }} [link to github issue]

  2. {{ summary }} [link to github issue]

Technical improvements

  1. {{ summary }} [link to github issue]

  2. {{ summary }} [link to github issue]