Ctrl+K

Troubleshooting

Troubleshooting#

There are so many things that can go wrong during normal software installation, and unfortunately there are even more steps during an EasyBuild installation.

Here is my list of problems that occur frequently and took me more than a trivial amount of time to find a solution for:

Illegal instruction#

  • Installation worked, but users report this when trying to run the software

This is most likely to be because our servers do not all have the same chipset. Morgan, at least, has chips with additional processor features compared to Earle (and Cousteau, but that has its own software stack now that it is a teaching server), and when you compile software with typical toolchains, they auto-configure themselves to take advantage of processor features wherever possible - hence the optimization advantage of EasyBuild over Conda.

The straightforward solution to this problem is to only perform installations on the oldest/worst server, i.e.: currently Earle. Currently none of our servers are completely incompatible with each other, but this could in theory happen, at which point either multiple software versions would be needed, or you would have to customise every installation to explicitly not use certain compilation flags.

There will also be an issue when software that uses CUDA is required for use of the Dayhoff graphics cards - that can only be installed on Dayhoff, but any dependencies installed at the same time will be optimized for Dayhoff and may cause issues on other servers. This can be solved by ensuring that all dependencies are installed elsewhere before the software that requires CUDA.

Dependencies from other toolchains#

  • EasyBuild claims that a module doesn’t exist for the toolchain you are using, but you think a compatible one does exist

Toolchains follow a hierarchy, as seen in the diagram in that section, which means that you can depend on, for instance, a GCC-compiled modulefor a foss-compiled module. However, sometimes this doesn’t work, and I’m not entirely sure why, but I think it happens when you write your own EasyConfig that uses a lower toolchain and try to depend on it, or possibly you have compilation dependencies outside of the toolchain that cause the issue.

Whatever the reason, the way around it is to specify the toolchain of a dependency. Really this is just specifying the suffix of the dependency and so it can come in useful in other situations.

# For instance
('HMMER', '3.3.2', '-foss-2020b', True)

Python dependencies#

  • Installation fails due to missing python dependencies

Although EasyBuild will use pip to install and test python packages, it doesn’t install package dependencies - you must specify them in the EasyConfig. This can be a rabbit hole, where you put in the listed dependencies for a piece of software, and it turns out that they have dependencies and so on all the way down.

The only way I have found around doing this step-by-step is as follows:

  • Load all of your python-relevant module dependencies, i.e.: the python version, SciPy-bundle, Biopython etc.

  • Create a new virtual environment:

# Create a new environment
python -m venv /path/to/venv

# Activate it
source /path/to/venv/bin/activate

  • Use pip list to write a list of existing packages to a file

  • Use pip to install the packages you are aware you need - dependencies will automatically install

  • Use pip list once more to write a list of packages after installation

  • Compare the two lists and you get the list of all dependencies you will need to put into your EasyConfig

Python package names#

  • Installation fails at testing stage when a python package can’t be found

When a python package is part of an installation, it is tested by default to see if the package can be loaded within python. Some python packages can’t be loaded this way, or worse, they have a different name when loaded in this way. Fortunately there is an argument you can give to get around this in your EasyConfig.

# A couple of examples from my EasyConfig for Sphinx 7.2.6
# When the package cannot be ordinarily loaded:
('sphinxcontrib_applehelp', '1.0.8', {
    'modulename': False}),

# When it has a different name:
('beautifulsoup4', '4.12.3', {
    'modulename': 'bs4'}),

Patches#

  • Installation fails and it’s the code’s fault

Sometimes the installation log indicates that the error is due to some obvious bad code. For instance, the deliberate use of a compilation flag that the processor doesn’t support, a reference to a deprecated function in a dependency, or just bad code. If you can solve this with manual intervention - literally editing the code before it’s compiled - then you can write a patch to make your EasyConfig robust and reproducible.

  • Download the source package, or run your EasyConfig with --stop source

  • Decompress the package and make a copy cp -r source source.orig

  • Make whatever changes are required in source for the software to compile correctly

  • Run a diff command to generate the patch diff -ruN source.org source > name-version-toolchain.patch

  • Add the patch to your EasyConfig

Licensed software#

  • EasyBuild will tell you that it cannot find the source file (sometimes explaining why)

Sometimes software requires a license, although not necessarily one you have to pay for. EasyBuild will not download sources for licensed software and you will have to download the source manually and then use --sourcepath /path/to/source to tell it where to find the file.

Contents