Issue with deployment of pip installed python packages with command line tools

[rbeucher]

The med-utils package which provides the ilamb-tree-generator command line tool is currently installed using pip in our access-med environment:

- pip:
  - railroad-diagrams ### Unlisted dependency of pip and pyparsing
  - git+https://github.com/ACCESS-NRI/MED-utils

The problem is that during install/deploy a shebang is created:

#!/jobfs/109596945.gadi-pbs/squashfs-root/opt/conda/access-med-0.7/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from access_med_utils.ilamb import tree_generator
if __name__ == '__main__':
   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
   sys.exit(tree_generator())

This leads to the following error:

bash: /opt/conda/access-med-0.7/bin/ilamb-tree-generator: /jobfs/109596945.gadi-pbs/squashfs-root/opt/conda/access-med-0.7/bin/python: bad interpreter: No such file or directory

@rhaegar325 I think that's the cause of the issue but I could be wrong.
@dsroberts do you have a suggestion on how to fix this?

[dsroberts]

Interesting. The --relocate-prefix argument in this line is supposed to handle that. I guess it doesn't work for pip packages installed that way? I notice you have MED-utils on your conda channel, why not install from there?

[rbeucher]

Yes that is what I thought we should do.

[dsroberts]

I had other ideas in case --relocate-prefix didn't work out. My first thought was to maintain a fork of micromamba that doesn't resolve path symlinks, or to run sed on every script in the bin directory. Neither of these ideas are good, so but if --relocate-prefix doesn't work one of those might be the only option without reverting back to mamba and a base environment.

[rbeucher]

Hi @dsroberts ,

I am having a similar problem.
I have tried to deploy a new version of our ESMValTool environment

module use /g/data/xp65/public/modules
module load conda/esmvaltool-0.4

Build and deployment went OK. I can also import the python packages from a notebook on ARE.

However, I cannot run the esmvaltool command from the CLI. I get:

esmvaltool version  
/g/data/xp65/public/apps/med_conda_scripts/esmvaltool-0.4.d/bin/esmvaltool: /g/data/xp65/public/apps/med_conda/envs/esmvaltool-0.4/bin/esmvaltool: /jobfs/120579340.gadi-pbs/squashfs-root/opt/conda/esmvaltool-0.4/bin/python: bad interpreter: No such file or directory
/g/data/xp65/public/apps/med_conda_scripts/esmvaltool-0.4.d/bin/esmvaltool: line 121: /g/data/xp65/public/apps/med_conda/envs/esmvaltool-0.4/bin/esmvaltool: Success

I was initially installing ESMValTool with pip in my environment file, I moved it to get ESMValTool from conda-forge but I still have the same issue.
See environment conda/esmvatool-0.4b which returns the same error.

I think the MED-condaenv is in sync with your branch. I don't understand what is going on.
Would you be able to help?

Thanks!

[rbeucher]

I have added the logs in /scratch/hh5/rb5533 if that helps

[Thomas-Moore-Creative]

In the past I've used the very useful CLEX HH5 analysis environments and have been able to pip install my own local packages on top.

To date I'm struggling to use the ACCESS-NRI version of these environments ( which I assume are the /g/data/xp65/public/modules & conda/access-med ARE settings ) with my local packages via pip.

Is this issue here possibly related to these problems?

[rbeucher]

Hi @Thomas-Moore-Creative ,

It is a different problem.

The analysis3 environment in xp65 are containerised environment. We haven't provided a workflow to use pip installed packages on top. It should be possible but is not currently supported.

@charles-turner-1 @jo-basevi do you want to comment on this?

Romain

[charles-turner-1]

I've actually noticed some issues myself with pip installing over the top of the analysis 3 environment with the new analysis3-25.04 environment. No issues with 25.02 or lower.

I'm not sure exactly what the cause is yet - been tied up with segfaults.

2 workarounds I can suggest:

1. Use 25.02 or lower.
2. If you want to use the most up to date environment, you can load local modules with this snippet of code:

import importlib.util
import sys

mpath = "/home/189/ct1163/intake-esm/intake_esm/__init__.py"


def import_from_path(module_name, file_path):
    """Import a module given its name and file path."""
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    return module

intake_esm = import_from_path("intake_esm", mpath)
intake_esm.__version__  # Only to check that the version I expect is loaded - versioneer appends a hash to this version - this will vary by package. 

This loads my local intake-esm install & does work on 25.04.

EDIT/ NB:

I'm actually trying to debug those segfault issues in intake-esm, which have found their way through to 25.04, so as of right now I would consider 25.04 to be broken w.r.t. anything intake catalogue related.

[jo-basevi]

I'm not much help with this specific module sorry, but I've been creating virtual environments on top of the payu modules (which effectively uses the same containerised setup to the environments deployed in this repository) for a number of CI tests. The tests load the module, runs python3 -m venv <path/to/test-venv> --system-site-packages && source <path/to/test-venv>/bin/activate to create and activate the virtual environment, and then pip installs any test packages. So far, if I've had issues, recreating the virtual environments seems to work for me.

[charles-turner-1]

You're not typically in a notebook are you @jo-basevi?

I've got a strong feeling the issues I'm having (I've just checked with python -c) are specific to jupyter notebooks - presumably some to do with the changes to notebook conda kernels stuff.

I'm guessing that @Thomas-Moore-Creative is also having issues specifically in notebooks?

[Thomas-Moore-Creative]

I'm guessing that @Thomas-Moore-Creative is also having issues specifically in notebooks?

Correct @charles-turner-1. I'm in notebooks on ARE, historically using hh5 analysis 3 environments and now xp65 access-med, using pipto install my own local code on top via something like:

!pip install --user -e /g/data/path_to_my_code_repo

[jo-basevi]

You're not typically in a notebook are you @jo-basevi?

Yes, I haven't been using jupyter notebooks, so have no idea on what the error could be sorry.

[rbeucher]

Hi @Thomas-Moore-Creative,

Just a quick update, I have tested the install of packages using pip from a notebook on ARE

!pip install mypackage

It defaults to installing to user installation (e.g. /home/563/$USER/.local/lib/python3.11/site-packages) because the environments are containerised and read-only.
It works for me using

module use /g/data/xp65/public/modules
module load conda/analysis3-25.04

R