Temporary deps compatibility

[kessler-frost]

Recently I found that a ModuleNotFound error was thrown on the logs even though a pip Deps was passed to the electron. We might need to verify whether the deps functions are running or not as part of the main function on lambda.

[venkatBala]

@kessler-frost for the lambda, just a couple of points that I faced when working on this

1. AWS runs all lambda functions on a server whose filesystem is strictly read-only. (The only writable directory that I found was /tmp) . I am inclined to think that DepsPips might have failed.. but if you found that they did run in your tests, then where the stuff gets installed might not be in PYTHONPATH

2. As per AWS docs, all lambda deps are to be parcelled up in either as docker image or a zip deployment archive and AWS run-time sets up the environment in which the lambda_handler runs. Modifying the environment as the lambda runs may not be possible. I have not tried to do this, so not a 100% sure whether this can be done, maybe you've seen different in your tests

hope this helps!

[cjao]

I also experienced this recently. Lambda's incompatibility with Pip Deps is less of a problem than the fact that the dep installation seems to fail silently.

[cjao]

The Pip Deps are getting installed in Lambda's environment according to pip:

@ct.electron(executor="awslambda", deps_pip=ct.DepsPip(packages=["scipy", "matplotlib"]))
def task():
    import subprocess
    proc = subprocess.run(["pip", "list"], capture_output=True, check=True)
    return proc.stdout.decode("utf-8")

@ct.lattice
def workflow():
    return task()

dispatch_id = ct.dispatch(workflow)
result_object = ct.get_result(dispatch_id, wait=True)
print(result_object.result)

Package         Version
--------------- -------
boto3           1.20.32
botocore        1.23.32
contourpy       1.0.5
cycler          0.11.0
fonttools       4.37.2
jmespath        0.10.0
kiwisolver      1.4.4
matplotlib      3.6.0
numpy           1.23.3
packaging       21.3
Pillow          9.2.0
pip             22.0.4
pyparsing       3.0.9
python-dateutil 2.8.2
rapid_client    0.0.0
s3transfer      0.5.2
scipy           1.9.1
setuptools      56.0.0
six             1.16.0
urllib3         1.26.6

[cjao]

Here is Lambda's sys.path:

['/var/task', '/opt/python/lib/python3.8/site-packages', '/opt/python', '/var/runtime', '/var/lang/lib/python38.zip', '/var/lang/lib/python3.8', '/var/lang/lib/python3.8/lib-dynload', '/var/lang/lib/python3.8/site-packages', '/opt/python/lib/python3.8/site-packages', '/opt/python']

However, pip packages are installed under /tmp/.local/lib/python.38/site-packages:

Package         Version Location                                Installer
--------------- ------- --------------------------------------- ---------
boto3           1.20.32 /var/runtime                            pip
botocore        1.23.32 /var/runtime                            pip
jmespath        0.10.0  /var/runtime                            pip
numpy           1.23.3  /tmp/.local/lib/python3.8/site-packages pip
pip             22.0.4  /var/lang/lib/python3.8/site-packages   pip
python-dateutil 2.8.2   /var/runtime                            pip
rapid_client    0.0.0   /var/lang/lib/python3.8/site-packages
s3transfer      0.5.2   /var/runtime                            pip
scipy           1.9.1   /tmp/.local/lib/python3.8/site-packages pip
setuptools      56.0.0  /var/lang/lib/python3.8/site-packages   pip
six             1.16.0  /var/runtime                            pip
urllib3         1.26.6  /var/runtime                            pip

It should be enough to add this directory to sys.path in the lambda function handler or using a Call dep.

[cjao]

This works:

def append_python_path():
    import sys
    sys.path.append("/tmp/.local/lib/python3.8/site-packages")
    
calldep = ct.DepsCall(append_python_path)

@ct.electron(executor="awslambda", call_before=[calldep], deps_pip=ct.DepsPip(packages=["numpy"]))
def task():
    import numpy
    return numpy.__version__

@ct.lattice
def workflow():
    return task()

dispatch_id = ct.dispatch(workflow)()

ct.get_result(dispatch_id, wait=True).result

1.23.3

Of course the user should keep in mind that /tmp is limited to 512mb so pytorch probably wouldn't fit there

[venkatBala]

@cjao I wonder if PYTHONPATH environment variable can be modified in the base image with this custom path (with the appropriate python version), then it will work hopefully without any additional modifications... thoughts?

[cjao]

@AlejandroEsquivel

[AlejandroEsquivel]

@cjao I think we can fit up to 10 GB in lambda storage based on the following aws docs

@venkatBala we are currently not using the covalent base executor image however once we do I think modifying the PYTHONPATH would work.

Maybe the quickest fix for this however until we use the container approach for lambda would be to append the sys path in the PYTHON_EXEC_SCRIPT @cjao