External dependency on serverless job from Airflow is not working on s3 path and workspace

[manojpatil04]

I am working on use case where we have to run python script from serverless job through Airflow. when we are trying to trigger serverless job and passing external dependency as wheel from s3 path or workspace path it is not working, but on volume it is working.

Is serverless job currently not supporting dependencies from s3 path or workspace.

"environment_key": "Test_environment" }],

                "environments": [

                {

                "environment_key": "Test_environment",

                "spec":

                {

                "client": "1",

                "dependencies":["wheel file path from s3 or workspace"]

                }

                }],

[Walter_C]

It seems that is should be supported, are you using the following format for the URI: { "whl": "s3://my-bucket/library.whl" }?

[manojpatil04]

I am using these external dependencies as below,

"environment_key": "Test_environment",

                "spec":

                {

                "client": "1",

                "dependencies":["wheel file path from s3 or workspace"]

                }

                }],

for volumes it is working, but for S3 and workspace it is not working, we don't need to specifically add "whl" in the dependency.

[Walter_C]

And does the compute has proper permissions to access the S3?

[ColeBerl_37881]

I am having a similar issue. I am on Azure, running a Python Wheel workflow from a whl file stored in a Volume. The workflow runs successfully on a normal job cluster, but fails on serverless with the message `run failed with error message Unexpected error during library installation`. Almost impossible to debug due to the lack of logs when using serverless. 

[Walter_C]

As per serverless compute limitations I can see the following: 

• Task libraries are not supported for notebook tasks. Use notebook-scoped libraries instead. See Notebook-scoped Python libraries.