topic Re: External dependency on serverless job from Airflow is not working on s3 path and workspace in Data Engineering

External dependency on serverless job from Airflow is not working on s3 path and workspace

manojpatil04 — Wed, 11 Dec 2024 04:57:58 GMT

I am working on use case where we have to run python script from serverless job through Airflow. when we are trying to trigger serverless job and passing external dependency as wheel from s3 path or workspace path it is not working, but on volume it is working.

Is serverless job currently not supporting dependencies from s3 path or workspace.

"environment_key": "Test_environment" }],

"environments": [

{

"environment_key": "Test_environment",

"spec":

{

"client": "1",

"dependencies":["wheel file path from s3 or workspace"]

}

}],

Re: External dependency on serverless job from Airflow is not working on s3 path and workspace

Walter_C — Wed, 11 Dec 2024 11:39:47 GMT

It seems that is should be supported, are you using the following format for the URI: { "whl": "s3://my-bucket/library.whl" }?

Re: External dependency on serverless job from Airflow is not working on s3 path and workspace

manojpatil04 — Wed, 18 Dec 2024 09:17:57 GMT

I am using these external dependencies as below,

"environment_key": "Test_environment",

"spec":

{

"client": "1",

"dependencies":["wheel file path from s3 or workspace"]

}

}],

for volumes it is working, but for S3 and workspace it is not working, we don't need to specifically add "whl" in the dependency.

Re: External dependency on serverless job from Airflow is not working on s3 path and workspace

Walter_C — Wed, 18 Dec 2024 14:14:58 GMT

And does the compute has proper permissions to access the S3?

Re: External dependency on serverless job from Airflow is not working on s3 path and workspace

ColeBerl_37881 — Wed, 18 Dec 2024 16:23:48 GMT

I am having a similar issue. I am on Azure, running a Python Wheel workflow from a whl file stored in a Volume. The workflow runs successfully on a normal job cluster, but fails on serverless with the message `run failed with error message Unexpected error during library installation`. Almost impossible to debug due to the lack of logs when using serverless.

Re: External dependency on serverless job from Airflow is not working on s3 path and workspace

Walter_C — Wed, 18 Dec 2024 16:36:14 GMT

As per serverless compute limitations I can see the following:

Task libraries are not supported for notebook tasks. Use notebook-scoped libraries instead. See Notebook-scoped Python libraries.