Databricks Community

manojpatil04

I am working on use case where we have to run python script from serverless job through Airflow. when we are trying to trigger serverless job and passing external dependency as wheel from s3 path or workspace path it is not working, but on volume it ...

manojpatil04 · 12-03-2024

I am using GCP-Databricks 14days free trail, what would happened after 14days trail is over. what happened with the already created workspaces.

manojpatil04 · 12-03-2024

I am working on setting up GCP Databricks and successfully created first GCP-Databricks workspace, but what I observed is it is incurring additional charges even i am using 14days free trail. It is GKE cluster with 4 nodes which are spin up as part o...

manojpatil04

I am using these external dependencies as below,"environment_key": "Test_environment", "spec": { "client": "1", "dependencies":["wheel file path from s3 or workspace"] } ...

manojpatil04

@GabrieleMuciacc , in case of serverless compute job this can be pass as external dependency you can't use libraries. "tasks": [{ "task_key": task_id, "spark_python_task": { "python_file": py_file, ...

manojpatil04 · 12-04-2024

Thank you @BigRoux,Just want to dig more into this as is there any way to reduce this nodes using CLI or creating customer managed network.

Databricks Community

User Stats

User Activity

External dependency on serverless job from Airflow is not working on s3 path and workspace

GCP Databricks 14days free trail

GCP Databricks GKE cluster with 4 nodes

Re: External dependency on serverless job from Airflow is not working on s3 path and workspace

Re: Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`

Re: GCP Databricks GKE cluster with 4 nodes