Hi,

We are working on a migration project from Cloudera to Databricks.

All our code is in .py files and we decided to keep the same in Databricks as well and try to execute the same from GIT through Databricks workflows.

We have two kinds of exit functionality requirements:

1. Soft exit with sys.exit(0)- when the job criteria meet some condition, then we do soft exist with sys.exit(0), which basically terminates the job softly by marking the job as successful

2. Job Terminate with sys.exit(1) - when the job criteria meet some condition, then we do terminate the job with sys.exit(1), which basically terminates the job by marking the job as failed.

the above criteria work as expected in any Python environment but not in Databricks.

In Databricks, both sys.exit(1) as well as sys.exit(0) are marked as Job failed.

I read an article in Community and somebody mentioned that "Usage of spark.stop(), sc.stop() , System.exit() in your application can cause this behavior. Databricks manages the context shutdown on its own. Forcefully closing it can cause this abrupt behavior."

Link: https://community.databricks.com/t5/data-engineering/why-do-i-see-my-job-marked-as-failed-on-the-dat...

If this is true, then what is the best alternative to achieve sys.exit(0) in Databricks?

Any help would greatly be appreciated.

FYI: we have code in .py files and we don't want to use the Notebooks for PROD jobs.

Here is an example to execute and see the behavior: