How to run a python task that uses click for CLI operations

[Garrus990]

Hey,

in my application I am using click to facilitate CLI operations. It works locally, in notebooks, when scripts are run locally, but it fails in Databricks. I defined a task that, as an entrypoint, accepts the file where the click-decorated function resides, I pass the required params and everything works just fine up until the end of the task's run. Then the following error is raised:

SystemExit: 0
An exception has occurred, use %tb to see the full traceback.

---------------------------------------------------------------------------
Exit                                      Traceback (most recent call last)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:1088, in BaseCommand.main(self, args, prog_name, complete_var, standalone_mode, windows_expand_args, **extra)
   1080             return rv
   1081         # it's not safe to `ctx.exit(rv)` here!
   1082         # note that `rv` may actually contain data like "1" which
   1083         # has obvious effects
   (...)
   1086         # even always obvious that `rv` indicates success/failure
   1087         # by its truthiness/falsiness
-> 1088         ctx.exit()
   1089 except (EOFError, KeyboardInterrupt) as e:
   1090     echo(file=sys.stderr)

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:692, in Context.exit(self, code)
    690 def exit(self, code: int = 0) -> "te.NoReturn":
    691     """Exits the application with a given exit code."""
--> 692     raise Exit(code)

Exit: 0

During handling of the above exception, another exception occurred:

SystemExit: 0

I want to reiterate that the task is successful in the sense that it executes what is requested and stores the data I want to store. The error appears AFTER the code is executed and it isn't even manifested in the run's logs - they are impeccable. In click's documentation it says: "This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught." I was trying to catch the execution, but it's all in vain as, most likely, I do not even know where to catch.

[VZLA]

The SystemExit issue you’re seeing is typical with Click, as it’s designed for standalone CLI applications and automatically calls sys.exit() after running a command. This behavior can trigger SystemExit exceptions in non-CLI environments, like Databricks notebooks or task workflows, where an exit might be interpreted as an error.

• Which Databricks Runtime Version is being used, and does it match your local Python version?
• Are you running this Click command as a scheduled job, a triggered job, or manually in a notebook?
• Is the Click application using a single command or multiple nested commands?
• Do any Click options have default values or callbacks that might behave differently across environments?
• When running in Databricks, does the task log show a complete run before the SystemExit error, or is the error interrupting the task?
• Have you tested different Click settings, such as setting standalone_mode=False in the Click function, to see if behavior changes?

[Rodra]

Hello @VZLA ,

I am having the exact same problem. Here is the answer for your questions:

- Which Databricks Runtime Version is being used, and does it match your local Python version?
I am using 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12) runtime

- Are you running this Click command as a scheduled job, a triggered job, or manually in a notebook?
A triggered job using DatabricksSubmitRunOperator

- Is the Click application using a single command or multiple nested commands?
The command is brt evaluate --config /path/to/config --set some.value.changed.to=2

- Do any Click options have default values or callbacks that might behave differently across environments?
As long as I can tell, it doesn't.

- When running in Databricks, does the task log show a complete run before the SystemExit error, or is the error interrupting the task?
The error does not stop the task, which actually executes successfully. It happens right after the task finishes.

- Have you tested different Click settings, such as setting standalone_mode=False in the Click function, to see if behavior changes?

This is how my cli.py __main__ is implemented:

if __name__ == "__main__":
    cli(standalone_mode=False)

I tried different approaches to solve it, but I keep getting this error messages at the end of the job, which ends up in Failed state:

...
2025-05-09 14:15:50,857 - brt.cli - INFO - ✔ EvaluationJob finished successfully
2025-05-09 14:15:51,366 - root - ERROR - /databricks/python/lib/python3.11/site-packages/IPython/core/interactiveshell.py:3534: UserWarning:
2025-05-09 14:15:51,367 - root - ERROR - To exit: use 'exit', 'quit', or Ctrl-D.

[robbe]

Hi @Garrus990 @Rodra have you guys found a solution for this issue? I'm also having the same problem on Sevrerless compute v4.

Interestingly enough it seems working on job cluster with runtime 16.4 LTS 😕.