How to run a python task that uses click for CLI operations

[Garrus990]

Hey,

in my application I am using click to facilitate CLI operations. It works locally, in notebooks, when scripts are run locally, but it fails in Databricks. I defined a task that, as an entrypoint, accepts the file where the click-decorated function resides, I pass the required params and everything works just fine up until the end of the task's run. Then the following error is raised:

SystemExit: 0
An exception has occurred, use %tb to see the full traceback.

---------------------------------------------------------------------------
Exit                                      Traceback (most recent call last)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:1088, in BaseCommand.main(self, args, prog_name, complete_var, standalone_mode, windows_expand_args, **extra)
   1080             return rv
   1081         # it's not safe to `ctx.exit(rv)` here!
   1082         # note that `rv` may actually contain data like "1" which
   1083         # has obvious effects
   (...)
   1086         # even always obvious that `rv` indicates success/failure
   1087         # by its truthiness/falsiness
-> 1088         ctx.exit()
   1089 except (EOFError, KeyboardInterrupt) as e:
   1090     echo(file=sys.stderr)

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:692, in Context.exit(self, code)
    690 def exit(self, code: int = 0) -> "te.NoReturn":
    691     """Exits the application with a given exit code."""
--> 692     raise Exit(code)

Exit: 0

During handling of the above exception, another exception occurred:

SystemExit: 0

I want to reiterate that the task is successful in the sense that it executes what is requested and stores the data I want to store. The error appears AFTER the code is executed and it isn't even manifested in the run's logs - they are impeccable. In click's documentation it says: "This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught." I was trying to catch the execution, but it's all in vain as, most likely, I do not even know where to catch.

[VZLA]

The SystemExit issue you’re seeing is typical with Click, as it’s designed for standalone CLI applications and automatically calls sys.exit() after running a command. This behavior can trigger SystemExit exceptions in non-CLI environments, like Databricks notebooks or task workflows, where an exit might be interpreted as an error.

• Which Databricks Runtime Version is being used, and does it match your local Python version?
• Are you running this Click command as a scheduled job, a triggered job, or manually in a notebook?
• Is the Click application using a single command or multiple nested commands?
• Do any Click options have default values or callbacks that might behave differently across environments?
• When running in Databricks, does the task log show a complete run before the SystemExit error, or is the error interrupting the task?
• Have you tested different Click settings, such as setting standalone_mode=False in the Click function, to see if behavior changes?