topic Re: How to run a python task that uses click for CLI operations in Data Engineering

How to run a python task that uses click for CLI operations

Garrus990 — Wed, 16 Oct 2024 12:44:18 GMT

Hey,

in my application I am using click to facilitate CLI operations. It works locally, in notebooks, when scripts are run locally, but it fails in Databricks. I defined a task that, as an entrypoint, accepts the file where the click-decorated function resides, I pass the required params and everything works just fine up until the end of the task's run. Then the following error is raised:

SystemExit: 0 An exception has occurred, use %tb to see the full traceback. --------------------------------------------------------------------------- Exit Traceback (most recent call last) File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:1088, in BaseCommand.main(self, args, prog_name, complete_var, standalone_mode, windows_expand_args, **extra) 1080 return rv 1081 # it's not safe to `ctx.exit(rv)` here! 1082 # note that `rv` may actually contain data like "1" which 1083 # has obvious effects (...) 1086 # even always obvious that `rv` indicates success/failure 1087 # by its truthiness/falsiness -> 1088 ctx.exit() 1089 except (EOFError, KeyboardInterrupt) as e: 1090 echo(file=sys.stderr) File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:692, in Context.exit(self, code) 690 def exit(self, code: int = 0) -> "te.NoReturn": 691 """Exits the application with a given exit code.""" --> 692 raise Exit(code) Exit: 0 During handling of the above exception, another exception occurred: SystemExit: 0

I want to reiterate that the task is successful in the sense that it executes what is requested and stores the data I want to store. The error appears AFTER the code is executed and it isn't even manifested in the run's logs - they are impeccable. In click's documentation it says: "This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught." I was trying to catch the execution, but it's all in vain as, most likely, I do not even know where to catch.

Re: How to run a python task that uses click for CLI operations

VZLA — Thu, 31 Oct 2024 18:15:42 GMT

The SystemExit issue you’re seeing is typical with Click, as it’s designed for standalone CLI applications and automatically calls sys.exit() after running a command. This behavior can trigger SystemExit exceptions in non-CLI environments, like Databricks notebooks or task workflows, where an exit might be interpreted as an error.

Which Databricks Runtime Version is being used, and does it match your local Python version?
Are you running this Click command as a scheduled job, a triggered job, or manually in a notebook?
Is the Click application using a single command or multiple nested commands?
Do any Click options have default values or callbacks that might behave differently across environments?
When running in Databricks, does the task log show a complete run before the SystemExit error, or is the error interrupting the task?
Have you tested different Click settings, such as setting standalone_mode=False in the Click function, to see if behavior changes?

Re: How to run a python task that uses click for CLI operations

Rodra — Fri, 09 May 2025 14:26:25 GMT

Hello @VZLA ,

I am having the exact same problem. Here is the answer for your questions:

- Which Databricks Runtime Version is being used, and does it match your local Python version?
I am using 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12) runtime

- Are you running this Click command as a scheduled job, a triggered job, or manually in a notebook?
A triggered job using DatabricksSubmitRunOperator

- Is the Click application using a single command or multiple nested commands?
The command is brt evaluate --config /path/to/config --set some.value.changed.to=2

- Do any Click options have default values or callbacks that might behave differently across environments?
As long as I can tell, it doesn't.

- When running in Databricks, does the task log show a complete run before the SystemExit error, or is the error interrupting the task?
The error does not stop the task, which actually executes successfully. It happens right after the task finishes.

- Have you tested different Click settings, such as setting standalone_mode=False in the Click function, to see if behavior changes?

This is how my cli.py __main__ is implemented:

if __name__ == "__main__":
    cli(standalone_mode=False)

I tried different approaches to solve it, but I keep getting this error messages at the end of the job, which ends up in Failed state:

...
2025-05-09 14:15:50,857 - brt.cli - INFO - ✔ EvaluationJob finished successfully
2025-05-09 14:15:51,366 - root - ERROR - /databricks/python/lib/python3.11/site-packages/IPython/core/interactiveshell.py:3534: UserWarning:
2025-05-09 14:15:51,367 - root - ERROR - To exit: use 'exit', 'quit', or Ctrl-D.

Re: How to run a python task that uses click for CLI operations

robbe — Sat, 15 Nov 2025 09:18:22 GMT

Hi @Garrus990 @Rodra have you guys found a solution for this issue? I'm also having the same problem on Sevrerless compute v4.

Interestingly enough it seems working on job cluster with runtime 16.4 LTS 😕.

Re: How to run a python task that uses click for CLI operations

johalnes — Wed, 26 Nov 2025 14:39:26 GMT

Having the same issue with serverless compute. So if anyone finds a solution, please share! 😊

Re: How to run a python task that uses click for CLI operations

Garrus990 — Wed, 03 Dec 2025 12:36:02 GMT

Hey guys,

I think I managed to find a workaround. I will leave it here for everyone that is seeking the same answers, including future me.

What I did is basically this piece of code:

def main(): try: assign_variants(standalone_mode=False) except SystemExit as e: # Only suppress exit code 0 (success) if e.code != 0: raise # For success, exit cleanly sys.exit(0)

Where assign_variants is the name of my Click-decorated function. The function can be called in the script or pointed as an endpoint.