Hey,
in my application I am using click to facilitate CLI operations. It works locally, in notebooks, when scripts are run locally, but it fails in Databricks. I defined a task that, as an entrypoint, accepts the file where the click-decorated function resides, I pass the required params and everything works just fine up until the end of the task's run. Then the following error is raised:
SystemExit: 0
An exception has occurred, use %tb to see the full traceback.
---------------------------------------------------------------------------
Exit Traceback (most recent call last)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:1088, in BaseCommand.main(self, args, prog_name, complete_var, standalone_mode, windows_expand_args, **extra)
1080 return rv
1081 # it's not safe to `ctx.exit(rv)` here!
1082 # note that `rv` may actually contain data like "1" which
1083 # has obvious effects
(...)
1086 # even always obvious that `rv` indicates success/failure
1087 # by its truthiness/falsiness
-> 1088 ctx.exit()
1089 except (EOFError, KeyboardInterrupt) as e:
1090 echo(file=sys.stderr)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/click/core.py:692, in Context.exit(self, code)
690 def exit(self, code: int = 0) -> "te.NoReturn":
691 """Exits the application with a given exit code."""
--> 692 raise Exit(code)
Exit: 0
During handling of the above exception, another exception occurred:
SystemExit: 0
I want to reiterate that the task is successful in the sense that it executes what is requested and stores the data I want to store. The error appears AFTER the code is executed and it isn't even manifested in the run's logs - they are impeccable. In click's documentation it says: "This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught." I was trying to catch the execution, but it's all in vain as, most likely, I do not even know where to catch.
