Since moving to dbr 14.3 with python jobs I don't see the stack trace for exceptions

[alonisser]

or even the logs don't contain the error line I see (downloaded all logs file from the UI and checked them)

How can I see the stacktrace? it's essential to debug certain issues

[Kaniz_Fatma]

Hi @alonisser, 

• Ensure that any Python modules or packages your job relies on are available on all worker nodes. One way to achieve this is by including them in the Spark job’s dependencies.
• You can specify the --py-files option when running your streaming job. This option allows you to provide a comma-separated list of Python files that will be distributed to...¹.
• By doing this, you ensure that the necessary modules are available during job execution.
• Sometimes, issues can arise if your PATH is configured in a way that causes commands like spark-shell to run a previously installed binary instead of the one provided with Databricks Connect.
• Make sure that the Databricks Connect binaries take precedence in your PATH or remove any conflictin...
• If your code accidentally holds on to the SparkContext or SparkSession within a user-defined function (UDF), it can lead to job failures.
• Make sure that your UDFs do not inadvertently retain references to these objects
• Try setting the following configurations at the cluster level and then retry your job:
  • spark.databricks.python.defaultPythonRepl
  • pythonshell
  • spark.databricks.pyspark.py4j.pinnedThread
• Adjusting these settings might help resolve any issues related to Python execution

If you encounter any further issues or need additional assistance, feel free to ask! 

[alonisser]

Thanks for the answer, but i fail to see what it has to do with my questions. it's not a "general python error", I run lots of jobs with python on Databricks clusters and know how to run python jobs and dependencies, I'm pointing to a specific issue - with dbr 14, which I don't; encounter in older jobs