Debug UDFs using VSCode extension

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

I am trying to debug my python script using Databricks VSCode extension. I am using udf and pandas_udf in my script. Everything works fine except when the execution gets to the udf and pandas_udf usages. It then complains that "SparkContext or SparkSession should be created first.". I did some research and looks like SparkContext is not supported in Databricks Connect (read https://docs.databricks.com/en/dev-tools/databricks-connect/python/limitations.html). On the other hand I also found https://docs.databricks.com/en/dev-tools/databricks-connect/python/udf.html that says you can use UDFs with Databricks Connect. So I am confused. If you cannot debug UDFs with vscode extensions, how do you guys typically do this? Thanks for your help.