Yes, I did the same. However, so we have to switch the code from local (VS Code) implementation to Databricks runs (Jobs/Workflow). @Kaniz : Could you check this new issue?
@Kaniz : We exactly use your second solution. And we get same issuefrom databricks.connect import DatabricksSessionspark = DatabricksSession.builder.getOrCreate()schema = StructType([StructField('category', StringType(), True), StructField('weight', ...
Hi RobinDo you use Databricks Connect creating spark Session?from databricks.connect import DatabricksSessionspark = DatabricksSession.builder.getOrCreate()We are facing same issue on single user access cluster and in jobs.
