Ensure that your Databricks Connect is properly set up and is using the correct version compatible with your cluster’s runtime. For VS Code, any mismatches between the installed databricks-connect Python package version and the cluster runtime could lead to freezes or errors.

Also, Add detailed logging in your code to help identify where the freeze might be happening. You can add logs around the spark.sql operations to monitor the query execution phases and catch errors, if any.