The issue you're experiencing—where your script freezes in VS Code when running spark.sql locally using Databricks Connect, but works correctly when deployed—can result from several common causes related to Databricks Connect configuration, networking, environment mismatches, and limitations in interactive debugging setups.

Key Possible Causes

1. Databricks Connect Misconfiguration

• If Databricks Connect isn't fully or correctly configured, local commands like spark.sql may get stuck waiting for remote execution that never completes.

• Ensure your Databricks Connect version matches your Databricks Runtime version and Python version requirements.

2. Network and Firewall Issues

• Databricks Connect uses REST calls to communicate with the remote cluster. Local firewall, VPN, or proxy settings might block or slow down communication.

• Check that you can reach Databricks API endpoints from your local machine, and no network interruptions occur during debugging.

3. Python Environment Mismatch

• Incompatible library versions (PySpark, Databricks Connect, etc.) or environment differences between local and cluster can cause local jobs to hang.

• Make sure your local Python, PySpark, and Databricks Connect libraries match the versions on your cluster.

4. VS Code Interactive Debugger Limitations

• Debugging distributed workloads with remote calls can lead to freezes if the debugger tries to step into remote execution, which is not supported or can deadlock the UI.

• Try running the script without the debugger in a terminal (python script.py) to see if it works; if so, the issue may be specific to interactive debugging and not Databricks Connect itself.

5. Resource Initialization Delays or Deadlocks

• Creating a SparkSession using Databricks Connect can require initialization overhead. If the connection or authentication takes too long or hits an internal error, VS Code may freeze.

• Look for logs/output in the Databricks CLI or the Databricks Connect configuration folder for potential errors.

Troubleshooting Steps

• Validate Databricks Connect setup: Use databricks-connect test and check that all tests pass.

• Check network access: Try a simple REST call, such as listing clusters with the Databricks CLI.

• Run outside of debugger: Execute the script in a standard terminal session.

• Upgrade/downgrade libraries: Ensure all required libraries are compatible and up-to-date.

• Increase logging/debug output: Set environment variables to increase verbosity (PYSPARK_DEBUG=1, etc.).

• Clean/reinstall Databricks Connect: Sometimes a fresh install solves hidden dependency issues.

Additional Resources

• [Databricks Connect troubleshooting documentation]

• [Databricks Community discussions for hanging issues]

Summary:
VS Code debugger freezes with spark.sql calls are most often caused by misconfiguration, networking issues, or Python environment mismatch when using Databricks Connect. Try validating each point above, and test running outside the debugger for clues.