Databricks Community

Angus-Dawson · 07-26-2024

I'm trying to use Databricks Connect to run queries on Delta Tables locally. However, SQL queries using spark.sql don't seem to work properly, even though spark.read.table works.>>> from databricks.connect import DatabricksSession>>> spark = Databric...

Angus-Dawson

I managed to work around it in Poetry by using optional dependency groups, and then when I want to switch between Databricks Connect and local PySpark functionality I run this:poetry install --with <group x> --without <group y> --sync

Angus-Dawson · 09-16-2024

This is ridiculous. It's absolutely unacceptable as "intended behavior" for a professional software package to clobber the functionality of another package just by being installed.

Angus-Dawson · 09-11-2024

Thank you so much! That did the trick.

Angus-Dawson · 08-01-2024

Hi Kaniz,Could you clarify what you mean? I assume the session is correctly initialized since I can use `spark.read.table` without issue, but I don't know what else I should check.

Databricks Community

User Stats

User Activity

PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect

Re: Installing Databricks Connect breaks pyspark local cluster mode

Re: Installing Databricks Connect breaks pyspark local cluster mode

Re: PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect

Re: PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect