Databricks Community

Angus-Dawson · 06-04-2025

Per the release notes for Databricks Runtime 16.4 LTS, the environment has Apache Spark 3.5.2 and Delta Lake 3.3.1:https://docs.databricks.com/aws/en/release-notes/runtime/16.4ltsHowever, Delta Lake 3.3.1 is built on Spark 3.5.3; the newest version o...

Angus-Dawson · 07-26-2024

I'm trying to use Databricks Connect to run queries on Delta Tables locally. However, SQL queries using spark.sql don't seem to work properly, even though spark.read.table works.>>> from databricks.connect import DatabricksSession>>> spark = Databric...

Angus-Dawson · 10-28-2024

I managed to work around it in Poetry by using optional dependency groups, and then when I want to switch between Databricks Connect and local PySpark functionality I run this:poetry install --with <group x> --without <group y> --sync

Angus-Dawson · 09-16-2024

This is ridiculous. It's absolutely unacceptable as "intended behavior" for a professional software package to clobber the functionality of another package just by being installed.

Angus-Dawson · 09-11-2024

Thank you so much! That did the trick.

Angus-Dawson · 08-01-2024

Hi Kaniz,Could you clarify what you mean? I assume the session is correctly initialized since I can use `spark.read.table` without issue, but I don't know what else I should check.

Databricks Community

User Stats

User Activity

Databricks Runtime 16.4 LTS has inconsistent Spark and Delta Lake versions

PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect

Re: Installing Databricks Connect breaks pyspark local cluster mode

Re: Installing Databricks Connect breaks pyspark local cluster mode

Re: PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect

Re: PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect