Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2024 03:44 PM
I'm trying to use Databricks Connect to run queries on Delta Tables locally. However, SQL queries using spark.sql don't seem to work properly, even though spark.read.table works.
>>> from databricks.connect import DatabricksSession
>>> spark = DatabricksSession.builder.profile("<profile name>").serverless().getOrCreate()
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1722031752.676173 13015515 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
>>> spark.read.table('<table name>')
DataFrame[...]
>>> spark.sql("SELECT * FROM <table name>")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/adawson/.pyenv/versions/pyspark/lib/python3.10/site-packages/pyspark/sql/connect/session.py", line 733, in sql
data, properties = self.client.execute_command(cmd.command(self._client))
File "/Users/adawson/.pyenv/versions/pyspark/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py", line 1270, in execute_command
data, _, _, _, properties = self._execute_and_fetch(
File "/Users/adawson/.pyenv/versions/pyspark/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py", line 1717, in _execute_and_fetch
for response in self._execute_and_fetch_as_iterator(
File "/Users/adawson/.pyenv/versions/pyspark/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py", line 1693, in _execute_and_fetch_as_iterator
self._handle_error(error)
File "/Users/adawson/.pyenv/versions/pyspark/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py", line 2009, in _handle_error
self._handle_rpc_error(error)
File "/Users/adawson/.pyenv/versions/pyspark/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py", line 2097, in _handle_rpc_error
raise convert_exception(
pyspark.errors.exceptions.connect.ParseException:
[PARSE_EMPTY_STATEMENT] Syntax error, unexpected empty statement. SQLSTATE: 42617 (line 1, pos 0)
== SQL ==
^^^
Has anyone else run into this issue? I'm on an M2 Pro Mac, running Python 3.10, pyspark 3.5.1, databricks-connect 15.3.1, databricks-sdk 0.29.0, Databricks CLI v0.224.1.