Hello,
here is a small code-snippet:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('example_app').getOrCreate()
spark.sql('SHOW PARTITIONS database.table').show()
The output inside the Databricks-Notebook:
+-------------+-------+--------------------+
|projectNumber|plantId| name|
+-------------+-------+--------------------+
| xxxx| P0|***.yyyy............|
| yyyy| P2|***.yyyy............|
...
When I run the same code as above in Visual Studio Code, connected to the same cluster through Databricks-Connect, I receive this output:
+---------+
|partition|
+---------+
| xxxx|
| yyyy|
...
This output has the wrong column name and shows only the first partition.
This is strange. Everything is identical so the output should be the same.
I receive the correct partitions through sql-describe in both databricks-connect and databricks:
spark.sql('describe table database.table').show()
+--------------+-------------+-------+
| col_name| data_type|comment|
+--------------+-------------+-------+
|# Partitioning| | |
| Part 0|projectNumber| |
| Part 1| plantId| |
| Part 2| name| |
+--------------+-------------+-------+
The table is a delta-table, located in an azure blob-storage.
I tried to refresh the table but this makes no difference.
I found a difference in the Spark-UI SQL tab.
There are 3 queries for the db-connect run and 4 for the databricks run.
The physical execution plan is identical but the second query "Execute ShowPartitionsDeltaCommand" is missing in the db-connect run.
Queries for db-connect:
- Execute ShowPartitionsDeltaCommand | Output: [projectNumber, plantId, name]
- bigger execution plan (identical in both cases) | Output: [projectNumber, plantId, name]
- LocalTableScan | Output: [partition]
Queries for databricks:
- Execute ShowPartitionsDeltaCommand | Output: [projectNumber, plantId, name]
- bigger execution plan (identical in both cases) | Output: [projectNumber, plantId, name]
- Execute ShowPartitionsDeltaCommand | Output: [projectNumber, plantId, name]
- LocalTableScan | Output: [projectNumber, plantId, name]
I don´t know why and how but the 2 partitions get lost with the db-connect query.
Any ideas?
