Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-20-2024 05:39 AM
Same here:
pyspark="3.5.3"
delta-spark="3.2.1"
spark.sql("SHOW CATALOGS").show()
spark.sql("SHOW TABLES IN mycatalog.myschema").show()
spark.sql("DESCRIBE mycatalog.myschema.mytable").show()Code above is working with the three-level notation. (catalog.database.table)
When using the three-level notation with following code, the same error occurs:
from delta import DeltaTable
dt = DeltaTable.forName(spark, "mycatalog.myschema.mytable")Traceback (most recent call last):
File "/workspace/test.py", line 13, in <module>
dt = DeltaTable.forName(spark, "mycatalog.myschema.mytable")
File "/venv/lib/python3.10/site-packages/delta/tables.py", line 419, in forName
jdt = jvm.io.delta.tables.DeltaTable.forName(jsparkSession, tableOrViewName)
File "/venv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in __call__
return_value = get_return_value(
File "/venv/lib/python3.10/site-packages/pyspark/errors/exceptions/captured.py", line 185, in deco
raise converted from None
pyspark.errors.exceptions.captured.ParseException:
[PARSE_SYNTAX_ERROR] Syntax error at or near '.'.(line 1, pos 11)
== SQL ==
mycatalog.myschema.mytable
------------------^^^