Same error here, I am trying to save spark dataframe to Delta lake using JDBC driver and pyspark using this code:
#Spark session
spark_session = SparkSession.builder \
.appName("RCT-API") \
.config("spark.metrics.namespace", "rct-api") \
.config("spark.driver.extraClassPath", "/opt/workspace/databricks-jdbc-2.6.34-sources.jar") \
.config("spark.driver.extraLibrary", "/opt/workspace/databricks-jdbc-2.6.34-sources.jar") \
.config("spark.sql.execution.arrow.pyspark.enabled", "true") \
.config("spark.jars.packages", "com.databricks:databricks-jdbc:2.6.36") \
.getOrCreate()
#Loading spark dataframe from pandas dataframe
spark_df = spark_session.createDataFrame(pandas_df)
# Saving design table to DeltaLake
spark_df.write.format("jdbc").options(
url= connection_string,
driver= "com.databricks.client.jdbc.Driver",
dbtable= table_name).mode('overwrite').save()
I am facing the following error:
Caused by: com.databricks.client.support.exceptions.ErrorException: [Databricks][JDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: 42601, Query: CREATE TAB***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: [PARSE_SYNTAX_ERROR] org.apache.spark.sql.catalyst.parser.ParseException:
[PARSE_SYNTAX_ERROR] Syntax error at or near '"id"'. SQLSTATE: 42601 (line 1, pos 33)
Any idea? This configuration worked with general purpose clusters, but not SQL warehouses