Hi Community,
I'm trying to read the data from sample schema from table nation from data-bricks catalog via spark but i'm getting this error.
com.databricks.client.support.exceptions.GeneralException: [Databricks][JDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: null, Query: SELECT * FROM (SELECT n_nationkey ,n_name ,n_regionkey ,n_comment FROM samples.tpch.nation order by "n_nationkey" limit 50000 offset 0) SPARK_GEN_SUBQ_0 WHERE 1=0, Error message from Server: Configuration schema is not available
Here is my sample code.
from pyspark.sql import SparkSession
import os
os.environ['SPARK_HOME'] = '/opt/spark'
os.environ['PATH'] = os.environ['SPARK_HOME'] + '/bin:' + os.environ['PATH']
spark_dict = {
"hostname": "my_hostname",
"user": "my_user",
"password": "my_pass",
"database": "samples",
"schema": "tpch",
"driver": "com.databricks.client.jdbc.Driver",
"url": "jdbc:databricks://host:443/samples;transportMode=http;ssl=1;AuthMech=3;httpPath=httpPath",
"port": "443",
"query": "SELECT n_nationkey ,n_name ,n_regionkey ,n_comment FROM samples.tpch.nation order by \"n_nationkey\" limit 50000 offset 0"
}
spark = (
SparkSession.builder.appName("abc")
.config("spark.jars", "/home/jars/databricks-jdbc-2.6.34.jar")
.getOrCreate()
)
df = spark.read.format("jdbc").options(**spark_dict).load()
df.show()
Can someone please guide me here how to fix the above error.
