Delta Lake Python

[delta_lake]

I have setup a virtual environment inside my existing hadoop cluster. Since the current cluster does not have spark >3 , so i installed delta spark using virtual environment. While trying to access the hdfs which is kerberose one, Getting below error. Can you one help to advise on this

Client auth mode SIMPLE, mechanism is not supported on zone, Auth:kerberos_only

from delta import *
    
builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
   .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
   .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.read.parquet("hdfs/parquet_path")

[User16753724663]

Hi @Vasanth P​ ,

Could you please confirm the DBR version that you are using in the cluster?

There are multiple DBR versions that is available with different spark and scala version which can utilised directly in databricks cluster.