Delta Lake Python

delta_lake — Fri, 17 Sep 2021 09:36:53 GMT

I have setup a virtual environment inside my existing hadoop cluster. Since the current cluster does not have spark >3 , so i installed delta spark using virtual environment. While trying to access the hdfs which is kerberose one, Getting below error. Can you one help to advise on this

Client auth mode SIMPLE, mechanism is not supported on zone, Auth:kerberos_only

from delta import *
    
builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
   .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
   .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.read.parquet("hdfs/parquet_path")

Re: Delta Lake Python

User16753724663 — Tue, 28 Sep 2021 17:24:25 GMT

Hi @Vasanth P ,

Could you please confirm the DBR version that you are using in the cluster?

There are multiple DBR versions that is available with different spark and scala version which can utilised directly in databricks cluster.

topic Delta Lake Python in Data Engineering

Delta Lake Python

Re: Delta Lake Python