Delta Lake Python
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā09-17-2021 02:36 AM
I have setup a virtual environment inside my existing hadoop cluster. Since the current cluster does not have spark >3 , so i installed delta spark using virtual environment. While trying to access the hdfs which is kerberose one, Getting below error. Can you one help to advise on this
Client auth mode SIMPLE, mechanism is not supported on zone, Auth:kerberos_only
from delta import *
builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.read.parquet("hdfs/parquet_path")
- Labels:
-
Current Cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā09-28-2021 10:24 AM
Hi @Vasanth Pā ,
Could you please confirm the DBR version that you are using in the cluster?
There are multiple DBR versions that is available with different spark and scala version which can utilised directly in databricks cluster.