delta lake in Apache Spark
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2023 02:43 PM
Hi,
As per documentation https://docs.delta.io/latest/quick-start.html , we can configure DeltaCatalog using spark.sql.catalog.spark_catalog.
The Iceberg supports two Catalog implementations (https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs):
- Replacing the session catalog (spark_catalog) - using org.apache.iceberg.spark.SparkSessionCatalog , It adds support for Iceberg tables to Spark’s built-in catalog, and delegates to the built-in catalog for non-Iceberg tables
- Custom Catalog - org.apache.iceberg.spark.SparkCatalog - supports a Hive Metastore or a Hadoop warehouse as
Do we have an option similar to Iceberg in Delta Lake; where we can configure a custom catalog?
Labels: