DatabricksUniForm

[Akash_Wadhankar]

Hi Community members, 

I tried creating a Delta UniForm table using databricks notebook. I created a database without providing location. It took the dbfs default storage location. On top of that I was able to create a Delta UniForm table. 

Then I tried creating database with AWS s3 location. Database was created. Then I tried putting a Delta UniForm table and I was getting error :

Error in SQL statement: AnalysisException: Unknown configuration was specified: delta.enableIcebergCompatV2

It seems the Delta UniForm table is created successfully on dbfs but failed to get created on AWS s3.

AWS s3 API for Delta Uniform not supported yet?

[Kaniz_Fatma]

Hi @Akash_Wadhankar, It appears you’ve encountered an issue while working with Delta tables in Databricks, specifically when trying to create a Delta table in an AWS S3 location.

1. Error Message: Unknown configuration was specified: delta.enableIcebergCompatV2

   • This error suggests that the configuration option delta.enableIcebergCompatV2 is not recognized.
   • The delta.enableIcebergCompatV2 option is related to compatibility with the Iceberg format, but it seems to be causing issues in your case.

2. Delta Tables on AWS S3:

   • When working with Delta tables on AWS S3, there are a few considerations:
     • Transactional Guarantees: AWS S3 does not provide atomic write operations like “put-if-absent,” which are essential for transactional guarantees in Delta tables. As a result, additional coordination mechanisms are required.
     • LogStore Implementation: Ensure that you’re using the correct implementation of LogStore for your storage system. Using the default HDFSLogStore on non-HDFS storage systems can lead to errors¹.
     • AWS Credentials: To access S3, you need to pass your AWS access key ID and secret access key. In recent versions of the delta-rs library, you can provide these credentials as part of the storage...².

3. Solution:

   • Make sure you’re using the correct version of the delta-rs library (v0.6.0 or newer).
   • Pass your AWS credentials (access key ID and secret access key) using the storage_options parameter when creating the DeltaTable.
   • Example code snippet:

     from deltalake import DeltaTable
     
     storage_options = {
         "AWS_ACCESS_KEY_ID": "your_access_key",
         "AWS_SECRET_ACCESS_KEY": "your_secret_key"
     }
     
     url = 's3://bucket/sales_data/'
     dt = DeltaTable(url, storage_options=storage_options)
     # Perform further operations on dt
     

4. Certificate Issue:

   • If you encounter certificate-related errors, consider using a certificate (if required by your S3 setup) or check the default certificate path.
   • Unfortunately, the error message doesn’t provide the exact path, but you might need to explore the deltalake documentation or seek community support for this specific issue.

Remember to adjust the placeholders (your_access_key, your_secret_key, and s3://bucket/sales_data/) with your actual AWS credentials and S3 path. If you continue to face issues, consider checking the deltalake package documentation or reaching out for further assistance. 😊🚀