cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Additional permission for Delta compared to Parquet

brickster_2018
Databricks Employee
Databricks Employee

I am trying to create a Delta table and it seems the Delta table requires additional permissions on the parent folder of the table. The command failed telling permission errors. I tried to create a parquet table and it works fine. 

1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Databricks Employee
Databricks Employee

Delta table is the non-hive compatible format. So there must also be permissions for a client to access the path to the databaseโ€™s location so that it can create a new temporary โ€œdirectoryโ€ there. This comes from Spark SQLโ€™s handling of external tables. Delta tables just happen to always go through this path. 

See Apache Spark code https://github.com/apache/spark/blob/branch-2.4/sql/hive/src/main/scala/org/apache/spark/sql/hive/Hi... for more information

View solution in original post

1 REPLY 1

brickster_2018
Databricks Employee
Databricks Employee

Delta table is the non-hive compatible format. So there must also be permissions for a client to access the path to the databaseโ€™s location so that it can create a new temporary โ€œdirectoryโ€ there. This comes from Spark SQLโ€™s handling of external tables. Delta tables just happen to always go through this path. 

See Apache Spark code https://github.com/apache/spark/blob/branch-2.4/sql/hive/src/main/scala/org/apache/spark/sql/hive/Hi... for more information

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group