cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta table property is not set.

Data_Analytics1
Contributor III

I have set the delta table property at cluster level.

spark.databricks.delta.retentionDurationCheck.enabled false

When I create a new table, retentionDurationCheck property is not shown in the table details. But when I set this with ALTER TABLE for a specific delta table, I can see the property in the table details.

4 REPLIES 4

Anonymous
Not applicable

@Mahesh Chahareโ€‹ :

The spark.databricks.delta.retentionDurationCheck.enabled configuration property is used to enable or disable retention duration checks for Delta tables created in a Databricks workspace. When this property is set to true (which is the default), Delta will periodically check and delete any files that exceed the retention period set on the table. If it's set to false, then the retention duration check is disabled.

However, setting this property at the cluster level only affects new Delta tables that are created after the property is set. For existing Delta tables, you will need to set this property on each table using the

ALTER TABLE command.

To set the property on a Delta table, you can use the following command:

ALTER TABLE <table-name> SET TBLPROPERTIES ('delta.retentionDurationCheck.enabled'='false');

This will set the delta.retentionDurationCheck.enabled property to false for the specified Delta table. Once this property is set on the table, you should be able to see it in the table details.

Note that the SET TBLPROPERTIES command can also be used to set other table-level properties for Delta tables, such as retention period and vacuum options.

@Suteja Kanuriโ€‹ 

I have created a new table after configuring the property spark.databricks.delta.retentionDurationCheck.enabled as false, but I can't see that property as false in table details.

@Suteja Kanuriโ€‹ 

When I am using ALTER TABLE catalog.schema.tbl1 SET TBLPROPERTIES ('delta.retentionDurationCheck.enabled'='false'); it is throwing the below exception.

AnalysisException: Unknown configuration was specified: delta.retentionDurationCheck.enabled

It is working when I used below query,

ALTER TABLE catalog.schema.tbl1 SET TBLPROPERTIES ('retentionDurationCheck.enabled'='false'); but I think it will not serve the purpose.

Can you please check, what exactly the property should be at cluster level and at table level?

Anonymous
Not applicable

Hi @Mahesh Chahareโ€‹ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group