cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta table property is not set.

Data_Analytics1
Contributor III

I have set the delta table property at cluster level.

spark.databricks.delta.retentionDurationCheck.enabled false

When I create a new table, retentionDurationCheck property is not shown in the table details. But when I set this with ALTER TABLE for a specific delta table, I can see the property in the table details.

4 REPLIES 4

Anonymous
Not applicable

@Mahesh Chahare​ :

The spark.databricks.delta.retentionDurationCheck.enabled configuration property is used to enable or disable retention duration checks for Delta tables created in a Databricks workspace. When this property is set to true (which is the default), Delta will periodically check and delete any files that exceed the retention period set on the table. If it's set to false, then the retention duration check is disabled.

However, setting this property at the cluster level only affects new Delta tables that are created after the property is set. For existing Delta tables, you will need to set this property on each table using the

ALTER TABLE command.

To set the property on a Delta table, you can use the following command:

ALTER TABLE <table-name> SET TBLPROPERTIES ('delta.retentionDurationCheck.enabled'='false');

This will set the delta.retentionDurationCheck.enabled property to false for the specified Delta table. Once this property is set on the table, you should be able to see it in the table details.

Note that the SET TBLPROPERTIES command can also be used to set other table-level properties for Delta tables, such as retention period and vacuum options.

@Suteja Kanuri​ 

I have created a new table after configuring the property spark.databricks.delta.retentionDurationCheck.enabled as false, but I can't see that property as false in table details.

@Suteja Kanuri​ 

When I am using ALTER TABLE catalog.schema.tbl1 SET TBLPROPERTIES ('delta.retentionDurationCheck.enabled'='false'); it is throwing the below exception.

AnalysisException: Unknown configuration was specified: delta.retentionDurationCheck.enabled

It is working when I used below query,

ALTER TABLE catalog.schema.tbl1 SET TBLPROPERTIES ('retentionDurationCheck.enabled'='false'); but I think it will not serve the purpose.

Can you please check, what exactly the property should be at cluster level and at table level?

Anonymous
Not applicable

Hi @Mahesh Chahare​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.