https://community.databricks.com/t5/data-engineering/how-do-we-manage-data-recency-in-databricks/m-p/22061#M15079 <P>When using delta tables in databricks, you have the advantage of delta cache which accelerates data reads by creating copies of remote files in nodes’ local storage using a fast intermediate data format.&nbsp;At the beginning of each query delta tables auto-update to the latest version - this way data is always recent. </P><P></P><P>However, if &nbsp;it is acceptable for results to be stale for a short duration of time, you could lower the latency of queries further. This is done by setting the Spark session configuration variable spark.databricks.delta.stalenessLimit with a time string value, e.g 1h, 15m, 1d&nbsp;</P><P></P> Wed, 23 Jun 2021 00:43:42 GMT sajith_appukutt 2021-06-23T00:43:42Z https://community.databricks.com/t5/data-engineering/how-do-we-manage-data-recency-in-databricks/m-p/22060#M15078 <P>I want to know how databricks maintain data recency in databricks </P> Mon, 21 Jun 2021 12:57:04 GMT https://community.databricks.com/t5/data-engineering/how-do-we-manage-data-recency-in-databricks/m-p/22060#M15078 User16826994223 2021-06-21T12:57:04Z https://community.databricks.com/t5/data-engineering/how-do-we-manage-data-recency-in-databricks/m-p/22061#M15079 <P>When using delta tables in databricks, you have the advantage of delta cache which accelerates data reads by creating copies of remote files in nodes’ local storage using a fast intermediate data format.&nbsp;At the beginning of each query delta tables auto-update to the latest version - this way data is always recent. </P><P></P><P>However, if &nbsp;it is acceptable for results to be stale for a short duration of time, you could lower the latency of queries further. This is done by setting the Spark session configuration variable spark.databricks.delta.stalenessLimit with a time string value, e.g 1h, 15m, 1d&nbsp;</P><P></P> Wed, 23 Jun 2021 00:43:42 GMT https://community.databricks.com/t5/data-engineering/how-do-we-manage-data-recency-in-databricks/m-p/22061#M15079 sajith_appukutt 2021-06-23T00:43:42Z