Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)

- - Certifications
- - Learning Paths
- - Databricks Product Tours
- - Get Started Guides

- - Get Started Resources
- - Announcements
- - Community Articles
- - Databricks TV
- - Learning Events
- - MVP Articles
- - Product Platform Updates
- - Support FAQs
- - Technical Blog
- - Community Events
- - BrickTalks TV

- - Databricks Academy Learners
  - Databricks Academy Learners Forum
- - Regional and Interest Groups
- - Private Groups

- - Databricks Community Champions
- - Khoros Community Forums Support (Not for Databricks Product Questions)
- - Databricks Community Code of Conduct

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi,

It is currently not possible to specify a list of tables to refresh and their refresh policies (full/normal) in a Lakeflow Job.

It can be done via the REST API, but it's messy.

For example, if you need some tables or views refreshed more regularly, the only option at the moment is to bake this into a notebook using the REST API and schedule/trigger the notebook.

It would be useful if you could specify a list of tables in a pipeline job in the ui or DAB yaml and their refresh policy.

Regards

Toby