topic Re: How to refresh a single table in Delta Live Tables? in Data Engineering

How to refresh a single table in Delta Live Tables?

tinai_long — Mon, 09 May 2022 09:14:59 GMT

Suppose I have a Delta Live Tables framework with 2 tables:

Table 1 ingests from a json source,
Table 2 reads from Table 1 and runs some transformation.

In other words, the data flow is json source -> Table 1 -> Table 2.

Now if I find some bugs in the transformation Table 1 -> Table 2, how can I re-run only the transformation Table 1 -> Table 2 and leave Table 1 intact?

If I use Full Refresh, it would refresh Table 1 & rerun the json ingestion as well...

Re: How to refresh a single table in Delta Live Tables?

Dooley — Wed, 11 May 2022 22:14:42 GMT

@Long Tran , The best way to achieve this would be a work around. If you could sacrifice a row of code in your Table 1 or add a row of Nulls into Table 1 without causing problems for yourself further down your pipeline, I suggest you try this Retain manual deletes or updates - "You can manually delete or update the record from raw_user_table and do a refresh operation to recompute the downstream tables."

However, I want to note that if you are ingesting from a source that has no new data to ingest, the full refresh probably wont re-ingest the same data causing duplication. Try it out on a subsection of your data.

Re: How to refresh a single table in Delta Live Tables?

Anonymous — Mon, 20 Jun 2022 17:27:53 GMT

Hey @Long Tran

Does @Sara Dooley response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help.

We'd love to hear from you.

Cheers!

Re: How to refresh a single table in Delta Live Tables?

tinai_long — Mon, 20 Jun 2022 18:42:16 GMT

Hi, sorry I am new here - how do I mark the answer as resolved? Thanks a lot.

Re: How to refresh a single table in Delta Live Tables?

tinai_long — Mon, 20 Jun 2022 18:42:57 GMT

sorry for the delayed response, User16460565755155528764's answer is very helpful.

Re: How to refresh a single table in Delta Live Tables?

tinai_long — Mon, 20 Jun 2022 18:43:02 GMT

Thank you so much!

Re: How to refresh a single table in Delta Live Tables?

Anonymous — Tue, 21 Jun 2022 15:45:14 GMT

Hi @Long Tran

Thank you so much for getting back to us. It's really great of you to mark the answer as best.

We really appreciate your time.

Wish you a great Databricks journey ahead!

Re: How to refresh a single table in Delta Live Tables?

Felipe — Thu, 30 Jun 2022 04:48:28 GMT

An update to anyone finding this thread nowadays.

This is possible using the reset.allowed property as documented here: https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-cookbook.html#retain-manual-deletes-or-updates

Re: How to refresh a single table in Delta Live Tables?

cpayne_vax — Thu, 08 Feb 2024 22:14:25 GMT

I want to tag onto this thread because I have the same need to refresh only a single table within a larger DLT pipeline. Unfortunately it seems the links in the accepted answer and in Felipe's follow up no longer contain the correct information. Is there a proper way to do this now, in 2024?

Re: How to refresh a single table in Delta Live Tables?

cpayne_vax — Fri, 09 Feb 2024 19:05:35 GMT

Answering my own question: nowadays (February 2024) this can all be done via the UI.

When viewing your DLT pipeline there is a "Select tables for refresh" button in the header. If you click this, you can select individual tables, and then in the bottom right corner there are options to "Full refresh selection" or "Refresh selection." Select "Full" in order to start your table over clean.

Re: How to refresh a single table in Delta Live Tables?

eballinger — Fri, 20 Dec 2024 13:40:50 GMT

Just wondering if there is a way to do this through code / not using the UI. Its now Dec 2024. We would like to handle this in testing using our workflows

Thanks

Re: How to refresh a single table in Delta Live Tables?

RiyazAliM — Mon, 23 Dec 2024 09:27:04 GMT

Hi @eballinger - Refreshing a single or set of tables in a DLT is possible via code using Databricks SDK.

Please follow the below doc for the same - https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/pipelines.html

In the above doc, search for `refresh_selection`, it will accept a list of tables that you want to refresh in a pipeline.

Snippet of the same:

Re: How to refresh a single table in Delta Live Tables?

eballinger — Tue, 24 Dec 2024 13:54:59 GMT

Thanks Riz,

I have not worked with the API yet but just tried your suggestion and it worked great. I will share my code for someone else looking for this:

myList = ["table1","table6"]

#Now run the pipeline

update_dlt2 = w.pipelines.start_update(refresh_selection=myList, pipeline_id="my_token_string")

print("Started pipeline ...")