cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Is it possible to create/update non dlt table in init phase of dlt task?

PassionateDBD
New Contributor II

We have a dlt task that is written in python. Is it possible to create or update a delta table programatically from inside a dlt task? The delta table would not be managed from inside the dlt task because we never want to fully refresh that table. The table is more of a "logging table" where we only append configurations when they change over time. It would be fine if we are able to update the table once every time that we start the dlt task for example.

1 REPLY 1

PassionateDBD
New Contributor II

Thanks for you reply @Retired_mod ! I'm aware of the possibility to create or not create a table based on some parameter.

What I'm trying to figure out is basically how to achieve following:

-DLT pipeline starts and logs some information to a delta table.

-On each DLT pipeline restart a new row should be appended to the same delta table.

-If I run a full refresh of the DLT pipeline then the delta table where the data has been saved on each startup should remain untouched leaving "the log" of things that were saved during DLT pipeline startups untouched.

The delta table may be a DLT table or a non DLT delta table but the functionality should be the one that I described.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group