Temporary streaming tables (CDC)

mbaas
New Contributor III

‎07-24-2024 05:06 AM

I am currently using the `apply_changes` feature. I saw for the regular decorator `dlt.table` you can create temporary tables. I do not see the option you could use this feature with `dlt.create_streaming_table(`, in the sql version it looks it is supported: CREATE OR REFRESH TEMPORARY STREAMING TABLE.

Is there any other way I could create a table without the use of `dlt.create_streaming_table`?

raphaelblg
Databricks Employee
Databricks Employee

‎07-24-2024 02:51 PM - edited ‎07-24-2024 02:54 PM

Hi @mbaas, support for temporary tables by using the create_streaming_table() function is under development at the current moment. 

 

Is there any other way I could create a table without the use of `dlt.create_streaming_table`?

Yes, through the CREATE STREAMING TABLE SQL statement on DLT or DBSQL. In the future this might also be possible through serverless notebooks or serverless job compute.

 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks
0 Kudos

mbaas
New Contributor III
In response to raphaelblg

‎07-25-2024 06:13 AM

Hi @raphaelblg, good news you guys are working on it. I am using python file deployed by databricks bundles. I think it is not possible to mix SQL with python in dlt pipelines right?

raphaelblg
Databricks Employee
Databricks Employee
In response to mbaas

‎07-25-2024 07:41 AM - edited ‎07-25-2024 07:51 AM

@mbaas you're right, it's not possible to mix %sql with %python magic commands in DLT pipelines source code notebooks. 

Adding to the @Icassatti explanation on DLT temporary tables:

"Create a table, but do not publish metadata for the table. The temporary keyword instructs Delta Live Tables to create a table that is available to the pipeline but should not be accessed outside the pipeline. To reduce processing time, a temporary table persists for the lifetime of the pipeline that creates it, and not just a single update". Source: Delta Live Tables Python language reference 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

View solution in original post

0 Kudos

Icassatti
New Contributor III

‎07-24-2024 09:03 PM

Read this articles:

Delta Live Tables Python language reference - Azure Databricks | Microsoft Learn

The APPLY CHANGES APIs: Simplify change data capture with Delta Live Tables - Azure Databricks | Mic...

Even you could define as temporary, it would be storage into catalog _Databricks_Internal, so forget temporary in this case.

You must have an append only data source before you apply changes