cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Temporary streaming tables (CDC)

mbaas
New Contributor III

I am currently using the `apply_changes` feature. I saw for the regular decorator `dlt.table` you can create temporary tables. I do not see the option you could use this feature with `dlt.create_streaming_table(`, in the sql version it looks it is supported: CREATE OR REFRESH TEMPORARY STREAMING TABLE.

Is there any other way I could create a table without the use of `dlt.create_streaming_table`?

1 ACCEPTED SOLUTION

Accepted Solutions

raphaelblg
Databricks Employee
Databricks Employee

@mbaas you're right, it's not possible to mix %sql with %python magic commands in DLT pipelines source code notebooks. 

Adding to the @Icassatti explanation on DLT temporary tables:

"Create a table, but do not publish metadata for the table. The temporary keyword instructs Delta Live Tables to create a table that is available to the pipeline but should not be accessed outside the pipeline. To reduce processing time, a temporary table persists for the lifetime of the pipeline that creates it, and not just a single update". Source: Delta Live Tables Python language reference 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

View solution in original post

4 REPLIES 4

raphaelblg
Databricks Employee
Databricks Employee

Hi @mbaas, support for temporary tables by using the create_streaming_table() function is under development at the current moment. 

 

Is there any other way I could create a table without the use of `dlt.create_streaming_table`?

Yes, through the CREATE STREAMING TABLE SQL statement on DLT or DBSQL. In the future this might also be possible through serverless notebooks or serverless job compute.

 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

mbaas
New Contributor III

Hi @raphaelblg, good news you guys are working on it. I am using python file deployed by databricks bundles. I think it is not possible to mix SQL with python in dlt pipelines right?

raphaelblg
Databricks Employee
Databricks Employee

@mbaas you're right, it's not possible to mix %sql with %python magic commands in DLT pipelines source code notebooks. 

Adding to the @Icassatti explanation on DLT temporary tables:

"Create a table, but do not publish metadata for the table. The temporary keyword instructs Delta Live Tables to create a table that is available to the pipeline but should not be accessed outside the pipeline. To reduce processing time, a temporary table persists for the lifetime of the pipeline that creates it, and not just a single update". Source: Delta Live Tables Python language reference 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

Icassatti
New Contributor III

Read this articles:

Delta Live Tables Python language reference - Azure Databricks | Microsoft Learn

The APPLY CHANGES APIs: Simplify change data capture with Delta Live Tables - Azure Databricks | Mic...

Even you could define as temporary, it would be storage into catalog _Databricks_Internal, so forget temporary in this case.

You must have an append only data source before you apply changes

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group