Temporary streaming tables (CDC)

[mbaas]

I am currently using the `apply_changes` feature. I saw for the regular decorator `dlt.table` you can create temporary tables. I do not see the option you could use this feature with `dlt.create_streaming_table(`, in the sql version it looks it is supported: CREATE OR REFRESH TEMPORARY STREAMING TABLE.

Is there any other way I could create a table without the use of `dlt.create_streaming_table`?

[raphaelblg]

Hi @mbaas, support for temporary tables by using the create_streaming_table() function is under development at the current moment. 

Is there any other way I could create a table without the use of `dlt.create_streaming_table`?

Yes, through the CREATE STREAMING TABLE SQL statement on DLT or DBSQL. In the future this might also be possible through serverless notebooks or serverless job compute.

[mbaas]

Hi @raphaelblg, good news you guys are working on it. I am using python file deployed by databricks bundles. I think it is not possible to mix SQL with python in dlt pipelines right?

[raphaelblg · Accepted solution]

@mbaas you're right, it's not possible to mix %sql with %python magic commands in DLT pipelines source code notebooks. 

Adding to the @Icassatti explanation on DLT temporary tables:

"Create a table, but do not publish metadata for the table. The temporary keyword instructs Delta Live Tables to create a table that is available to the pipeline but should not be accessed outside the pipeline. To reduce processing time, a temporary table persists for the lifetime of the pipeline that creates it, and not just a single update". Source: Delta Live Tables Python language reference 

[Icassatti]

Read this articles:

Delta Live Tables Python language reference - Azure Databricks | Microsoft Learn

The APPLY CHANGES APIs: Simplify change data capture with Delta Live Tables - Azure Databricks | Mic...

Even you could define as temporary, it would be storage into catalog _Databricks_Internal, so forget temporary in this case.

You must have an append only data source before you apply changes