Difference @dlt.table and @dlt.create_table decorator

[RaccoonRadio]

Hi!

I'm currently trying to stream data from an Azure Event Hub (kafka) using DLT. The provided example (https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/event-hubs) works well.

I saw in different examples the usage of two different decorators "@dlt.table" and "@dlt.create_table" but I could not find out which the correct one is. It works with both, also if the table does not exist when starting the DLT-pipeline.

Can anybody tell me which the correct one is and what the differences are?

Thanks!

[Kaniz_Fatma · Accepted solution]

Hi @RaccoonRadio , 

In Databricks Delta Live Tables (DLT), both @dlt.table and @dlt.create_table decorators are used, but they serve slightly different purposes. Here's the distinction:

1. @dlt.table:
   • This decorator is used to define a Delta Live Table (DLT).
   • It declares a table schema and instructs DLT to track changes to that table.
   • You use @dlt.table when you want to work with an existing table and track its changes as part of your DLT pipeline.

Example:

@dlt.table('my_existing_table')
def my_existing_table():
    return spark.readStream.format("delta").load("/path/to/delta_table")

2. @dlt.create_table:
   • This decorator is used to create a new Delta Live Table.
   • It also declares a table schema but additionally creates a new table if it doesn't already exist.
   • You use @dlt.create_table when you want to create a new table for your DLT pipeline.

Example:

@dlt.create_table('my_new_table')
def my_new_table():
    return spark.readStream.format("delta").load("/path/to/delta_table")
​

 

Both decorators can be used depending on your specific use case.

If you want to work with an existing table, @dlt.table is appropriate.

If you want to create a new table, @dlt.create_table is the right choice.

The choice between them depends on whether you deal with pre-existing data or generate new tables as part of your DLT pipeline.

[ac0]

@Kaniz_Fatma , Are you suggesting that the first time a pipeline runs, the below decorator should be used:

@dlt.create_table

 And the once the pipeline runs, the notebook should be edited to use this decorator:

@dlt.table