cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Difference @dlt.table and @dlt.create_table decorator

RaccoonRadio
New Contributor

Hi!

I'm currently trying to stream data from an Azure Event Hub (kafka) using DLT. The provided example (https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/event-hubs) works well.

I saw in different examples the usage of two different decorators "@dlt.table" and "@dlt.create_table" but I could not find out which the correct one is. It works with both, also if the table does not exist when starting the DLT-pipeline.

Can anybody tell me which the correct one is and what the differences are?

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz_Fatma
Community Manager
Community Manager

Hi @RaccoonRadio , 

In Databricks Delta Live Tables (DLT), both @dlt.table and @dlt.create_table decorators are used, but they serve slightly different purposes. Here's the distinction:

  1. @dlt.table:
    • This decorator is used to define a Delta Live Table (DLT).
    • It declares a table schema and instructs DLT to track changes to that table.
    • You use @dlt.table when you want to work with an existing table and track its changes as part of your DLT pipeline.

Example:

@dlt.table('my_existing_table')
def my_existing_table():
    return spark.readStream.format("delta").load("/path/to/delta_table")
  1. @dlt.create_table:
    • This decorator is used to create a new Delta Live Table.
    • It also declares a table schema but additionally creates a new table if it doesn't already exist.
    • You use @dlt.create_table when you want to create a new table for your DLT pipeline.

Example:

@dlt.create_table('my_new_table')
def my_new_table():
    return spark.readStream.format("delta").load("/path/to/delta_table")
​
 
Both decorators can be used depending on your specific use case.
 
If you want to work with an existing table, @dlt.table is appropriate.
 
If you want to create a new table, @dlt.create_table is the right choice.
 
The choice between them depends on whether you deal with pre-existing data or generate new tables as part of your DLT pipeline.

View solution in original post

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @RaccoonRadio , 

In Databricks Delta Live Tables (DLT), both @dlt.table and @dlt.create_table decorators are used, but they serve slightly different purposes. Here's the distinction:

  1. @dlt.table:
    • This decorator is used to define a Delta Live Table (DLT).
    • It declares a table schema and instructs DLT to track changes to that table.
    • You use @dlt.table when you want to work with an existing table and track its changes as part of your DLT pipeline.

Example:

@dlt.table('my_existing_table')
def my_existing_table():
    return spark.readStream.format("delta").load("/path/to/delta_table")
  1. @dlt.create_table:
    • This decorator is used to create a new Delta Live Table.
    • It also declares a table schema but additionally creates a new table if it doesn't already exist.
    • You use @dlt.create_table when you want to create a new table for your DLT pipeline.

Example:

@dlt.create_table('my_new_table')
def my_new_table():
    return spark.readStream.format("delta").load("/path/to/delta_table")
​
 
Both decorators can be used depending on your specific use case.
 
If you want to work with an existing table, @dlt.table is appropriate.
 
If you want to create a new table, @dlt.create_table is the right choice.
 
The choice between them depends on whether you deal with pre-existing data or generate new tables as part of your DLT pipeline.

ac0
New Contributor III

@Kaniz_Fatma , Are you suggesting that the first time a pipeline runs, the below decorator should be used:

 

@dlt.create_table

 

 And the once the pipeline runs, the notebook should be edited to use this decorator:

 

@dlt.table

 

 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!