cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Identity column in DLT using Python

kskistad
New Contributor III

How would I implement the Identity column in Delta Live Tables using Python syntax?

GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( [ START WITH start ] [ INCREMENT BY step ] ) ]

 

1 ACCEPTED SOLUTION

Accepted Solutions

LaurentLeturgez
New Contributor III
New Contributor III

Hi @Kory Skistad​ 

Please find below the table schema definition to use in a python dlt pipeline. You can see it mentions the identity column definition.

@dlt.table(

comment="Raw data on sales",

schema="""

customer_id STRING,

customer_name STRING,

number_of_line_items STRING,

order_datetime STRING,

order_number LONG,

order_day_of_week STRING GENERATED ALWAYS AS (dayofweek(order_datetime))

""",

partition_cols = ["order_day_of_week"])

def sales():

return ("...")

This example has been taken from this page: https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-python-ref.html

By the way, be careful about how it is generated regarding you are dealing with a streaming table or not: https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-faqs-issues.html#how-do-i-...

HTH

Laurent

View solution in original post

1 REPLY 1

LaurentLeturgez
New Contributor III
New Contributor III

Hi @Kory Skistad​ 

Please find below the table schema definition to use in a python dlt pipeline. You can see it mentions the identity column definition.

@dlt.table(

comment="Raw data on sales",

schema="""

customer_id STRING,

customer_name STRING,

number_of_line_items STRING,

order_datetime STRING,

order_number LONG,

order_day_of_week STRING GENERATED ALWAYS AS (dayofweek(order_datetime))

""",

partition_cols = ["order_day_of_week"])

def sales():

return ("...")

This example has been taken from this page: https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-python-ref.html

By the way, be careful about how it is generated regarding you are dealing with a streaming table or not: https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-faqs-issues.html#how-do-i-...

HTH

Laurent

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!