topic Re: How to define write Option in a DLT using Python? in Data Engineering

How to define write Option in a DLT using Python?

pablociu — Wed, 24 May 2023 18:56:21 GMT

In a normal notebook I would save metadata to my Delta table using the following code:

(
  df.write
    .format("delta")
    .mode("overwrite")
    .option("userMetadata", user_meta_data) 
    .saveAsTable("my_table")
)

But I couldn't find online how can I set up the '.option("userMetadata...' in a DLT. Is there a document I couldn't find that would explain how to set up this?

Re: How to define write Option in a DLT using Python?

jose_gonzalez — Tue, 06 Jun 2023 18:51:18 GMT

I would like to share the following docs page https://docs.databricks.com/delta-live-tables/python-ref.html

Re: How to define write Option in a DLT using Python?

United_Communit — Wed, 07 Jun 2023 12:25:51 GMT

In Delta lab you can set up User MetaData so i will give you some tips

from delta import DeltaTable

# Create or load your Delta table

delta_table = DeltaTable.forPath(spark, "path_to_delta_table")

# Define your user metadata myccpay

user_meta_data = {"key1": "value1", "key2": "value2"}

# Set the userMetadata option in your Delta table write operation

delta_table.alias("df").merge(

df.alias("new_data"),

"df.id = new_data.id"

).whenMatchedUpdateAll()

.whenNotMatchedInsertAll()

.option("userMetadata", user_meta_data)

.execute()

you should try this code.