Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2024 09:27 AM
I have an Azure Storage Data Table that I would like to update based on records that were just streamed into a Delta Live Table. Below is example code:
@Dlt.create_table(
comment="comment",
table_properties={
"pipelines.autoOptimize.managed": "true"
},
partition_cols = ["EventId"]
)
def live_table_test():
return (
spark.readStream.format("cloudFiles")
.options(**cloudfile)
.load(upload_path)
.selectExpr("*", "_metadata.file_path as file_path")
)I want to update an Azure Storage Data Table based on records that were just streamed into that table. Would something like this work?
@Dlt.create_table(
comment="comment",
table_properties={
"pipelines.autoOptimize.managed": "true"
},
partition_cols = ["EventId"]
)
def live_table_test():
df = (
spark.readStream.format("cloudFiles")
.options(**cloudfile)
.load(upload_path)
.selectExpr("*", "_metadata.file_path as file_path")
)
update_azure_table(df)
return dfI need to be able to get the list of distinct EventIds from the df and pass them into that function. How can I do this? And should I be using the "dlt.create_table" decorator? Or should I use the "dlt.table" decorator? I'm finding almost no documentation about this online.