Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-18-2023 06:34 AM
Hi,
I would like to know if it is possible to get the target schema, programmatically, inside a DLT.
In DLT pipeline settings, destination, target schema.
I want to run more idempotent pipelines. For example, my target table has the fields: reference_date, customer_id, val1, val2, etc
I would like to, BEFORE writing to it, get the max reference_date currently in the target table, so I can query appropriately the source tables.
Currently DLT does not allow to create another live table reading to it before.
I could use 'spark.table', but to make it fully automatic, I would like to have the target schema.
target_table_name = 'whatevertable'
@dlt.table(
name = target_table_name
)
def update_target():
max_date = (
spark
.table(get_dlt_target_schema() + '.' + target_table_name)
.select(F.max('reference_date'))
)
....
....
....Thank you
Labels:
- Labels:
-
DLT