Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-27-2023 07:12 AM
I don;t belive your code was working before, at least the one you pasted above that has spacebars in table names as DLT throw me errors that it could not register it.
I made some correction to code and and it works ok.
I have added underscored "_" to table names, changed decorators "@Dlt" to "@dlt" and changed "Temporary" to "temporary"
I had to drop your expectations and fake one column with static value.
import dlt
from pyspark.sql.functions import lit
@dlt.table(
name="Temp_Table",
table_properties={"quality" : "silver"},
temporary=True
)
def Temp_Table():
return (
spark.sql("SELECT * FROM priv_wojciech_bukowski.dss_gold.dim_dss_date")
.withColumn("is_bad_data", lit('xxx'))
)
@dlt.table(
name="Clean_Table",
table_properties={"quality" : "silver"}
)
def get_clean_data():
return (
dlt.read("Temp_Table")
.filter("is_bad_data='xxx'")
)