Wojciech_BUK
Valued Contributor III

I don;t belive your code was working before, at least the one you pasted above that has spacebars in table names as DLT throw me errors that it could not register it.
I made some correction to code and and it works ok.

I have added underscored "_"  to table names, changed decorators "@Dlt" to "@dlt" and changed "Temporary" to "temporary"

I had to drop your expectations and fake one column with static value.

Wojciech_BUK_0-1703689873573.png

 

 

import dlt

from pyspark.sql.functions import lit

@dlt.table(
name="Temp_Table",
table_properties={"quality" : "silver"},
temporary=True
)
def Temp_Table():
    return (
        spark.sql("SELECT * FROM priv_wojciech_bukowski.dss_gold.dim_dss_date")
        .withColumn("is_bad_data", lit('xxx'))
)


@dlt.table(
name="Clean_Table",
table_properties={"quality" : "silver"}
)
def get_clean_data():
    return (
        dlt.read("Temp_Table")
        .filter("is_bad_data='xxx'")
)