Hi Experts,
I have on Bronze layer all delta merge files (Parquet) format.
I am converting these files into delta live tables in silver layer. While doing so, I am unable to add current time stamp column.
Following is the script:
from pyspark.sql.functions import *
from pyspark.sql.types import *
import dlt
currentdate= spark.sql("select string(current_timestamp)").collect()[0][0]
@Dlt.table
def t_stream_silver_00_XXXX():
return (
spark.readStream
.format("delta")
.option("cloudFiles.format", "parquet")
.option(key = "SILVER_LOADING_TIME", value =currentdate)
.load("abfss://bronze-prod-XXXXX@adlsg2SAPPPPpsi001.dfs.core.windows.net/PROD/XXXX/Delta/00_XXXX/")
)
it didt not throw any error at the same time no column being added. Can you please help?
