Adding new field to Delta live Table

Madalian — Wed, 13 Mar 2024 11:56:13 GMT

Hi Experts,

I have on Bronze layer all delta merge files (Parquet) format.

I am converting these files into delta live tables in silver layer. While doing so, I am unable to add current time stamp column.

Following is the script:

from pyspark.sql.functions import *
from pyspark.sql.types import *
import dlt
currentdate= spark.sql("select string(current_timestamp)").collect()[0][0]
@Dlt.table
def t_stream_silver_00_XXXX():
return (
spark.readStream
.format("delta")
.option("cloudFiles.format", "parquet")
.option(key = "SILVER_LOADING_TIME", value =currentdate)
.load("abfss://bronze-prod-XXXXX@adlsg2SAPPPPpsi001.dfs.core.windows.net/PROD/XXXX/Delta/00_XXXX/")
)

it didt not throw any error at the same time no column being added. Can you please help?

topic Adding new field to Delta live Table in Get Started Discussions

Adding new field to Delta live Table