Apply Changes Error in DLT Pipeline

VReddy1601 — Tue, 27 Aug 2024 07:30:39 GMT

Hi Team,

Am Trying to use Apply changes from Bronze to Silver using the below.

@dlt.table(

name="Silver_Orders",

comment="This table - hive_metastore.silver.Orders reads data from the Bronze layer and writes it to the Silver layer.",

table_properties={"quality": "silver"},

path="/mnt/silver/Future-Orders/xxx/Orders"

)

def olo_orders_silver():

try:

# Read the Bronze table

bronze_df = spark.table("hive_metastore.bronze.xxx_Orders")

# Apply changes from the Bronze table to the Silver table

dlt.apply_changes(

target="Silver_Orders",

source=bronze_df,

keys=["OrderId", "Label"],

sequence_by=F.col("processing_time") # Assuming you have a timestamp column to order changes

)

Here

1) bronze_df = spark.table("hive_metastore.bronze.Orders")

That means it is the data frame with name bronze_df created on the Hive table. This will be source streaming table from Bronze layer.

2) Silver_Orders is the Target Table which have mount location.

While am running this Validation is notebook , am getting the below error. Kindly help to suggest what is causing the issue?

pyspark.errors.exceptions.captured.AnalysisException: APPLY CHANGE target 'Silver_Orders' should not have any query defined other than APPLY CHANGE INTO.

topic Apply Changes Error in DLT Pipeline in Get Started Discussions

Apply Changes Error in DLT Pipeline