Apply Changes Error in DLT Pipeline

VReddy1601
New Contributor

‎08-27-2024 12:30 AM

Hi Team,

 

Am Trying to use Apply changes from Bronze to Silver using the below. 

 

               
  @dlt.table(
           name="Silver_Orders",
           comment="This table - hive_metastore.silver.Orders reads data from the Bronze layer and writes it to the Silver layer.",
           table_properties={"quality": "silver"},
           path="/mnt/silver/Future-Orders/xxx/Orders"
        )
        def olo_orders_silver():
            try:
                # Read the Bronze table
                bronze_df = spark.table("hive_metastore.bronze.xxx_Orders")
               
                # Apply changes from the Bronze table to the Silver table
                dlt.apply_changes(
                    target="Silver_Orders",
                    source=bronze_df,
                    keys=["OrderId", "Label"],
                    sequence_by=F.col("processing_time")  # Assuming you have a timestamp column to order changes
                )
Here
1) bronze_df = spark.table("hive_metastore.bronze.Orders") 
That means it is the data frame with name bronze_df created on the Hive table. This will be source streaming table from Bronze layer. 
 
2) Silver_Orders is the Target Table which have mount location. 
 
While am running this Validation is notebook , am getting the below error. Kindly help to suggest what is causing the issue?
 
pyspark.errors.exceptions.captured.AnalysisException: APPLY CHANGE target 'Silver_Orders' should not have any query defined other than APPLY CHANGE INTO.
0 Kudos