Simple append for a DLT

jrod123 · ‎02-25-2025

Looking for some help getting unstuck re: appending to DLTs in Databricks. I have successfully extracted data via API endpoint, done some initial data cleaning/processing, and subsequently stored that data in a DLT. Great start. But I noticed that each time the pipeline runs, all of the previous rows are overwritten. The AI assistant and separate google searches have proven worthless thus far to help me understand why I cannot simply append data from each run to the DLT. I manually added a timestamp column to ensure that each run's data is unique. And each time it runs, I can verify that the data is fresh. I just only see the new data (old is overwritten). According to my research, append is supposedly the default behavior when writing to a DLT, but that's not happening and I don't understand why. Attempts to explicitly define the append properties for the DLT (both in the notebook and pipeline settings) have not helped. Here is an simple example of what I'm trying (and failing) to do:

import dlt

from pyspark.sql.functions import current_timestamp

# Function to generate sample data

def generate_data():

data = [

(1, "A"),

(2, "B"),

(3, "C")

]

df = spark.createDataFrame(data, ["id", "value"])

df = df.withColumn("timestamp", current_timestamp())

return df

# Define the Delta Live Table

@dlt.table(

name="example_table",

comment="A simple example table",

table_properties={"pipelines.appendOnly": "true"}

)

def create_example_table():

return generate_data()