Declarative Pipelines: set Merge Schema to False

[a_user12]

Dear Team!

I want to prevent at a certain table that the schema is automatically updated. With plain strucutred streaming I can do the following:

silver_df.writeStream \
    .format("delta") \
    .option("mergeSchema", "false") \
    .option("checkpointLocation", checkpoint_path) \
    .outputMode("append") \
    .table("silver_table")

How can I set mergeSchema=false with Declarative Pipelines?

[szymon_dybczak]

Hi @a_user12 ,

Did you try something like in a link below? Of course in your case you want to set it to "false":

H Learn Data Engineering: Databricks Delta Live Table | by THE BRICK LEARNING | Medium

[a_user12]

@szymon_dybczak  - thank you for your response 

I try:

@dlt.table(
    name="deserialized",
    comment="Raw messages from Kafka topic as JSON",
    table_properties={
        "pipelines.autoOptimize.managed": "true",
        "pipelines.autoCompact.managed": "true"
    }
)
def deserialize():
    # Read from Kafka
    return spark.readStream \
            .table("stringified") \
            .withColumn("payload", from_json(col("payload"),None,{"schemaLocationKey": "x"})) \
            .select("topic","timestamp","payload") \
            .withColumn("new-x",lit("foo"))
    



@dlt.table(
    name="enriched_table",
    table_properties={
        "pipelines.autoOptimize.managed": "true",
        "pipelines.autoCompact.managed": "true"
    }
)
def enriched_table():
    return spark.readStream.option("mergeSchema","false").table("deserialized")
        #.withColumn("new",lit("new"))  # ensure columns match exactly    

I would expect, that if the attribute "nex-x" is not existing in the table "enriched table" yet I get an error. Indeed, it is simply adding the new column in the "neriched table".