.py script execution failed but succeeded when run in Python notebook

[HuyNguyen]

Background:

My code executing without problem if run in a python notebook. However, the same code fails when execute from a .py script in the workspace. Seems like the 2 execution methods don't have identical version of the packages

Error message: AttributeError: 'DeltaMergeBuilder' object has no attribute 'withSchemaEvolution' 

Code:

    from delta.tables import DeltaTable

    # Perform the merge with schema evolution
    merge_result = delta_table.alias("target") \
        .merge(
            df.alias("source"),
            f"target.{id_column_name} = source.{id_column_name}"
        ) \
        .withSchemaEvolution() \
        .whenMatchedUpdateAll() \
        .whenNotMatchedInsertAll() \
        .execute()

For more context, this .withSchemaEvolution() setting was availabled from runtime 16.0 onwards: Databricks Runtime 16.0 | Databricks Documentation My serverless run time is 16.1, as confirmed by running: 

spark.sql("select current_version().dbr_version").show()

[mark_ott]

The error "AttributeError: 'DeltaMergeBuilder' object has no attribute 'withSchemaEvolution'" when running the same code from a .py script but not in a Python notebook is likely caused by a mismatch in the Delta Lake or Databricks Runtime versions or environment differences between these two execution contexts.

Key points explaining this issue and solution options:

• The method .withSchemaEvolution() on the DeltaMergeBuilder is available starting from Databricks Runtime 16.0 onwards for enabling automatic schema evolution during merge operations. Your runtime 16.1 supports this feature in principle.​

• This error occurs when the environment running the .py script is using an older version of the Delta Lake library or Databricks runtime than the interactive notebook environment. The notebook likely uses the newer runtime where .withSchemaEvolution() exists, while the script's environment might be using a version of the libraries that do not yet support this method.​

• Serverless compute environments and job compute may have different package or runtime configurations. One documented case shows schema evolution with .withSchemaEvolution() fails on serverless compute due to unsupported Spark configurations, even if the runtime version nominally supports it.​

• Verify the PySpark, Delta Lake, and Databricks Runtime versions in both environments by printing the version info or checking package versions programmatically.

• Ensure that the Python script execution environment is running the same runtime version (16.1) and uses the same package versions as the notebook, particularly for delta-spark and Spark packages.

• If the mismatch persists, consider explicitly setting Spark configurations such as spark.databricks.delta.schema.autoMerge.enabled to true and/or using alternative methods like setting .option("mergeSchema", "true") on DataFrame writes to enable schema changes.​

• Another workaround is to run the script as a Databricks job or in a runtime environment explicitly configured to match the notebook runtime.

In summary, although you are on runtime 16.1 which supports .withSchemaEvolution(), the Python script environment probably uses an older or incompatible delta-spark package version or runtime configuration. Aligning the runtime and package versions between the notebook and script mode will resolve the AttributeError. Configuring Spark session properly for schema evolution is also advised.

References:

• The .withSchemaEvolution() method requires Databricks Runtime 16.0+.​

• Serverless compute environment may lack support causing the missing attribute error.​

• Setting Spark conf spark.databricks.delta.schema.autoMerge.enabled as true is an alternative for schema evolution