Configuration spark.sql.sources.partitionOverwriteMode is not available.

Bram
New Contributor II

‎09-05-2023 06:53 AM

Dear,

 

In the current setup, we are using dbt as a modeling tool for our data lakehouse.

For a specific use case, we want to use the insert_overwrite strategy, where dbt will replace all data for a specific partition:

Databricks configurations | dbt Developer Hub (getdbt.com)

 

Hereby the specific dbt model configuration:

{{

    config(

        materialized='incremental',

        partition_by=['zcFiscalYearPeriod'],

        file_format='delta',  

        incremental_strategy='insert_overwrite',

        on_schema_change='sync_all_columns'

    )

}}

 

When dbt is execution the needed query’s, we retrieve the following message:

Qry:

/* {"app": "dbt", "dbt_version": "1.5.6", "dbt_databricks_version": "1.5.5", "databricks_sql_connector_version": "2.7.0", "profile_name": "user", "target_name": "default", "node_id": "model.data_platform.prep_InventoryStockDay_test"} */

set

  spark.sql.sources.partitionOverwriteMode = DYNAMIC

 

Msg:

Configuration spark.sql.sources.partitionOverwriteMode is not available.

 

 

Maybe important to note that we are connecting via a sql endpoint, to a unity-catalog enabled cluster.

 

We already contacted the DBT-team and everything looks good on dbt side, maybe it’s a setting in databricks we need to change?

0 Kudos