Databricks Community

soumiknow

We have a date (DD/MM/YYYY) partitioned BQ table. We want to update a specific partition data in 'overwrite' mode using PySpark. So to do this, I applied 'spark.sql.sources.partitionOverwriteMode' to 'DYNAMIC' as per the spark bq connector documentat...

soumiknow · 08-29-2024

I have a multi-tenant Azure app. I am using this app's credentials to read ADLS container files from Databricks cluster using PySpark dataframe.I need to set this 'additionallyAllowedTenants' flag value to '*' or a specific tenant_id of the multi-ten...

soumiknow

The init_script worked to remove the unwanted jars from the cluster. This is good news.But whenever I tried to install the required jar on cluster configuration using library source as GCS path (gs://spark-lib/bigquery/spark-3.5-bigquery-0.41.0.jar),...

soumiknow

I raised an issue to spark-biquery-connector where they mentioned to use 'spark-3.5-bigauery-0.41.0.jar' whereas I was using Databricks Runtime Version: 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12) which include spark-bigquery-with-dependencies...

soumiknow

'spark.sql.sources.partitionOverwriteMode' is visible under Spark UI Environment tab. So this is not the issue.

soumiknow

I already tried this before, which was not worked. Then I tried with 'DYNAMIC' instead of 'dynamic'. But no luck.One thing I found that, even though I set the 'spark.sql.sources.partitionOverwriteMode' to 'DYNAMIC' in cluster advance options or even ...

soumiknow

Hi @Walter_C ,Thank you for your response over this issue.I tried both the options:- setting up the spark configuration at session level, but did not work- used the replaceWhere option for a specific partition date, but that also did not work.In both...

Databricks Community

User Stats

User Activity

BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMIC

How to add 'additionallyAllowedTenants' in Databricks config or PySpark config?

Re: BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMI

Re: BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMI

Re: BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMI

Re: BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMI

Re: BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMI