Remove the Default JAR:

• Since spark-bigquery-with-dependencies_2.12-0.41.0.jar is included by default in Databricks Runtime 15.4 LTS, and you need to exclude it. This can be done by creating an init script to remove the JAR file from the cluster.

Create an Init Script:

• Create an init script that removes the default spark-bigquery-with-dependencies_2.12-0.41.0.jar from the cluster. Here is an example of what the script might look like:

  #!/bin/bash
  rm /databricks/jars/spark-bigquery-with-dependencies_2.12-0.41.0.jar

Upload the Init Script:

• Upload this script to a location accessible by Databricks, such as DBFS (Databricks File System).

Configure the Cluster to Use the Init Script:

• Go to the cluster configuration page in Databricks.
• Under the "Advanced Options" section, find the "Init Scripts" tab.
• Add the path to your init script (e.g., dbfs:/path/to/your/init-script.sh).

Add the Custom JAR:

• Upload the spark-3.5-bigquery-0.41.0.jar to DBFS or another accessible location.
• In the cluster configuration, go to the "Libraries" tab.
• Choose "Install New" and select "DBFS" or the appropriate option where your JAR is stored.
• Provide the path to the spark-3.5-bigquery-0.41.0.jar.

Restart the Cluster:

• Restart the cluster to apply the changes. The init script will run, removing the default JAR, and the new JAR will be added to the cluster.