Hi Suteja,

We verified the code and removed unwanted command also still we are facing issue. We tried clear cache using spark.catalog.clearCache() this code. Still there is no change in cluster performance.

Truncate table is now working fine. I found different for executing in the query in 12.0 runtime version.

import java.sql.{Connection, DriverManager, Statement}

val jdbcUrl = dbutils.secrets.get(scope = "azurekeyvault-scope", key = "nycsqlserver-DW-ConnectionString")

val user = dbutils.secrets.get(scope = "azurekeyvault-scope", key = "DW-Username")

val password = dbutils.secrets.get(scope = "azurekeyvault-scope", key = "DW-Password")

val updateQuery = "Truncate table tablename"

var conn: Connection = null

var stmt: Statement = null

try {

 conn = DriverManager.getConnection(jdbcUrl, user, password)

 stmt = conn.createStatement()

 stmt.executeUpdate(updateQuery)

} catch {

 case e: Exception => e.printStackTrace()

} finally {

 if (stmt != null) stmt.close()

 if (conn != null) conn.close()

}

By using above code I can execute the queries.

On the periodically restart part is there any chance to improve the performance with out restarting the cluster. Else, Is there any automatic cluster restart option available in databricks to avoid manual restart.