Hi all,

Currently we are using Driver: Standard_D32s_v3 · Workers: Standard_D32_v3 · 2-8 workers · 6.4 Extended Support (includes Apache Spark 2.4.5, Scala 2.11) cluster. For this we are running 24/7 streaming notebook on trigger of every minute and 5 more pipelines with different time line triggers. when we are starting the all triggers after 24hrs normal execution. Later streaming and pipeline related notebook are started failed. It means notebooks execution time is increasing and every command taking more time to execute.Above image shows cluster metrics after 24 hrs of execution.

https://dbricks.co/3VWeuRJ after going through this link I started upgrading the 12.0 runtime version. For this I am facing issue while executing DDL and DML queries in the below mentioned format.

import com.microsoft.azure.sqldb.spark.config.Config

import com.microsoft.azure.sqldb.spark.connect._

import com.microsoft.azure.sqldb.spark.query._

val query = "Truncate table tablename"

val config = Config(Map(

 "url"     -> dbutils.secrets.get(scope = "azurekeyvault-scope", key = "DW-URL"),

 "databaseName" -> dbutils.secrets.get(scope = "azurekeyvault-scope", key = "DW-DBName"),

 "user"     -> dbutils.secrets.get(scope = "azurekeyvault-scope", key = "DW-Username"),

 "password"   -> dbutils.secrets.get(scope = "azurekeyvault-scope", key = "DW-Password"),

 "queryCustom" -> query

))

sqlContext.sqlDBQuery(config)

can anyone suggest best way of approach to resolve my issue.

Thanks and Regards.