Per API docs on StreamingQuery.stop(),
https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/streaming/StreamingQuery.html
It says, this stops the execution of this query if it is running and waits until the termination of the query...
Enclose the SQL string in double quotes, assign an empty string to param_end, and on the SQL string, enclose the param_end in single quotes, Eg:param_end = ""stmt_insert_audit_record = "INSERT INTO default.batch_run (task_name, start, end, status) v...
@Karthikeyan Palanisamy , "Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient" - This can probably due to a connection issue with HMS. Can you check the connectivity?
@Sriram Kumar Are they just inserts? Then you can optimize it without affecting it: https://docs.databricks.com/optimizations/isolation-level.html#write-conflicts-on-databricks.
