Hello @Retired_mod,
Thank you for the reply, after analyzing my code. I got to know that I was trying to create SparkSession inside the try block which automatically terminates the session when the process is finished. So for now I have removed try block. But now I'm facing another issue. Providing the error log below:

24/03/18 09:41:55 ERROR MicroBatchExecution: Query [id = 88d62783-1d5d-4a7e-b75c-875fa5d42892, runId = 24ec7106-189a-49f8-88ec-b4cb213d1496] terminated with error
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Incompatible format detected.

You are trying to write to `abfss://pdpdeltalake@stpdpdeltalakepoc.dfs.core.windows.net/test/di/sap-cic-plant-plant/` using Delta, but there is no
transaction log present. Check the upstream job to make sure that it is writing
using format("delta") and that you are trying to write to the table base path.