Hi @Debayan Mukherjee​ 

I don't have a custom spark conf (except the following line in order to make it ignore the missing file)

spark.sql.files.ignoreMissingFiles true

The cluster conf

Policy: Unrestricted
Multi node
Access mode: Single user
Databricks runtime version: 11.3 LTS (Scala 2.12, Spark 3.3.0)
Worker type: r5d.xlarge
Workers: 2 (64 GB Memory 8 cores)
Driver type: Same as worker (32 GB Memory, 4 Cores)

I'm using Unity Catalog also if that helps.