@Raman_Unifeye  and @Coffee77  there was a situation when a parquet file were deleted because it was obsolete when a vacuum was executed. Post which the job started to fail, it was saying unable to find the parquet file, even though it was reading an external location. The cluster had the old metadata of the delta file and not the latest one. Once i cleared the cache it started to work as expected.

Before I cleared the cache, I tried reading the same delta file from a different cluster, it was able to read without any issues.

More about the cluster, it was a long running interactive cluster in which the job was running at a frequent interval.