12-10-2024 04:19 AM
@bcsalay This could possibly be a cache issue. Are you doing dataframe.cache() anywhere in your code? if so please follow that by using unpersist()Also can you use the below config at cluster levelspark.databricks.io.cache.enabled false
