cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

File not found error. Does OPTIMIZE deletes initial versions of the delta table?

Data_Analytics1
Contributor III

df = (spark.readStream.format("delta")\

              .option("readChangeFeed", "true")\

              .option("startingVersion", 1)\

              .table("CatalogName.SchemaName.TableName")

             )

display(df)

A file referenced in the transaction log cannot be found. This occurs when data has been manually deleted from the file system rather than using the table `DELETE` statement.

When I see the path of the file, file is not present there. It was neither deleted manually nor Optimized.

Does delta table have any default setting to optimize the table? When I check the history of the table, few records of OPTIMIZE is there and after that OPTIMIZE I was not able to see previous versions of the table though the complete data is available but I cannot see the initial version data of the table.

1 ACCEPTED SOLUTION

Accepted Solutions

swethaNandan
New Contributor III
New Contributor III

Had you run vacuum on the table? Vacuum can clean up data files marked for removal and are older than retention period.

Optimize compacts files and marks the small files for removal, but does not physically remove the data files

View solution in original post

1 REPLY 1

swethaNandan
New Contributor III
New Contributor III

Had you run vacuum on the table? Vacuum can clean up data files marked for removal and are older than retention period.

Optimize compacts files and marks the small files for removal, but does not physically remove the data files

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!