Recover files from previous cluster execution

[carlosna]

I saved a file with results by just opening a file via fopen("filename.csv", "a").

Once the execution ended (and the cluster shutted down) I couldn't retrieve the file.

I found that the file was stored in "/databricks/driver", and that folder empties when the cluster shuts down.

Is there any way I can retrieve it?

[Kaniz_Fatma · Accepted solution]

Hi @carlosna , Unfortunately, if the file was saved to the /databricks/driver location, it will be lost when the cluster is shut down. The driver node of a Databricks cluster is ephemeral, meaning that its storage is attached to the machine hosting the driver node, and is destroyed when the node is terminated.

To prevent this from happening, you should save your results to a more reliable location, such as DBFS (Databricks file system) or a cloud storage service such as Azure Blob Storage or AWS S3.

You can use DBFS to save data in an S3- like interface. The DBFS provides an abstraction over underlying cloud storage, allowing you to store different types of data such as notebooks, libraries, and data files. The files are persisted independently of the cluster's lifecycle, so you can save files in DBFS even after the cluster has terminated.

[Kaniz_Fatma]

Hi @carlosna , I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution.