01-16-2023 05:34 AM
Determining location of DBIO file fragments. This operation can take some time.
What does this mean, and how do I prevent it from having to perform this apparently-expensive operation every time? This happens even when all the underlying tables are Delta tables.
01-16-2023 09:25 AM
Hey @Ajay Pandey ,
That message is related to delta caching, basically if a cluster is constantly scaling up or down then occasionally you might lose delta cache pieces. Determining the location of DBIO file fragments is the operation determining which executors the files were cached.
This is something that can be helped by trying a newer DBR such as 11.3 or 12.X. You could also try turning off the cache by setting the below configuration in the notebook and observing the behaviour:
spark.conf.set("spark.databricks.io.cache.enabled", "false")
You could also try optimizing the table(s)
%sql Optimize [table name]
01-16-2023 09:25 AM
Hey @Ajay Pandey ,
That message is related to delta caching, basically if a cluster is constantly scaling up or down then occasionally you might lose delta cache pieces. Determining the location of DBIO file fragments is the operation determining which executors the files were cached.
This is something that can be helped by trying a newer DBR such as 11.3 or 12.X. You could also try turning off the cache by setting the below configuration in the notebook and observing the behaviour:
spark.conf.set("spark.databricks.io.cache.enabled", "false")
You could also try optimizing the table(s)
%sql Optimize [table name]
01-18-2023 10:45 PM
Thanks
01-16-2023 09:32 AM
That is a message about the delta cache. It’s determines on which executors it has what cached, to route tasks for best cached locality. Optimizing your table more frequently so there are fewer files will make this better
U can try:
%sql Optimize [table name]
01-18-2023 10:45 PM
Thanks
01-19-2023 02:44 AM
Determining location of DBIO file fragments" is a message that may be displayed during the boot process of a computer running the NetApp Data ONTAP operating system. This message indicates that the system is currently in the process of identifying and locating the DBIO (Data Block Input/Output) file fragments on the storage system. This process is necessary in order to ensure that all data on the system is accessible and in a consistent state.
The time it takes to complete this process can depend on several factors, such as the number of disks in the system, the amount of data stored on the disks, and the performance of the disks themselves. However, there are a few things you can do to potentially speed up this process:
Keep in mind that this process is an important step in ensuring data integrity, it should not be skipped or rushed. It's crucial to be patient and let the process finish.
01-21-2023 11:21 PM
Thanks
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group