Where can I find those delta table log files?

JacobLi_LN
New Contributor II

‎11-18-2024 09:37 AM

I created a delta table with SQL command CREATE TABLE, and inserted several records into with INSERT statements. Now it can be seen from the catalog.

JacobLi_LN_0-1731951036765.png

But I want to understand how delta works, and would like to see where are those log files stored.

Even when I already toggled the dbfs option on, I still cannot find the files within dbfs.

JacobLi_LN_1-1731951121485.png

JacobLi_LN_2-1731951201603.png

The table seems to be located under a storage account but I do not have the permission to directly interact with it

JacobLi_LN_3-1731951363574.png

Anyone knows how to resolve this?

 

0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎11-18-2024 10:59 AM

To locate the log files for your Delta table, please first note that Delta Lake stores its transaction log files in a specific directory within the table's storage location. These log files are for maintaining the ACID properties and enabling features like time travel and versioning.

Given that the table is located under a storage account and you do not have permission to directly interact with it, this is likely the reason why you are unable to see the log files in DBFS. The transaction log files are typically stored in a _delta_log directory within the table's root directory

 

Since you mentioned you do not have permission to interact with the storage account directly, you might need to request access from your administrator or the person managing the storage account. Without the necessary permissions, you won't be able to view or interact with the files directly.

 

If you have access to Databricks utilities, you can use the dbutils.fs commands to navigate and list the contents of the table's directory. For example:

dbutils.fs.ls('dbfs:/path/to/your/delta/table/_delta_log/')

View solution in original post