I am getting an error when I am trying to view the underlying files of the managed delta table in unity catalog
such as
from pyspark.sql.functions import *
table_directory = "workspace.db_bronze.test_01"
data = [{"x": 1, "y": 2}]
df = spark.createDataFrame(data)
(
df.write.format('delta')
.mode("overwrite")
.saveAsTable(table_directory)
)
details = spark.sql(f"""DESCRIBE DETAIL workspace.db_bronze.test_01""")
display(details)
# showing some columns below; location is blank
format | id | name | description | location |
delta | some_value | workspace.db_bronze.test_01 | null | |
Also, the following generates an error
df = spark.table("workspace.db_bronze.test_01")
input_files = df.inputFiles()
# print("Table files:")
for file_path in input_files:
print(f" {file_path}")
# Extract
if input_files:
import os
table_directory = os.path.dirname(input_files[0])
# Simple directory listing - no recursion
files = dbutils.fs.ls(table_directory)
print(files)
# Error - Input path url 's3://url' overlaps with managed storage within 'ListFiles' call. .
Do I understand that, paths for managed tables (Unity Catalog creates a new directory in the Unity Catalog-configured storage location associated with the containing schema) are not accessible? Databricks Manged Paths