I have the below schema
schema = StructType([
StructField(
name="Test",
dataType=StringType(),
nullable=False,
metadata={
"description": "This is to test metadata description."
}
)
])
data = [('Test1',), ('Test2',), ('Test3',)]
df = spark.createDataFrame(data, schema=schema)
and I am writing to datalake
df.write.mode("overwrite").format("delta").saveAsTable(table_name)
Now when I read the from the unity catalog table and try to access the metadata, I don't see it.
from pyspark.sql import functions as F
df = spark.read.format("delta").table(table_name)
print(df.schema['Test'].metadata)
Where is the metadata from that column stored? I have checked information_schema.columns as well and I don't see it.