dataframe column structtype metadata not getting saved to unitycatalog

vk217
Contributor

I have the below schema

schema = StructType([
StructField(
name="Test",
dataType=StringType(),
nullable=False,
metadata={
"description": "This is to test metadata description."
}
)
])

data = [('Test1',), ('Test2',), ('Test3',)]
df = spark.createDataFrame(data, schema=schema)

and I am writing to datalake 

df.write.mode("overwrite").format("delta").saveAsTable(table_name)

Now when I read the from the unity catalog table and try to access the metadata, I don't see it.

 

from pyspark.sql import functions as F
df = spark.read.format("delta").table(table_name)
print(df.schema['Test'].metadata)

 

Where is the metadata from that column stored? I have checked information_schema.columns as well and I don't see it.