dataframe column structtype metadata not getting saved to unitycatalog

vk217 — Sat, 04 Nov 2023 23:51:24 GMT

I have the below schema

schema = StructType([
StructField(
name="Test",
dataType=StringType(),
nullable=False,
metadata={
"description": "This is to test metadata description."
}
)
])

data = [('Test1',), ('Test2',), ('Test3',)] df = spark.createDataFrame(data, schema=schema)

and I am writing to datalake

df.write.mode("overwrite").format("delta").saveAsTable(table_name)

Now when I read the from the unity catalog table and try to access the metadata, I don't see it.

from pyspark.sql import functions as F df = spark.read.format("delta").table(table_name) print(df.schema['Test'].metadata)

Where is the metadata from that column stored? I have checked information_schema.columns as well and I don't see it.

topic dataframe column structtype metadata not getting saved to unitycatalog in Get Started Discussions

dataframe column structtype metadata not getting saved to unitycatalog