Spark Dataframe Metadata
Spark Dataframe is structurally the same as the table. However, it does not store any schema information in the metadata store. Instead, we have a runtime metadata catalog to store the Dataframe schema information. It is similar to the metadata store, but Spark will create it at the runtime to store schema information in the catalog
Why, reasons for storing schema information in the catalog?
Please find below*******************link
https://sharikrishna26.medium.com/spark-dataframe-metadata-a2544c16081d
