sharikrishna26.medium.com

SIRIGIRI
Databricks Partner

Spark Dataframe Metadata

Spark Dataframe is structurally the same as the table. However, it does not store any schema information in the metadata store. Instead, we have a runtime metadata catalog to store the Dataframe schema information. It is similar to the metadata store, but Spark will create it at the runtime to store schema information in the catalog

Why, reasons for storing schema information in the catalog?

Please find below*******************link

https://sharikrishna26.medium.com/spark-dataframe-metadata-a2544c16081d

Rishabh-Pandey
Databricks MVP

Great ​

Rishabh Pandey

Aviral-Bhardwaj
Esteemed Contributor III

this is awesome thanks

AviralBhardwaj