Comparing schemas of two dataframes

dream — Sat, 05 Aug 2023 11:14:12 GMT

So I was comparing schemas of two different dataframe using this code:

>>> df1.schema == df2.schema Out: False

But the thing is, both the schemas are completely equal.

When digging deeper I realized that some of the StructFields() that should have been equal have different metadata property

{'name': 'customer_id', 'dataType': StringType(), 'nullable': True, 'metadata': {}} {'name': 'customer_id', 'dataType': StringType(), 'nullable': True, 'metadata': {'scale': 0}}

What does this metadata property do?

Edit:

>>> all(str(x) == str(y) for x, y in zip(df1.schema, df2.schema)) Out: True

Re: Comparing schemas of two dataframes

Ajay-Pandey — Sat, 05 Aug 2023 14:56:42 GMT

Hi @dream ,

In this case, you can go with dataframe.dtypes for comparing the schema or datatypes for two dataframe
Metadata store information about column properties

topic Comparing schemas of two dataframes in Data Engineering

Comparing schemas of two dataframes

Re: Comparing schemas of two dataframes