https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14076#M8628 <P>good post thanks</P> Sun, 01 Jan 2023 08:09:52 GMT Aviral-Bhardwaj 2023-01-01T08:09:52Z https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14074#M8626 <P><B>Spark Dataframes Schema</B></P><P></P><P><B>Schema inference is not reliable.</B></P><P>We have the following problems in schema inference:</P><OL><LI>Automatic inferring of schema is often incorrect</LI><LI>Inferring schema is additional work for Spark, and it takes some extra time</LI><LI>Schema inference is conflicting with the schema validation</LI></OL><P>4. It might also change the column order</P><P></P><P>We have two approaches to do it.</P><OL><LI>Schema DDL String</LI><LI>Struct Type Object</LI></OL><P></P><P></P><P>Further Detailed description please refer this link</P><P></P><P><A href="https://sharikrishna26.medium.com/spark-dataframes-schema-6fe1f90a56c" target="test_blank">https://sharikrishna26.medium.com/spark-dataframes-schema-6fe1f90a56c</A></P><P></P><P></P><P>Please like,share,comment</P><P></P><P>Happy New year 2023</P><P></P> Sat, 31 Dec 2022 13:38:45 GMT https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14074#M8626 SIRIGIRI 2022-12-31T13:38:45Z https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14075#M8627 <P>Thanks for sharing </P> Sat, 31 Dec 2022 16:26:30 GMT https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14075#M8627 Rishabh-Pandey 2022-12-31T16:26:30Z https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14076#M8628 <P>good post thanks</P> Sun, 01 Jan 2023 08:09:52 GMT https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14076#M8628 Aviral-Bhardwaj 2023-01-01T08:09:52Z https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14077#M8629 <P>one other difference between those 2 approaches is that In Schema DDL String approach we use STRING, INT etc.. But In Struct Type Object approach we can only use Spark datatypes such as StringType(), IntegerType(), etc..</P> Mon, 02 Jan 2023 03:05:25 GMT https://community.databricks.com/t5/data-engineering/sharikrishna26-medium-com/m-p/14077#M8629 Varshith 2023-01-02T03:05:25Z