https://community.databricks.com/t5/get-started-discussions/dataframe-vs-spark-sql/m-p/57257#M6353 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/86267">@SamAWS</a>&nbsp;- when you want to query the underlying data in the dataframe by creating a view or table, Spark SQL is preferred. Same can be achieved using pyspark by retaining them as a dataframe. Spark SQL is used by wider audience and easy to develop.&nbsp;</P> Sat, 13 Jan 2024 22:13:24 GMT shan_chandra 2024-01-13T22:13:24Z https://community.databricks.com/t5/get-started-discussions/dataframe-vs-spark-sql/m-p/57254#M6352 <P>When should I use DataFrame over Spark SQL?</P><P>If DataFrame is better, then why do I need Spark SQL?</P> Sat, 13 Jan 2024 18:18:14 GMT https://community.databricks.com/t5/get-started-discussions/dataframe-vs-spark-sql/m-p/57254#M6352 SamAWS 2024-01-13T18:18:14Z https://community.databricks.com/t5/get-started-discussions/dataframe-vs-spark-sql/m-p/57257#M6353 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/86267">@SamAWS</a>&nbsp;- when you want to query the underlying data in the dataframe by creating a view or table, Spark SQL is preferred. Same can be achieved using pyspark by retaining them as a dataframe. Spark SQL is used by wider audience and easy to develop.&nbsp;</P> Sat, 13 Jan 2024 22:13:24 GMT https://community.databricks.com/t5/get-started-discussions/dataframe-vs-spark-sql/m-p/57257#M6353 shan_chandra 2024-01-13T22:13:24Z