DataFrame vs. Spark SQL

SamAWS
New Contributor III

‎01-13-2024 10:18 AM

When should I use DataFrame over Spark SQL?

If DataFrame is better, then why do I need Spark SQL?

0 Kudos

shan_chandra
Databricks Employee
Databricks Employee

‎01-13-2024 02:13 PM

@SamAWS - when you want to query the underlying data in the dataframe by creating a view or table, Spark SQL is preferred. Same can be achieved using pyspark by retaining them as a dataframe. Spark SQL is used by wider audience and easy to develop.