When to use Dataframes API over Spark SQL

parthibsg
New Contributor II

‎08-29-2022 07:56 PM

Hello Experts,

I am new to Databricks. Building data pipelines, I have both batch and streaming data.

Should I use Dataframes API to read csv files then convert to parquet format then do the transformation?

or

write to table using CSV then use Spark SQL to do transformation?.

Appreciate pros and cons and which one is better

Thank you

Rathinam

Labels:

Debayan
Databricks Employee
Databricks Employee

‎08-30-2022 01:09 PM

Hi Rathinam, It would be better to understand the pipeline more in this situation. Writing to table using CSV and then using spark SQL will be faster in few cases than the other one.