When to use Dataframes API over Spark SQL

[parthibsg]

Hello Experts,

I am new to Databricks. Building data pipelines, I have both batch and streaming data.

Should I use Dataframes API to read csv files then convert to parquet format then do the transformation?

or

write to table using CSV then use Spark SQL to do transformation?.

Appreciate pros and cons and which one is better

Thank you

Rathinam

[Debayan]

Hi Rathinam, It would be better to understand the pipeline more in this situation. Writing to table using CSV and then using spark SQL will be faster in few cases than the other one.