https://community.databricks.com/t5/data-engineering/when-to-use-dataframes-api-over-spark-sql/m-p/33196#M24270 <P>Hi Rathinam, It would be better to understand the pipeline more in this situation. Writing to table using CSV and then using spark SQL will be faster in few cases than the other one. </P> Tue, 30 Aug 2022 20:09:02 GMT Debayan 2022-08-30T20:09:02Z https://community.databricks.com/t5/data-engineering/when-to-use-dataframes-api-over-spark-sql/m-p/33195#M24269 <P>Hello Experts,</P><P></P><P>I am new to Databricks. Building data pipelines, I have both batch and streaming data.</P><P>Should I use Dataframes API to read csv files then convert to parquet format then do the transformation? </P><P>or</P><P>write to table using CSV then use Spark SQL to do transformation?.</P><P>Appreciate pros and cons and which one is better</P><P></P><P>Thank you</P><P>Rathinam</P> Tue, 30 Aug 2022 02:56:29 GMT https://community.databricks.com/t5/data-engineering/when-to-use-dataframes-api-over-spark-sql/m-p/33195#M24269 parthibsg 2022-08-30T02:56:29Z https://community.databricks.com/t5/data-engineering/when-to-use-dataframes-api-over-spark-sql/m-p/33196#M24270 <P>Hi Rathinam, It would be better to understand the pipeline more in this situation. Writing to table using CSV and then using spark SQL will be faster in few cases than the other one. </P> Tue, 30 Aug 2022 20:09:02 GMT https://community.databricks.com/t5/data-engineering/when-to-use-dataframes-api-over-spark-sql/m-p/33196#M24270 Debayan 2022-08-30T20:09:02Z