topic Re: Convert SQL Query to Dataframe in Data Engineering

Convert SQL Query to Dataframe

143260 — Wed, 31 Jul 2024 14:25:52 GMT

Hello,

Being relatively new to the Databricks world, I'm hoping someone can show me how to take a SQL query and put the results into a dataframe. As part of data validation project, I'd like to cross join two dataframes.

Re: Convert SQL Query to Dataframe

Antoine_B — Wed, 31 Jul 2024 14:40:21 GMT

From a pyspark notebook, you could do:

df = spark.sql("SELECT * FROM my_table WHERE ...")

Then you can use this df and crossjoin it to another DataFrame

If you are new to databricks, I suggest you should follow some of the self paced lessons in databricks-academy, it helped me a lot when I started. For example: https://www.databricks.com/training/catalog/apache-spark-programming-with-databricks-1875

Re: Convert SQL Query to Dataframe

tt_mmo — Fri, 03 Jan 2025 20:15:11 GMT

Just to add here. So this is using the library(SparkR) package. My code is below:

library(SparkR)

df = sql("SELECT * FROM TABLE_SCHEMA.TABLE_NAME")

the comment above did not work directly for me. Hope this helps.