Convert SQL Query to Dataframe

143260
New Contributor

‎07-31-2024 07:25 AM

Hello,

Being relatively new to the Databricks world, I'm hoping someone can show me how to take a SQL query and put the results into a dataframe. As part of data validation project, I'd like to cross join two dataframes.

0 Kudos

Antoine_B
Contributor

‎07-31-2024 07:31 AM - edited ‎07-31-2024 07:40 AM

From a pyspark notebook, you could do:

df = spark.sql("SELECT * FROM my_table WHERE ...")

Then you can use this df and crossjoin it to another DataFrame

If you are new to databricks, I suggest you should follow some of the self paced lessons in databricks-academy, it helped me a lot when I started. For example: https://www.databricks.com/training/catalog/apache-spark-programming-with-databricks-1875

tt_mmo
New Contributor II
In response to Antoine_B

‎01-03-2025 12:15 PM

Just to add here. So this is using the library(SparkR) package. My code is below:

library(SparkR)

df = sql("SELECT * FROM TABLE_SCHEMA.TABLE_NAME")
 
the comment above did not work directly for me. Hope this helps. 
0 Kudos