Convert SQL Query to Dataframe

[143260]

Hello,

Being relatively new to the Databricks world, I'm hoping someone can show me how to take a SQL query and put the results into a dataframe. As part of data validation project, I'd like to cross join two dataframes.

[Antoine_B]

From a pyspark notebook, you could do:

df = spark.sql("SELECT * FROM my_table WHERE ...")

Then you can use this df and crossjoin it to another DataFrame

If you are new to databricks, I suggest you should follow some of the self paced lessons in databricks-academy, it helped me a lot when I started. For example: https://www.databricks.com/training/catalog/apache-spark-programming-with-databricks-1875

[tt_mmo]

Just to add here. So this is using the library(SparkR) package. My code is below:

library(SparkR)

df = sql("SELECT * FROM TABLE_SCHEMA.TABLE_NAME")

the comment above did not work directly for me. Hope this helps.