topic Run pyspark queries from outside databricks in Data Engineering

Run pyspark queries from outside databricks

SowmyaDesai — Tue, 03 Sep 2024 15:51:05 GMT

I have written a Notebook which would execute pyspark query. I then execute it remotely from outside databricks environment using /api/2.1/jobs/run-now, which would then run the notebook. I also want to retrieve the results from this job execution. How should I do that?

response = requests.post(

f"{DATABRICKS_INSTANCE}/api/2.1/jobs/run-now",

headers={"Authorization": f"Bearer {API_TOKEN}"},

json={

"job_id": JOB_ID,

"notebook_params": {

"query": SQL_QUERY1

}

)

Notebook which would run pyspark

dbutils.widgets.text("query", "")

query = dbutils.widgets.get("query")

# Execute the query

spark = SparkSession.builder.getOrCreate()

df = spark.sql(query)

df.show()

# Return a value from the notebook

#dbutils.notebook.exit('hello!')

#return('Hello')

Re: Run pyspark queries from outside databricks

szymon_dybczak — Tue, 03 Sep 2024 16:07:30 GMT

Hi @SowmyaDesai ,

For this use case it's much better to use statement execution API which gives you ability to run SQL statement and fetch the results:

https://docs.databricks.com/api/workspace/statementexecution

Re: Run pyspark queries from outside databricks

SowmyaDesai — Wed, 04 Sep 2024 05:11:22 GMT

Thanks for responding. I did go through this link. It talks about executing on SQL warehouse though. Is there a way we can execute queries on Databricks clusters instead?

Databricks has this connector for SQL https://docs.databricks.com/en/dev-tools/python-sql-connector.html , it supports SQL queries. But I do not see easier option for supporting pyspark queries. Any idea of how to do that?

spark = SparkSession.builder.getOrCreate()

df = spark.sql(query)

Re: Run pyspark queries from outside databricks

szymon_dybczak — Wed, 04 Sep 2024 05:51:09 GMT

Hi @SowmyaDesai ,

So if you want to run queries outside databricks you can use Databricks Connect. Databricks Connect allows you to connect popular IDEs such as Visual Studio Code, PyCharm, RStudio Desktop, IntelliJ IDEA, notebook servers, and other custom applications to Databricks compute:

Databricks Connect for Python | Databricks on AWS