cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 

Running large volume of SQL queries in Python notebooks. How to minimise overheads/maintenance.

dispersion
New Contributor

โ€Ž02-13-2023 05:39 AM

I have around 200 SQL queries id like to run in databricks python notebooks. Id like to avoid creating an ETL process for each of the 200 SQL processes.

Any suggestions on how to run the queries in a way that it loops through them so i have minimum amount of components to maintain.

0 Kudos
2 REPLIES 2

Anonymous
Not applicable

โ€Ž02-13-2023 11:19 AM

It depends what you want to do. You can certainly use python to write loops over spark.sql() queries.

Anonymous
Not applicable

โ€Ž02-21-2023 02:17 AM

Hi @Chris Frenchโ€‹ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

0 Kudos
Announcements