cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
vanepet
by • New Contributor II
  • 14034 Views
  • 5 replies
  • 2 kudos

Is it possible to use multiprocessing or threads to submit multiple queries to a database from Databricks in parallel?

We are trying to improve our overall runtime by running queries in parallel using either multiprocessing or threads. What I am seeing though is that when the function that runs this code is run on a separate process it doesnt return a dataFrame with...

Data Engineering
Reply
  • 14034 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
BapsDBS
New Contributor II
  • 2 kudos

Thanks for the links mentioned above. But both of them uses raw python to achieve parallelism. Does this mean Spark (read PySpark) does exactly provisions for parallel execution of functions or even notebooks ? We used a wrapper notebook with ThreadP...

  • 2 kudos
4 More Replies
gilo12
by • New Contributor III
  • 1369 Views
  • 2 replies
  • 0 kudos

Running multiple Databricks queries with golang client

I am using github.com/databricks/databricks-sql-go to connect and run queries on my Databricks warehouse. However when I try to run multiple queries:insert into....; insert into....;It fails and complains on syntax errors.[PARSE_SYNTAX_ERROR] Syntax ...

Data Engineering
Reply
  • 1369 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Gil Omer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your...

  • 0 kudos
1 More Replies
Ryan_Chynoweth
by • Esteemed Contributor
  • 774 Views
  • 1 replies
  • 0 kudos

Can I have multiple queries in a pipeline writing to the same target table?

Data Engineering
Reply
  • 774 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

No, each table must be defined once. You can Use UNION If you need to combine multiple inputs to create a table. Adding or removing UNION from an incremental table is a breaking operation that requires a full-refresh.

  • 0 kudos
Labels