Databricks Community

huyd · 11-22-2022

I am doing a batch load, using the JDBC driver from a database table. I am noticing in Sparkui, that there is both memory and disk spill, but only on one executor. I am also, noticing that when trying to use the JDBC parallel read, it seems to run sl...

huyd · 12-29-2022

If you can convert your Python udfs to sql udfs. These play nice adaptive query executions and won’t have performance penalties.

huyd · 12-28-2022

its whats posted by Databricks, so, I can’t really confirm, but when I wrote the exam (v2), I encountered similar type questions.

huyd · 12-17-2022

check your read cell, "Delimeter"

huyd · 12-17-2022

I studied, using both. If you intended on writing the V2, which is still available, you could use V2 or V3, where the V3 course will touch on more topics. I would suggest studying the V3 and writing the V3, it covers more topics, as more features ha...

huyd · 12-12-2022

This might help https://dustinvannoy.com/2022/05/06/parallel-ingest-spark-notebook/Write a function to pass your table list via array, into a queue for threading

Databricks Community

User Stats

User Activity

Optimizing a batch load process, reading with the JDBC driver

Re: Calling a python function (def) in databricks

Re: Certified Data Engineer Associate - v2 vs. v3 (Databricks Academy)

Re: hi all - I have created a data frame and would like to save in delta format using df.write.format("delta").saveAsTable("tablename") Its not working and throws " AnalysisException" . Please advise .

Re: Certified Data Engineer Associate - v2 vs. v3 (Databricks Academy)

Re: Is it possible to use multiprocessing or threads to submit multiple queries to a database from Databricks in parallel?