cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Thor
New Contributor III
since ‎05-19-2023
4 weeks ago

User Stats

  • 7 Posts
  • 0 Solutions
  • 4 Kudos given
  • 4 Kudos received

User Activity

Asynchronous progress tracking with foreachbatch

4 weeks ago
Hello,currently the doc says that async progress tracking is available only for Kafka sink:https://docs.databricks.com/en/structured-streaming/async-progress-checking.htmlI would like to know if it would work for any sink that is "exactly once"?I exp...

Native code in Databricks clusters

08-30-2024
Is it possible to install our own binaries (lib or exec) in Databricks clusters and use JNI to execute them?I guess that Photon is native code as far as I could read so it must use a similar technic.

Dynamically change spark.task.cpus

07-31-2023
Hello,I'm facing a problem with big tarballs to decompress and to fit in memory I had to limit Spark processing too many files at the same time so I changed the following property on my 8 cores VMs cluster:spark.task.cpus 4 This setting is the thresh...

What is the performance impact of changing dataSkippingNumIndexedCols from 32 to 16?

05-19-2023
I already improved a lot the performances of our ETL (x20 !) but I still want to know where I can improve performances. I seems that tables stats and column indexing slow down a bit writings so I want to decrease dataSkippingNumIndexedCols to match t...

How to remove duplicates in a Delta table?

05-19-2023
I made multiple inserts (by error) in a Delta table and I have now strict duplicates, I feel like it's impossible to delete them if you don't have a column "IDENTITY" to distinguish lines (the primary key is RLOC+LOAD_DATE):it sounds odd to me not to...
Kudos from
Kudos given to