Databricks

Thor · 07-31-2023

Hello,I'm facing a problem with big tarballs to decompress and to fit in memory I had to limit Spark processing too many files at the same time so I changed the following property on my 8 cores VMs cluster:spark.task.cpus 4 This setting is the thresh...

Thor · 05-19-2023

I already improved a lot the performances of our ETL (x20 !) but I still want to know where I can improve performances. I seems that tables stats and column indexing slow down a bit writings so I want to decrease dataSkippingNumIndexedCols to match t...

Thor · 05-19-2023

I made multiple inserts (by error) in a Delta table and I have now strict duplicates, I feel like it's impossible to delete them if you don't have a column "IDENTITY" to distinguish lines (the primary key is RLOC+LOAD_DATE):it sounds odd to me not to...

Thor · 09-08-2023

I agree at 100% with you. How can you find a proper value when the workload is completely between the first run and the following regular runs?

Thor · 05-22-2023

maybe the information I need stands in the value '_metadata.row_index'

Databricks

User Stats

User Activity

Dynamically change spark.task.cpus

What is the performance impact of changing dataSkippingNumIndexedCols from 32 to 16?

How to remove duplicates in a Delta table?

Re: Changing shuffle.partitions with spark.conf in a spark stream - isn't respected even after a che

Re: How to remove duplicates in a Delta table?