cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
alejandrofm
Valued Contributor
since ‎02-07-2022
‎08-07-2023

User Stats

  • 50 Posts
  • 1 Solutions
  • 8 Kudos given
  • 26 Kudos received

User Activity

Configure job to use one cluster instance to multiple jobs

07-31-2023
Hi! I have several tiny jobs that run in parallel and I want them to run on the same cluster:- Tasks type Python Script: I send the parameters this way to run the pyspark scripts.- Job compute cluster created as (copied JSON from Databricks Job UI)Ho...

How to change compression codec of sql warehouse written files?

06-02-2023
Hi, I'm currently starting to use SQL Warehouse, and we have most of our lake in a compression different than snappy.How can I set the SQL warehouse to use a compression like gzip, zstd, on CREATE, INSERT, etc?Tried this:set spark.sql.parquet.compre...

Lot of write shuffle on optimize + ZORDER, is it normal?

04-20-2023
Hi! I'm optimizing several Tb of partitioned data on ZSTD lvl 9.It surprises me the level of shuffle write, it could make sense because of ZORDER but I want to be sure that I'm not missing something, here is some context: Could I be missing something...

Understand if the configs I use to SparkSession.builder still make sense for Databricks 10+

03-30-2023
Hi! I currently have this as an old generic template with amends over time to optimize Databricks Spark execution, can you help me to know if this still makes sense for v10-11-12 or if there are new recommendations? Maybe some of this is making my pr...

AppendDataExecV1 Taking a lot of time

03-04-2023
Hi, I have a Pyspark job that takes about an hour to complete, when looking at the SQL tab on Spark UI I see this:Those processes run for more than 1 minute on a 60-minute process.This is Ganglia for that period (the last snapshot, will look into a l...
View more