Databricks Community

Coffee77

Spark shuffles can be confusing at first. I put together this simple illustration to explain the concept visually. Hope it's useful!

Coffee77

 How does Databricks process terabytes of data in just minutes?It's not about a single technology. It's the perfect combination of:里 Data Partitioning + ⚡ Apache Spark (distributed processing) +  Delta Lake (an optimized storage format).Just like a...

Coffee77 · 02-24-2026

 En este video, creado como recordatorio para mi mala memoria a largo plazo, explico de forma sencilla: ✅ Qué es un DataFrame ✅ Cómo se distribuye en particiones ✅ Cómo se ejecuta en un cluster (driver y workers) ✅ Qué ocurre en un shuffle✅ Relació...

Coffee77 · 01-20-2026

Here is a very short video on how to use collect() method with caution as it could overwhelm the cluster in some big data scenarios!!!! Now only in Spanish but I guess youtube will do the trick to translate into very correct English xDD

Coffee77 · 11-14-2025

Hi guys,I am trying to use table_changes CDF function in an external SQL function run it from an SQL Warehouse serverless cluster in order to retrieve a set of changes from a given table. Here is the documentation for this fuunction:https://docs.data...

Coffee77 · 02-24-2026

Take a look at this: https://docs.databricks.com/aws/en/jobs/monitor#finished-runs-count-graph I think that is the correct expected way of getting you're looking for.

Coffee77 · 02-24-2026

Nothing to do with that. You can use Free Edition in Azure for sure. I guess your problem is you should create your workspace in Azure, not in AWS. Here are more details about it: https://learn.microsoft.com/en-us/azure/databricks/getting-started/fre...

Coffee77 · 02-24-2026

Recently, I am creating some "self-reminder" videos for helping my long-term poor human memory  and maybe to help others. Understand internals of Dataframes, how partitions are related to jobs, stages, shuffles and tasks and, how transformations or ...

Coffee77 · 02-23-2026

This is a fairly generic question because the answer depends not only on the worker node configuration, but also on the Spark configuration—particularly how partitions are handled before and after shuffles. Other important considerations include whet...

Coffee77 · 01-20-2026

Thanks for mentioning with a little bit of delay 

Databricks Community

User Stats

User Activity

What Is a Spark Shuffle? A Simple Visual Explanation ⭐

🚀 How does Databricks process terabytes of data in just minutes?

🇪🇸 Por qué el DataFrame es el objeto de datos más importante en el procesamiento distribuido

Very short video on tip about Collect() method!!!

How to inject non-literal values in "table_changes" CDF function?

Re: Why count Run Status not Showing

Re: Issue on Service Credential creation for Azure Databricks access connector

Re: 🇪🇸 Por qué el DataFrame es el objeto de datos más importante en el procesamiento distribuido

Re: Best clusters configuration to process 100gb data??

Re: 🎬 Databricks Community 2025 Highlights | A Year, Built Together