Bloom Filters + Zonemaps: The Ultimate Query Optimization ComboAfter my zonemap post last week got great feedback, several of you asked about Bloom filter integration. Here's the complete implementation!Why Bloom Filters Changed EverythingZonemaps ar...
Just published new video on Databricks Performance Series to try to clearly explain how DataFrame caching over Delta Tables behaves when updates on underlying table are performed. I came across this use case in my recent project and struggled a littl...
Source Code with samples available at https://github.com/CafeConData/Spark-Caching-on-Delta-Tables
As promised @BS_THE_ANALYST , in this new video and summarized in post, I try to explain what Spark Caching and Databricks Disk Caching are and how Caching strategy can be leveraged by making these cool features work together: Spark Caching vs Databr...
As companies double down on machine learning (ML), one thing is obvious: a single model can’t solve every problem. Different datasets, different timelines, and different requirements make managing multiple models pretty tricky. And if you’ve ever wor...
Data governance is one of the most important pillars in any modern architecture. When building pipelines that process data at scale, ensuring data quality is not just a best practice—it is a critical necessity.Tools like Great Expectations (GX) were ...
Optimizing queries in Databricks isn’t just about adding indexes or tweaking SQL syntax — it’s about visibility. You can’t improve what you can’t measure. Fortunately, Databricks provides rich telemetry around query history that you can use to analyz...
Hi everyone! I wanted to share with you a post I wrote on Medium a while ago — it’s still very useful if you want to understand how to properly calculate Databricks cluster costs and get a realistic view of the differences: Databricks Serverless vs C...
Really interesting topic I'll take a look when possible Always interested in improving performance and saving cloud costs. Thanks for sharing.
Published new video in my recently created youtube channel about one of my favorite topics: performance. So, here is a new video whose goal is to explain clearly what lazy evaluation is and how to use caching to boost performance. https://studio.y...
Not sure if previous link works fine, here is the correct one https://youtu.be/pLGAr1VXQSQ?si=QaKzaNfzrNl_0Tv9
Here is the first episode in ENGLISH VERSION of a list of simple videos on Introduction to Databricks for beginners:Introduction to Databricks for Beginners - Episode 1 It contains previous and basic concepts to master before moving forward with Data...
Hi @Coffee77 ,Thanks for sharing with us. It looks promising. Can't wait for another episode
What is RAG?RAG (Retrieval-Augmented Generation) on Databricks refers to building and running AI applications that combine:Retrieval systems (like vector databases or search over documents)Generative AI models (such as LLMs like GPT)within the Databr...
Recently I earned the Databricks Machine Learning Professional certification and wanted to share my study journey. Before the exam, I worked on a project as a data engineer alongside data scientists (ML models, LLMs, MLflow). That led me to build a p...
Thanks a lot, my friend @BS_THE_ANALYST ! Really glad you found it useful . I’m sure when you dive into ML later this year, you’ll do awesome things with it. Appreciate the kind words about the project — means a lot! All the best to you too, and let’...
Here is the first episode of a serie of simple videos on Introduction to Databricks for beginners :https://youtu.be/kvglz79Ob-M?si=KnyCH74_HQ8jiO7SIt contains previous and basic concepts to master before moving forward with Databricks.
English version is ready : INTRODUCTION to DATABRICKS in English - Episode 1 ‌
The Data + AI Summit 2025 delivered several groundbreaking announcements, but none were more democratizing than the launch of the new Databricks Free Edition. Announced alongside a massive $100 million investment in training, this new offering provid...
Nicely outlined, @nayan_wylde!
Sign Up Go to https://www.databricks.com/try-databricksFill in the 2 steps box on the right hand side Note - It is important to select the Personal use section in the above step. Sign In Enter your details here https://accounts.cloud.databricks.com/...
Hi,I cannot signup to Community edition. When I try to sign up using this link https://www.databricks.com/try-databricks it first shows this pop up, non of these two options allows me to signup for community edition. I don't find option 'get started...
In a data migration project, I needed to generate the schema of a PostgreSQL table to use in my ETL process. I’d like to share the code snippet in case someone else needs it one day:from pyspark.sql import SparkSession import json import os from typi...
