Databricks Community is an open-source platform for data enthusiasts and professionals to discuss, share insights, and collaborate on everything related to Databricks. Members can ask questions, share knowledge, and support each other in an environment that ensures respectful interactions.
Hi all. If you've ever manually promoted resources from dev to prod on Databricks — copying notebooks, updating configs, hoping nothing breaks — this post is for you. I've been building a CI/CD set...
Hi, Is there a way to custom name an assetbundle file name and pass that to databricks bundle deploy? I mean right now I must use databricks.yml, so my question is whether there is a way to pass a ...
The Hidden Cost of Scaling the Lakehouse Over the past few years, many organizations have successfully migrated to Databricks to modernize their data platforms. The Lakehouse architecture has enable...
Introduction “AI First” - But Data Always Comes First I have been working in the data space for close to two decades. My journey started as an ETL developer and gradually evolved into roles spannin...
...Using PySpark, I mounted the S3 location and read the data directly into a DataFrame: df = spark.read.option("header", True).csv("s3a://roh-databricks-v1/ecommerce-data/S3 Order Line Items.csv") T...
Hey everyone 👋 I recently started my Databricks learning journey about a month ago, and I wanted to share what I’ve learned so far from one beginner to another. Here are a few highlights: 1️⃣ Und...
Zero-Downtime Unity Catalog Migration for 500TB Data Lake Just completed migrating 500TB to Unity Catalog without a single minute of downtime. Here's how: The Challenge 500 TB across 12,000 table...
Published new video in my recently created youtube channel about one of my favorite topics: performance. So, here is a new video whose goal is to explain clearly what lazy evaluation is an...
Just published new video on Databricks Performance Series to try to clearly explain how DataFrame caching over Delta Tables behaves when updates on underlying table are performed. I came across this ...
Recently I earned the Databricks Machine Learning Professional certification and wanted to share my study journey. Before the exam, I worked on a project as a data engineer alongside data scientists ...