- 8488 Views
- 5 replies
- 8 kudos
API Consumption on Databricks
In this blog, I will be talking about the building the architecture to serve the API consumption on Databricks Platform. I will be using Lakebase approach for this. It will be useful for this kind of API requirement.API Requirement: Performance:Curre...
- 8488 Views
- 5 replies
- 8 kudos
- 364 Views
- 2 replies
- 2 kudos
Resolved! My First Month Learning Databricks - Key Takeaways So Far.
Hey everyone I recently started my Databricks learning journey about a month ago, and I wanted to share what I’ve learned so far from one beginner to another.Here are a few highlights:1️⃣ Understanding the Lakehouse Concept - Realized how Databricks ...
- 364 Views
- 2 replies
- 2 kudos
- 2 kudos
I was planning to build an ETL pipeline, but I hadn’t considered using MLflow to predict sales and ratings. Thanks for the suggestion, I’ll work on creating this demo soon to test and enhance my skills.
- 2 kudos
- 551 Views
- 2 replies
- 5 kudos
I Tried Teaching Databricks About Itself — Here’s What Happened
Hi All, How are you doing today?I wanted to share something interesting from my recent Databricks work — I’ve been playing around with an idea I call “Real-Time Metadata Intelligence.” Most of us focus on optimizing data pipelines, query performance,...
- 551 Views
- 2 replies
- 5 kudos
- 5 kudos
I like the core idea. You are mining signals the platform already emits.I would start rules first, track small files ratio and average file size trend, watch skew per partition and shuffle bytes per input gigabyte. Compare job time to input size to c...
- 5 kudos
- 122 Views
- 0 replies
- 1 kudos
Last chance to register for our LIVE Lakebase BrickTalks session!
Join us tomorrow, Thursday, Nov 13 at 9 am PT for the latest BrickTalks! We'll talk about bringing data intelligence from your Lakehouse into every app. Register now. What you’ll learn: Use Lakebase (PostgreSQL-compatible, serverless OLTP) to serve...
- 122 Views
- 0 replies
- 1 kudos
- 224 Views
- 0 replies
- 1 kudos
How Upgrading to Databricks Runtime 16.4 sped up our Python script by 10x
Wanted to share something that might save others time and money. We had a complex Databricks script that ran over 1.5 hours, when the target was under 20 minutes. Initially tried scaling up the cluster, but real progress came from simply upgrading th...
- 224 Views
- 0 replies
- 1 kudos
- 228 Views
- 0 replies
- 1 kudos
Control Databricks Costs with AI & BI Dashboards - Video Summary
In this video, I try to showcase in a very simplified way how to enable and setup AI & BI dashboards to control costs and take actions. I hope this could be useful. I think it is a superb feature to get insights on costs while straightforward to setu...
- 228 Views
- 0 replies
- 1 kudos
- 384 Views
- 2 replies
- 10 kudos
Optimizing Delta Table Writes for Massive Datasets in Databricks
Problem StatementIn one of my recent projects, I faced a significant challenge: Writing a huge dataset of 11,582,763,212 rows and 2,068 columns to a Databricks managed Delta table.The initial write operation took 22.4 hours using the following setup:...
- 384 Views
- 2 replies
- 10 kudos
- 10 kudos
Hey @Louis_Frolio ,Thank you for the thoughtful feedback and great suggestions!A few clarifications:AQE is already enabled in my setup, and it definitely helped reduce shuffle overhead during the write.Regarding Column Pruning, in this case, the fina...
- 10 kudos
- 237 Views
- 0 replies
- 2 kudos
Another BrickTalks! Let's talk about bringing data intelligence from your Lakehouse into every app!
You asked, we delivered! Another BrickTalk is scheduled for Thursday, Nov 13 @ 9 AM PT with Pranav Aurora on how to bring data intelligence from your Lakehouse into every app and user, seamlessly and in real time. What you’ll learn: Use Lakebase (Po...
- 237 Views
- 0 replies
- 2 kudos
- 410 Views
- 3 replies
- 11 kudos
Community Fellows: Shout Out to our Bricksters!
At Databricks, our Community members deserve to get a great experience in our forums, with quality answers from the experts. Who better to help out our customers than Databricks employees aka Bricksters! To work towards this goal, we created the Comm...
- 410 Views
- 3 replies
- 11 kudos
- 11 kudos
Kudos to the DB team for keeping up with the community, but can you please work on your product as well?We are experiencing a lot of issues with your paid product: failures, crashes, slow starts and slow performance and the list goes on. Community wo...
- 11 kudos
- 269 Views
- 1 replies
- 1 kudos
Cómo crear clusters en Databricks paso a paso | All-Purpose, Jobs Compute, SQL Warehouses y Pools
Recently having some fun with Databricks, I created a series of videos in Spanish that I'd like to share here. I hope some of them could be interesting for Spanish or LATAM community Not sure if this is the most proper board to share or there is ano...
- 269 Views
- 1 replies
- 1 kudos
- 1 kudos
Añadido nuevo vídeo para crear clusters de tipo serverless para notebooks, jobs y DLTs https://youtu.be/RQvkssryjyQ?si=BkYI831mUK1vBE20
- 1 kudos
- 2527 Views
- 3 replies
- 9 kudos
Building a Metadata Table-Driven Framework Using LakeFlow Declarative (Formerly DLT) Pipelines
IntroductionScaling data pipelines across an organization can be challenging, particularly when data sources, requirements, and transformation rules are always changing. A metadata table-driven framework using LakeFlow Declarative (Formerly DLT) enab...
- 2527 Views
- 3 replies
- 9 kudos
- 9 kudos
Helpful article @TejeshS . I have a question like if I want to pass parameters from my workflow to pipeline, is it possible? if yes what will be the best approach.
- 9 kudos
- 3049 Views
- 17 replies
- 29 kudos
(Episode 1: Getting Data In) - Learning Databricks one brick at a time, using the Free Edition
Episode 1: Getting Data InLearning Databricks one brick at a time, using the Free Edition.Project IntroWelcome to everyone reading. My name’s Ben, a.k.a BS_THE_ANALYST, and I’m going to share my experiences as I learn the world of Databricks. My obje...
- 3049 Views
- 17 replies
- 29 kudos
- 29 kudos
Really interesting post @BS_THE_ANALYST Caching up with Databricks stuff again
- 29 kudos
- 260 Views
- 0 replies
- 1 kudos
SQL Scripting in Apache Spark™ 4.0
The Apache Spark™ 4.0 introduces a new feature for SQL developers and data engineers: SQL Scripting. As such, this feature enhances the power and extends the flexibility of Spark SQL, enabling users to write procedural code within SQL queries, with t...
- 260 Views
- 0 replies
- 1 kudos
- 1432 Views
- 6 replies
- 14 kudos
(Episode 3: Hands-on API Project) - Learning Databricks one brick at a time, using the Free Edition
Episode 3: APIsLearning Databricks one brick at a time, using the Free Edition.Project IntroWelcome to everyone reading. My name’s Ben, a.k.a BS_THE_ANALYST, and I’m going to share my experiences as I learn the world of Databricks. My objective is to...
- 1432 Views
- 6 replies
- 14 kudos
- 14 kudos
This is great, thanks for sharing Ben, will share with my data community.
- 14 kudos
- 1223 Views
- 3 replies
- 16 kudos
(Episode 2: Reading Excel Files) - Learning Databricks one brick at a time, using the Free Edition
Episode 2: Reading Excel FilesLearning Databricks one brick at a time, using the Free Edition.You can download the accompanying Notebook and Excel files used in the demonstration over on my GitHub:Excel Files & Notebook: https://github.com/BSanalyst...
- 1223 Views
- 3 replies
- 16 kudos
