- 1317 Views
- 2 replies
- 5 kudos
I Tried Teaching Databricks About Itself — Here’s What Happened
Hi All, How are you doing today?I wanted to share something interesting from my recent Databricks work — I’ve been playing around with an idea I call “Real-Time Metadata Intelligence.” Most of us focus on optimizing data pipelines, query performance,...
- 1317 Views
- 2 replies
- 5 kudos
- 5 kudos
I like the core idea. You are mining signals the platform already emits.I would start rules first, track small files ratio and average file size trend, watch skew per partition and shuffle bytes per input gigabyte. Compare job time to input size to c...
- 5 kudos
- 297 Views
- 0 replies
- 1 kudos
Last chance to register for our LIVE Lakebase BrickTalks session!
Join us tomorrow, Thursday, Nov 13 at 9 am PT for the latest BrickTalks! We'll talk about bringing data intelligence from your Lakehouse into every app. Register now. What you’ll learn: Use Lakebase (PostgreSQL-compatible, serverless OLTP) to serve...
- 297 Views
- 0 replies
- 1 kudos
- 796 Views
- 0 replies
- 1 kudos
How Upgrading to Databricks Runtime 16.4 sped up our Python script by 10x
Wanted to share something that might save others time and money. We had a complex Databricks script that ran over 1.5 hours, when the target was under 20 minutes. Initially tried scaling up the cluster, but real progress came from simply upgrading th...
- 796 Views
- 0 replies
- 1 kudos
- 1800 Views
- 0 replies
- 1 kudos
Control Databricks Costs with AI & BI Dashboards - Video Summary
In this video, I try to showcase in a very simplified way how to enable and setup AI & BI dashboards to control costs and take actions. I hope this could be useful. I think it is a superb feature to get insights on costs while straightforward to setu...
- 1800 Views
- 0 replies
- 1 kudos
- 3662 Views
- 2 replies
- 10 kudos
Optimizing Delta Table Writes for Massive Datasets in Databricks
Problem StatementIn one of my recent projects, I faced a significant challenge: Writing a huge dataset of 11,582,763,212 rows and 2,068 columns to a Databricks managed Delta table.The initial write operation took 22.4 hours using the following setup:...
- 3662 Views
- 2 replies
- 10 kudos
- 10 kudos
Hey @Louis_Frolio ,Thank you for the thoughtful feedback and great suggestions!A few clarifications:AQE is already enabled in my setup, and it definitely helped reduce shuffle overhead during the write.Regarding Column Pruning, in this case, the fina...
- 10 kudos
- 1068 Views
- 0 replies
- 2 kudos
Another BrickTalks! Let's talk about bringing data intelligence from your Lakehouse into every app!
You asked, we delivered! Another BrickTalk is scheduled for Thursday, Nov 13 @ 9 AM PT with Pranav Aurora on how to bring data intelligence from your Lakehouse into every app and user, seamlessly and in real time. What you’ll learn: Use Lakebase (Po...
- 1068 Views
- 0 replies
- 2 kudos
- 1198 Views
- 3 replies
- 11 kudos
Community Fellows: Shout Out to our Bricksters!
At Databricks, our Community members deserve to get a great experience in our forums, with quality answers from the experts. Who better to help out our customers than Databricks employees aka Bricksters! To work towards this goal, we created the Comm...
- 1198 Views
- 3 replies
- 11 kudos
- 11 kudos
Kudos to the DB team for keeping up with the community, but can you please work on your product as well?We are experiencing a lot of issues with your paid product: failures, crashes, slow starts and slow performance and the list goes on. Community wo...
- 11 kudos
- 922 Views
- 1 replies
- 1 kudos
Cómo crear clusters en Databricks paso a paso | All-Purpose, Jobs Compute, SQL Warehouses y Pools
Recently having some fun with Databricks, I created a series of videos in Spanish that I'd like to share here. I hope some of them could be interesting for Spanish or LATAM community Not sure if this is the most proper board to share or there is ano...
- 922 Views
- 1 replies
- 1 kudos
- 1 kudos
Añadido nuevo vídeo para crear clusters de tipo serverless para notebooks, jobs y DLTs https://youtu.be/RQvkssryjyQ?si=BkYI831mUK1vBE20
- 1 kudos
- 7203 Views
- 17 replies
- 29 kudos
(Episode 1: Getting Data In) - Learning Databricks one brick at a time, using the Free Edition
Episode 1: Getting Data InLearning Databricks one brick at a time, using the Free Edition.Project IntroWelcome to everyone reading. My name’s Ben, a.k.a BS_THE_ANALYST, and I’m going to share my experiences as I learn the world of Databricks. My obje...
- 7203 Views
- 17 replies
- 29 kudos
- 29 kudos
Really interesting post @BS_THE_ANALYST Caching up with Databricks stuff again
- 29 kudos
- 907 Views
- 0 replies
- 1 kudos
SQL Scripting in Apache Spark™ 4.0
The Apache Spark™ 4.0 introduces a new feature for SQL developers and data engineers: SQL Scripting. As such, this feature enhances the power and extends the flexibility of Spark SQL, enabling users to write procedural code within SQL queries, with t...
- 907 Views
- 0 replies
- 1 kudos
- 4102 Views
- 6 replies
- 14 kudos
(Episode 3: Hands-on API Project) - Learning Databricks one brick at a time, using the Free Edition
Episode 3: APIsLearning Databricks one brick at a time, using the Free Edition.Project IntroWelcome to everyone reading. My name’s Ben, a.k.a BS_THE_ANALYST, and I’m going to share my experiences as I learn the world of Databricks. My objective is to...
- 4102 Views
- 6 replies
- 14 kudos
- 14 kudos
This is great, thanks for sharing Ben, will share with my data community.
- 14 kudos
- 4629 Views
- 3 replies
- 16 kudos
(Episode 2: Reading Excel Files) - Learning Databricks one brick at a time, using the Free Edition
Episode 2: Reading Excel FilesLearning Databricks one brick at a time, using the Free Edition.You can download the accompanying Notebook and Excel files used in the demonstration over on my GitHub:Excel Files & Notebook: https://github.com/BSanalyst...
- 4629 Views
- 3 replies
- 16 kudos
- 593 Views
- 0 replies
- 1 kudos
Migrate External Tables to Managed
With managed tables, you can reduce your storage and compute costs thanks to predictive optimization or file list caching. Now it is really time to migrate external tables to managed ones, thanks to ALTER SET MANAGED functionality. Read more: - h...
- 593 Views
- 0 replies
- 1 kudos
- 2445 Views
- 1 replies
- 4 kudos
Hadoop Walked So Databricks Could Run
Are you familiar with this scenario: Your data team spends 80% of their time fixing infrastructure issues instead of extracting insights.In today’s data-driven world, organisations are drowning in data but starving for actionable insights. Traditiona...
- 2445 Views
- 1 replies
- 4 kudos
- 4 kudos
Great one!! @Senga98 All the best!
- 4 kudos
- 18342 Views
- 6 replies
- 3 kudos
Editing value of widget parameter within notebook code
I have a notebook with a text widget where I want to be able to edit the value of the widget within the notebook and then reference it in SQL code. For example, assuming there is a text widget named Var1 that has input value "Hello", I would want to ...
- 18342 Views
- 6 replies
- 3 kudos
- 3 kudos
It seems that only way to use parameters in sql code block is to use dbutils.widget and you cannot change those parameters without removing widget and setting it up again in code
- 3 kudos
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 72 | |
| 60 | |
| 42 |
