- 3510 Views
- 17 replies
- 29 kudos
(Episode 1: Getting Data In) - Learning Databricks one brick at a time, using the Free Edition
Episode 1: Getting Data InLearning Databricks one brick at a time, using the Free Edition.Project IntroWelcome to everyone reading. My name’s Ben, a.k.a BS_THE_ANALYST, and I’m going to share my experiences as I learn the world of Databricks. My obje...
- 3510 Views
- 17 replies
- 29 kudos
- 29 kudos
Really interesting post @BS_THE_ANALYST Caching up with Databricks stuff again
- 29 kudos
- 304 Views
- 0 replies
- 1 kudos
SQL Scripting in Apache Spark™ 4.0
The Apache Spark™ 4.0 introduces a new feature for SQL developers and data engineers: SQL Scripting. As such, this feature enhances the power and extends the flexibility of Spark SQL, enabling users to write procedural code within SQL queries, with t...
- 304 Views
- 0 replies
- 1 kudos
- 1642 Views
- 6 replies
- 14 kudos
(Episode 3: Hands-on API Project) - Learning Databricks one brick at a time, using the Free Edition
Episode 3: APIsLearning Databricks one brick at a time, using the Free Edition.Project IntroWelcome to everyone reading. My name’s Ben, a.k.a BS_THE_ANALYST, and I’m going to share my experiences as I learn the world of Databricks. My objective is to...
- 1642 Views
- 6 replies
- 14 kudos
- 14 kudos
This is great, thanks for sharing Ben, will share with my data community.
- 14 kudos
- 1567 Views
- 3 replies
- 16 kudos
(Episode 2: Reading Excel Files) - Learning Databricks one brick at a time, using the Free Edition
Episode 2: Reading Excel FilesLearning Databricks one brick at a time, using the Free Edition.You can download the accompanying Notebook and Excel files used in the demonstration over on my GitHub:Excel Files & Notebook: https://github.com/BSanalyst...
- 1567 Views
- 3 replies
- 16 kudos
- 368 Views
- 0 replies
- 1 kudos
Migrate External Tables to Managed
With managed tables, you can reduce your storage and compute costs thanks to predictive optimization or file list caching. Now it is really time to migrate external tables to managed ones, thanks to ALTER SET MANAGED functionality. Read more: - h...
- 368 Views
- 0 replies
- 1 kudos
- 575 Views
- 1 replies
- 4 kudos
Hadoop Walked So Databricks Could Run
Are you familiar with this scenario: Your data team spends 80% of their time fixing infrastructure issues instead of extracting insights.In today’s data-driven world, organisations are drowning in data but starving for actionable insights. Traditiona...
- 575 Views
- 1 replies
- 4 kudos
- 4 kudos
Great one!! @Senga98 All the best!
- 4 kudos
- 311 Views
- 0 replies
- 1 kudos
Validating pointer-based Delta comparison architecture using flatMapGroupsWithState in Structured St
Hi everyone,I’m leading an implementation where we’re comparing events from two real-time streams — a Source and a Target — in Databricks Structured Streaming (Scala).Our goal is to identify and emit “delta” differences between corresponding records ...
- 311 Views
- 0 replies
- 1 kudos
- 12420 Views
- 6 replies
- 3 kudos
Editing value of widget parameter within notebook code
I have a notebook with a text widget where I want to be able to edit the value of the widget within the notebook and then reference it in SQL code. For example, assuming there is a text widget named Var1 that has input value "Hello", I would want to ...
- 12420 Views
- 6 replies
- 3 kudos
- 3 kudos
It seems that only way to use parameters in sql code block is to use dbutils.widget and you cannot change those parameters without removing widget and setting it up again in code
- 3 kudos
- 4047 Views
- 6 replies
- 12 kudos
Attribute-Based Access Control (ABAC) in Databricks Unity Catalog
What Is ABAC and Why Does It Matter?Attribute-Based Access Control (ABAC) is a data governance model now available in Databricks, designed to offer fine-grained, dynamic, and scalable access control for data, AI assets, and files managed through Data...
- 4047 Views
- 6 replies
- 12 kudos
- 409 Views
- 0 replies
- 1 kudos
Databricks Policies and Bundles Inheritance: Let Policies Rule Your DABS
Just the policy_id can specify the entire cluster configuration. Yes, we can inherit default and fixed values from policies. Updating runtime version for 100s of jobs, for example, is much easier this way. Read more: - https://databrickster.med...
- 409 Views
- 0 replies
- 1 kudos
- 853 Views
- 1 replies
- 3 kudos
Relationship in databricks Genie
Now you can define relations also directly in Genie. It includes options like “Many to One”, “One to Many”, “One to One”, “Many to Many”. Read more: - https://databrickster.medium.com/relationship-in-databricks-genie-f8bf59a9b578 - https://www.su...
- 853 Views
- 1 replies
- 3 kudos
- 3 kudos
Thank you for sharing with the Community, @Hubert-Dudek!
- 3 kudos
- 967 Views
- 1 replies
- 3 kudos
Real-Time Mode in Apache Spark Structured Streaming
Real-Time Mode in Spark StreamingApache Spark™ Structured Streaming has been the backbone of mission-critical pipelines for years — from ETL to near real-time analytics and machine learning.Now, Databricks has introduced something game-changing: Real...
- 967 Views
- 1 replies
- 3 kudos
- 3 kudos
And now in Public Preview! Thank you for writing this up, @Yogesh_Verma_.
- 3 kudos
- 908 Views
- 1 replies
- 4 kudos
How Databricks Helped Me See Data Engineering Differently
Over the years working as a data engineer, I’ve started to see my role very differently. In the beginning, most of my focus was on building pipelines—extracting, transforming, and loading data so it could land in the right place. Pipelines were the g...
- 908 Views
- 1 replies
- 4 kudos
- 4 kudos
@Brahmareddy thanks for this! .Think you've nailed it on the head there. If the stakeholders trust the data and there's integrity, governance, and a single source of truth, you've got a recipe for a great product! Love this take @Brahmareddy . Really...
- 4 kudos
- 1093 Views
- 2 replies
- 6 kudos
(Teaser CHALLENGE - Community Data Pull) - Upcoming Challenge For the Community
Hey everybody, I've been dying to share this with the community. Over the last few weeks, I've been thinking about how I can do a Data Pull from the Community to highlight some of the cool stuff we all do! 拾. Below is a snippet of visual from the Dat...
- 1093 Views
- 2 replies
- 6 kudos
- 6 kudos
I'll aim to have the data for the challenge sorted and ready for next week! . I want to strip out some of the columns and figure out where is best to host the data . Potentially I could have it on the Databricks Marketplace or Github.All the best,BS
- 6 kudos
- 2760 Views
- 1 replies
- 2 kudos
9 Powerful 🚀 Spark Optimization Techniques in Databricks (With Real Examples)
IntroductionOne of our ETL pipelines used to take 10 hours to complete. After tuning and scaling in Databricks, it finished in just about 1 hour — a 90% reduction in runtime.That’s the power of Spark tuning.Databricks, built on Apache Spark, is a po...
- 2760 Views
- 1 replies
- 2 kudos
- 2 kudos
This is a fantastic breakdown of Spark optimization techniques, @savlahanish27!Definitely helpful for anyone working on performance tuning in Databricks.
- 2 kudos
