- 3454 Views
- 7 replies
- 11 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 3454 Views
- 7 replies
- 11 kudos
- 11 kudos
As an additional tip for those working towards both the Associate and Professional certifications, I recommend avoiding a long gap between the two exams to maintain your momentum. If possible, try to schedule them back-to-back with just a few days in...
- 11 kudos
- 117 Views
- 1 replies
- 0 kudos
Understanding Coalesce, Skewed Joins, and Why AQE Doesn't Always Intervene
In Spark, data skew can be the silent killer of performance. One wide partition pulling in 90% of the data?But even with AQE (Adaptive Query Execution) turned on in Databricks, skewness isn't always automatically identified— and here’s why.What Is co...
- 117 Views
- 1 replies
- 0 kudos
- 0 kudos
@mark_ott , this question seems right up your alley. Care to comment?
- 0 kudos
- 9346 Views
- 3 replies
- 5 kudos
Materials to pass Databricks Data Engineering Associate Exam
Hi Guys, I have passed it already some time ago, but just recently have summarized all the materials which helped me to do it. Pay special attention to GitHub repository, which contains many great exercises prepared by Databricks teamhttps://youtu.be...
- 9346 Views
- 3 replies
- 5 kudos
- 5 kudos
I passed my Databricks Data Engineering Associate exam after studying with https://bit.ly/4iaflcm. Their extensive collection of mock tests and Practice Software significantly boosted my score to 93%.
- 5 kudos
- 103 Views
- 0 replies
- 0 kudos
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECT
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECTI got stuck with the above error when using `spark.read.table().display()` or directly query the table using %sql.While the display method is just one...
- 103 Views
- 0 replies
- 0 kudos
- 199 Views
- 0 replies
- 0 kudos
How to train a Convolutional Neural Network on Databricks with Tensorflow and Keras
Here is how to trained a lightweight Convolutional Neuronal Network (CNN) to detect pneumonia from chest X-rays pictures on Azure Databricks. I promise no LLMs, no hype, just real-world deep learning:1. Built it with TensorFlow & Keras on Databricks2...
- 199 Views
- 0 replies
- 0 kudos
- 276 Views
- 0 replies
- 0 kudos
When Did the Data Go Wrong? Using Delta Lake Time Travel for Investigation in Databricks
I. IntroductionData pipelines are the lifeblood of modern data-driven organizations. However, even the most robust pipelines can experience unexpected issues: data corruption, erroneous updates, or sudden data drops. When these problems occur, quickl...
- 276 Views
- 0 replies
- 0 kudos
- 606 Views
- 0 replies
- 1 kudos
Real Lessons in Databricks Schema, Streaming, and Unity Catalog
Hey Databricks community,I wanted to take a moment to share some things I’ve learned while working with Databricks in real projects—especially around schema management, Unity Catalog, Autoloader, and streaming jobs. These are the kinds of small detai...
- 606 Views
- 0 replies
- 1 kudos
- 457 Views
- 0 replies
- 1 kudos
Inclusion of special characters while saving or downloading as a csv
Hi All, I have data which looks like this High Corona40% 50cl Pm £13.29 but when saving it as a csv it is getting converted into High Corona40% 50cl Pm £13.29 . wherever we have the euro sign . I thing to note here is while displaying the data i...
- 457 Views
- 0 replies
- 1 kudos
- 572 Views
- 0 replies
- 1 kudos
Use Query Patterns to Suggest Indexes Dynamically
Hey folks,Ever notice how a query that used to run super fast suddenly starts dragging? We’ve all been there. As data grows, those little inefficiencies in your SQL start showing up — and they show up hard. That’s where something cool comes in: using...
- 572 Views
- 0 replies
- 1 kudos
- 3384 Views
- 6 replies
- 4 kudos
My Journey with Schema Management in Databricks
When I first started handling schema management in Databricks, I realized that a little bit of planning could save me a lot of headaches down the road. Here’s what I’ve learned and some simple tips that helped me manage schema changes effectively. On...
- 3384 Views
- 6 replies
- 4 kudos
- 4 kudos
Haha, glad it made sense! Joao.Try it out, and if you run into any issues, just let me know. Always happy to help! And best friends? You got it!
- 4 kudos
- 478 Views
- 0 replies
- 1 kudos
Unit Testing for Data Engineering: How to Ensure Production-Ready Data Pipelines
In today’s data-driven world, the success of any business use case relies heavily on trust in the data. This trust is built upon key pillars such as data accuracy, consistency, freshness, and overall quality. When organizations release data into prod...
- 478 Views
- 0 replies
- 1 kudos
- 481 Views
- 0 replies
- 0 kudos
The Future of Data Engineering: Smarter, Faster, and More Automated
Data Engineering has come a long way. From the days of manual ETL scripts to the modern world of automated, AI-driven data pipelines, the evolution has been nothing short of fascinating. As a data engineer working across various platforms, I’ve seen ...
- 481 Views
- 0 replies
- 0 kudos
- 3454 Views
- 7 replies
- 11 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 3454 Views
- 7 replies
- 11 kudos
- 11 kudos
As an additional tip for those working towards both the Associate and Professional certifications, I recommend avoiding a long gap between the two exams to maintain your momentum. If possible, try to schedule them back-to-back with just a few days in...
- 11 kudos
- 325 Views
- 0 replies
- 0 kudos
Optimizing Complex, Embedded Workflows with Databricks Cluster Pool
Managing complex, embedded workflows efficiently is a key challenge for enterprise architects. As organizations scale their data ecosystems, optimizing resource allocation becomes crucial. Databricks Cluster Pools offer a strategic solution to minimi...
- 325 Views
- 0 replies
- 0 kudos
- 5012 Views
- 2 replies
- 1 kudos
Optimizing Costs in Databricks by Dynamically Choosing Cluster Sizes
Databricks is a popular unified data analytics platform known for its powerful data processing capabilities and seamless integration with Apache Spark. However, managing and optimizing costs in Databricks can be challenging, especially when it comes ...
- 5012 Views
- 2 replies
- 1 kudos
- 1 kudos
How can this actually be used to choose a cluster pool for a Databricks workflow dynamically, that is, at run time? In other words, what can you actually do with the value of `selected_pool` other than printing it out?
- 1 kudos
- 325 Views
- 0 replies
- 1 kudos
Migrating from MySQL to Databricks: Real-time Insights That Matter
We successfully migrated a client’s MySQL databases to DB using a dual-approach that maintained 100% data integrity while enabling real-time analytics.After struggling with batch-based updates and analytics delays, we implemented:- One-time historica...
- 325 Views
- 0 replies
- 1 kudos
