- 160 Views
- 1 replies
- 0 kudos
Understanding Coalesce, Skewed Joins, and Why AQE Doesn't Always Intervene
In Spark, data skew can be the silent killer of performance. One wide partition pulling in 90% of the data?But even with AQE (Adaptive Query Execution) turned on in Databricks, skewness isn't always automatically identified— and here’s why.What Is co...
- 160 Views
- 1 replies
- 0 kudos
- 0 kudos
@mark_ott , this question seems right up your alley. Care to comment?
- 0 kudos
- 217 Views
- 0 replies
- 0 kudos
How to train a Convolutional Neural Network on Databricks with Tensorflow and Keras
Here is how to trained a lightweight Convolutional Neuronal Network (CNN) to detect pneumonia from chest X-rays pictures on Azure Databricks. I promise no LLMs, no hype, just real-world deep learning:1. Built it with TensorFlow & Keras on Databricks2...
- 217 Views
- 0 replies
- 0 kudos
- 501 Views
- 0 replies
- 0 kudos
The Future of Data Engineering: Smarter, Faster, and More Automated
Data Engineering has come a long way. From the days of manual ETL scripts to the modern world of automated, AI-driven data pipelines, the evolution has been nothing short of fascinating. As a data engineer working across various platforms, I’ve seen ...
- 501 Views
- 0 replies
- 0 kudos