- 2457 Views
- 2 replies
- 1 kudos
Automatic Liquid Clustering and PO
I spent some time to understand how to use automatic liquid clustering with dlt pipelines. Hope this can help you as well.Enable Predictive Optimization Use this code:# Enabling Automatic Liquid Clustering on a new table @dlt.table(cluster_by_auto=Tr...
- 2457 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Addy0_, thanks for sharing how to set it for existing table. Unfortunately, I think ALTER cannot be used with materialized view and streaming tables defined in dlt pipelines.I was looking for something similar to @dlt.table(cluster_by_auto=True, ...
- 1 kudos
- 1334 Views
- 0 replies
- 1 kudos
Databricks Data Classification
I encourage you to try out a new beta feature in Databricks called : Data Classification. It automatically classifies your catalog data and tag it with tags. Docs: https://docs.databricks.com/aws/en/lakehouse-monitoring/data-classification
- 1334 Views
- 0 replies
- 1 kudos
- 1226 Views
- 0 replies
- 1 kudos
Strong Databricks Fundamental - Gen Z
Why Databricks is the Future of Data Analytics for Gen ZIn the fast-paced world of data analytics, staying ahead of the curve is crucial. For Gen Z, who are digital natives and always on the lookout for the latest tech trends, understanding the diffe...
- 1226 Views
- 0 replies
- 1 kudos
- 4381 Views
- 1 replies
- 1 kudos
Post: Lakehouse Federation - Databricks
Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancement...
- 4381 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey Quick Question, Can we use it for the production version ? We have application server as SQL server, we are planning to use lakehouse federation so we can bypass creating and maintaining 100 of workflows. as we a small dataset I am not too sure o...
- 1 kudos
- 1396 Views
- 0 replies
- 1 kudos
Hub Star Modeling 2.0 for Medalion Architecture
Excited to share my latest publication on arXiv!“Hub Star Modeling 2.0 for Medallion Architecture” https://arxiv.org/abs/2504.08788This new version builds on the original Hub Star Modeling approach, published last year, and now tailored for the Meda...
- 1396 Views
- 0 replies
- 1 kudos
- 5495 Views
- 1 replies
- 6 kudos
Handling Complex Nested JSON in Databricks Using schemaHints
When I first got into managing schemas in Databricks, it took me a while to realize that putting in a little planning up front could save me a ton of headaches later on.I was working with these deeply nested, constantly changing JSON files. At first,...
- 5495 Views
- 1 replies
- 6 kudos
- 6 kudos
Great tip @genevive_mdonça! schemaHints help avoid issues with evolving JSON data, making data processing more reliable and easier to maintain. Thanks for sharing.
- 6 kudos
- 3999 Views
- 1 replies
- 0 kudos
Understanding Coalesce, Skewed Joins, and Why AQE Doesn't Always Intervene
In Spark, data skew can be the silent killer of performance. One wide partition pulling in 90% of the data?But even with AQE (Adaptive Query Execution) turned on in Databricks, skewness isn't always automatically identified— and here’s why.What Is co...
- 3999 Views
- 1 replies
- 0 kudos
- 0 kudos
@mark_ott , this question seems right up your alley. Care to comment?
- 0 kudos
- 2980 Views
- 0 replies
- 1 kudos
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECT
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECTI got stuck with the above error when using `spark.read.table().display()` or directly query the table using %sql.While the display method is just one...
- 2980 Views
- 0 replies
- 1 kudos
- 3345 Views
- 0 replies
- 1 kudos
Power BI to Databricks Semantic Layer Generator (DAX → SQL/PySpark)
Hi everyone!I’ve just released an open-source tool that generates a semantic layer in Databricks notebooks from a Power BI dataset using the Power BI REST API. Im not an expert yet, but it gets job done and instead of using AtScale/dbt/or the PBI Sem...
- 3345 Views
- 0 replies
- 1 kudos
- 1083 Views
- 0 replies
- 0 kudos
How to train a Convolutional Neural Network on Databricks with Tensorflow and Keras
Here is how to trained a lightweight Convolutional Neuronal Network (CNN) to detect pneumonia from chest X-rays pictures on Azure Databricks. I promise no LLMs, no hype, just real-world deep learning:1. Built it with TensorFlow & Keras on Databricks2...
- 1083 Views
- 0 replies
- 0 kudos
- 2950 Views
- 0 replies
- 1 kudos
Real Lessons in Databricks Schema, Streaming, and Unity Catalog
Hey Databricks community,I wanted to take a moment to share some things I’ve learned while working with Databricks in real projects—especially around schema management, Unity Catalog, Autoloader, and streaming jobs. These are the kinds of small detai...
- 2950 Views
- 0 replies
- 1 kudos
- 1299 Views
- 0 replies
- 1 kudos
Inclusion of special characters while saving or downloading as a csv
Hi All, I have data which looks like this High Corona40% 50cl Pm £13.29 but when saving it as a csv it is getting converted into High Corona40% 50cl Pm £13.29 . wherever we have the euro sign . I thing to note here is while displaying the data i...
- 1299 Views
- 0 replies
- 1 kudos
- 2171 Views
- 0 replies
- 1 kudos
Use Query Patterns to Suggest Indexes Dynamically
Hey folks,Ever notice how a query that used to run super fast suddenly starts dragging? We’ve all been there. As data grows, those little inefficiencies in your SQL start showing up — and they show up hard. That’s where something cool comes in: using...
- 2171 Views
- 0 replies
- 1 kudos
- 3765 Views
- 0 replies
- 1 kudos
Unit Testing for Data Engineering: How to Ensure Production-Ready Data Pipelines
In today’s data-driven world, the success of any business use case relies heavily on trust in the data. This trust is built upon key pillars such as data accuracy, consistency, freshness, and overall quality. When organizations release data into prod...
- 3765 Views
- 0 replies
- 1 kudos
- 7741 Views
- 0 replies
- 0 kudos
The Future of Data Engineering: Smarter, Faster, and More Automated
Data Engineering has come a long way. From the days of manual ETL scripts to the modern world of automated, AI-driven data pipelines, the evolution has been nothing short of fascinating. As a data engineer working across various platforms, I’ve seen ...
- 7741 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 72 | |
| 60 | |
| 43 |
