- 5589 Views
- 1 replies
- 6 kudos
Handling Complex Nested JSON in Databricks Using schemaHints
When I first got into managing schemas in Databricks, it took me a while to realize that putting in a little planning up front could save me a ton of headaches later on.I was working with these deeply nested, constantly changing JSON files. At first,...
- 5589 Views
- 1 replies
- 6 kudos
- 6 kudos
Great tip @genevive_mdonça! schemaHints help avoid issues with evolving JSON data, making data processing more reliable and easier to maintain. Thanks for sharing.
- 6 kudos
- 4044 Views
- 1 replies
- 0 kudos
Understanding Coalesce, Skewed Joins, and Why AQE Doesn't Always Intervene
In Spark, data skew can be the silent killer of performance. One wide partition pulling in 90% of the data?But even with AQE (Adaptive Query Execution) turned on in Databricks, skewness isn't always automatically identified— and here’s why.What Is co...
- 4044 Views
- 1 replies
- 0 kudos
- 0 kudos
@mark_ott , this question seems right up your alley. Care to comment?
- 0 kudos
- 3017 Views
- 0 replies
- 1 kudos
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECT
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECTI got stuck with the above error when using `spark.read.table().display()` or directly query the table using %sql.While the display method is just one...
- 3017 Views
- 0 replies
- 1 kudos
- 3388 Views
- 0 replies
- 1 kudos
Power BI to Databricks Semantic Layer Generator (DAX → SQL/PySpark)
Hi everyone!I’ve just released an open-source tool that generates a semantic layer in Databricks notebooks from a Power BI dataset using the Power BI REST API. Im not an expert yet, but it gets job done and instead of using AtScale/dbt/or the PBI Sem...
- 3388 Views
- 0 replies
- 1 kudos
- 1099 Views
- 0 replies
- 0 kudos
How to train a Convolutional Neural Network on Databricks with Tensorflow and Keras
Here is how to trained a lightweight Convolutional Neuronal Network (CNN) to detect pneumonia from chest X-rays pictures on Azure Databricks. I promise no LLMs, no hype, just real-world deep learning:1. Built it with TensorFlow & Keras on Databricks2...
- 1099 Views
- 0 replies
- 0 kudos
- 2963 Views
- 0 replies
- 1 kudos
Real Lessons in Databricks Schema, Streaming, and Unity Catalog
Hey Databricks community,I wanted to take a moment to share some things I’ve learned while working with Databricks in real projects—especially around schema management, Unity Catalog, Autoloader, and streaming jobs. These are the kinds of small detai...
- 2963 Views
- 0 replies
- 1 kudos
- 1316 Views
- 0 replies
- 1 kudos
Inclusion of special characters while saving or downloading as a csv
Hi All, I have data which looks like this High Corona40% 50cl Pm £13.29 but when saving it as a csv it is getting converted into High Corona40% 50cl Pm £13.29 . wherever we have the euro sign . I thing to note here is while displaying the data i...
- 1316 Views
- 0 replies
- 1 kudos
- 2190 Views
- 0 replies
- 1 kudos
Use Query Patterns to Suggest Indexes Dynamically
Hey folks,Ever notice how a query that used to run super fast suddenly starts dragging? We’ve all been there. As data grows, those little inefficiencies in your SQL start showing up — and they show up hard. That’s where something cool comes in: using...
- 2190 Views
- 0 replies
- 1 kudos
- 3822 Views
- 0 replies
- 1 kudos
Unit Testing for Data Engineering: How to Ensure Production-Ready Data Pipelines
In today’s data-driven world, the success of any business use case relies heavily on trust in the data. This trust is built upon key pillars such as data accuracy, consistency, freshness, and overall quality. When organizations release data into prod...
- 3822 Views
- 0 replies
- 1 kudos
- 7801 Views
- 0 replies
- 0 kudos
The Future of Data Engineering: Smarter, Faster, and More Automated
Data Engineering has come a long way. From the days of manual ETL scripts to the modern world of automated, AI-driven data pipelines, the evolution has been nothing short of fascinating. As a data engineer working across various platforms, I’ve seen ...
- 7801 Views
- 0 replies
- 0 kudos
- 1505 Views
- 0 replies
- 0 kudos
Optimizing Complex, Embedded Workflows with Databricks Cluster Pool
Managing complex, embedded workflows efficiently is a key challenge for enterprise architects. As organizations scale their data ecosystems, optimizing resource allocation becomes crucial. Databricks Cluster Pools offer a strategic solution to minimi...
- 1505 Views
- 0 replies
- 0 kudos
- 2065 Views
- 0 replies
- 1 kudos
Migrating from MySQL to Databricks: Real-time Insights That Matter
We successfully migrated a client’s MySQL databases to DB using a dual-approach that maintained 100% data integrity while enabling real-time analytics.After struggling with batch-based updates and analytics delays, we implemented:- One-time historica...
- 2065 Views
- 0 replies
- 1 kudos
- 10747 Views
- 8 replies
- 6 kudos
Library Management via Custom Compute Policies and ADF Job Triggering
This guide is intended for those looking to install libraries on a cluster using a Custom Compute Policy and trigger Databricks jobs from an Azure Data Factory (ADF) linked service. While many users rely on init scripts for library installation, it i...
- 10747 Views
- 8 replies
- 6 kudos
- 6 kudos
Hi @hassan2 I had same issue and found solution.When I created POOL i created it as On-demand (not spot) and then policy only worked when I removed entire section "azure_attributes.spot_bid_max_price" from policy.Looks like "azure_attributes.spot_bi...
- 6 kudos
- 5155 Views
- 1 replies
- 1 kudos
Resolved! Log Custom Transformer with Feature Engineering Client
Hi everyone,I'm building a Pyspark ML Pipeline where the first stage is to fill nulls with zero. I wrote a custom class to do this since I cannot find a Transformer that will do this imputation. I am able to log this pipeline using ML Flow log model ...
- 5155 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @WarrenO , thanks for sharing that with the detailed code! I was able to reproduce the error, specifically the following error: AttributeError: module '__main__' has no attribute 'CustomAdder'File <command-1315887242804075>, line 3935 evaluator = ...
- 1 kudos
- 1170 Views
- 0 replies
- 0 kudos
The Hidden Security Risks in Stored Procedure Migrations—What Databricks Exposed
Your stored procedure migration to DB isn't just a 'copy-paste' job - it's a security nightmare waiting to happen.We discovered our 'trusted' stored procedures had hidden access patterns that nearly compromised our entire data governance model. Here'...
- 1170 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 73 | |
| 60 | |
| 43 |
