Databricks Community

Brahmareddy

Hey Databricks community,I wanted to take a moment to share some things I’ve learned while working with Databricks in real projects—especially around schema management, Unity Catalog, Autoloader, and streaming jobs. These are the kinds of small detai...

Brahmareddy

Hey folks,Ever notice how a query that used to run super fast suddenly starts dragging? We’ve all been there. As data grows, those little inefficiencies in your SQL start showing up — and they show up hard. That’s where something cool comes in: using...

Brahmareddy

Hey Data Engineers!  How are you all doing? Hope you're having a great time working with data!Let me share a quick story. A while ago, my team had a problem—our data pipeline was taking 4+ hours to run. We tried different things, but nothing worked....

Brahmareddy

Data Engineering has come a long way. From the days of manual ETL scripts to the modern world of automated, AI-driven data pipelines, the evolution has been nothing short of fascinating. As a data engineer working across various platforms, I’ve seen ...

Brahmareddy · 02-17-2025

Hello Austin Databricks Community,Hope this article finds you well.Ever spent hours tweaking partitions, only to find out your queries are still slow? Yeah, we’ve all been there. But guess what? Liquid Clustering in Delta Lake 3.0 makes all that manu...

Brahmareddy

Hi SamAdams,How are you doing today?, As per my understanding, You're on the right track here! When you change the S3 path for Autoloader, even if the files are exactly the same and just copied from the old bucket, Autoloader will treat them as new f...

Brahmareddy

Hi Dharanip,From what you’ve described, your logic looks good, but the issue comes from how Materialized Views work in Databricks. They only support incremental updates when the query is very simple and fully predictable (or "deterministic"). Since y...

Brahmareddy

Hi mydefaultlogin,How are you doing today?, As per my understanding, You're right—this happens because when you're running notebooks from your Git folder, Python knows exactly where your project root is and can easily find my_package. But when you de...

Brahmareddy

Hi Dharinip,How are you doing today? As per my understanding, it looks like your Materialized View is falling back to full recompute because the source data or query isn't eligible for incremental updates. Based on the message, there are a few blocke...

Brahmareddy

Hi hdu,How are you doing today?, As per my understanding, It sounds like you’re really close! That “No API found” error usually means either the wrong API endpoint is being used, or the cluster type doesn’t support ownership changes—for example, shar...

Databricks Community

User Stats

User Activity

Real Lessons in Databricks Schema, Streaming, and Unity Catalog

Use Query Patterns to Suggest Indexes Dynamically

Data Engineering: A Team Sport for Success!

The Future of Data Engineering: Smarter, Faster, and More Automated

Liquid Clustering: No More Partition Headaches!

Re: Migrating source directory in an existing DLT Pipeline with Autoloader

Re: Incrementalization issue in Materialized views

Re: Inconsistent PYTHONPATH, Git folders vs DAB

Re: Incrementalization issue in Materialized views

Re: Change cluster owner API call failed