Databricks Community

Nidhig631

Genie FamilyThink of the Genie family as a team of specialists. Each member has a different responsibility, but they all collaborate using the same trusted data.Article Link:Understanding the Databricks Genie Family: Which Genie Is Right for You? @mv...

Nidhig631

This is one of those Databricks concepts that becomes very easy once you stop thinking about features and start thinking about “Do I want to move the data or not?As a data engineer, think:Query Federation = “Read remotely.”Lakeflow Connect = “Bring t...

Nidhig631

Foreign Tables in Databricks are part of Lakehouse Federation and allow you to query data that lives outside Databricks without copying it into Delta tables first. Think of them as a Unity Catalog object that points to an external system.Let’s unders...

Nidhig631

As someone who spends hours every day building notebooks, debugging pipelines, and experimenting with AI agents in Databricks, there was always one thought in the back of my mind.“Why can’t Genie Code have a full-screen workspace where I can manage e...

Nidhig631 · 06-20-2026

Need some advice from the community.I am processing around 100 million records using:df.select(required_cols).distinct().write.saveAsTable(...)The source has 1000+ columns, but I'm selecting only 20 columns before applying DISTINCT.I have already ena...

Nidhig631

This is included with the Databricks managed table; there is no separate catalog managed table.

Nidhig631

From my analysis, the write phase is completing relatively quickly. The primary performance issue seems to be the DISTINCT operation, which is taking a significant amount of time due to the shuffle and deduplication overhead.

Nidhig631

Thanks for your response. However, I found it to be quite generic and more conversational than solution-oriented. I was expecting recommendations tailored to my specific problem statement, along with actionable steps or technical approaches to addres...

Nidhig631

I have already checked spark UI there is no data skewness, for such data load shuffle partition 400 is best.The third suggestion I will check but cost may increase in this.

Nidhig631

The data is overwrite mode anyways, just one day data.Distinct working is same as generating hashing key at the background for my use case.I have already checked on the solution suggested by you.

Databricks Community

User Stats

User Activity

Understanding the Databricks Genie Family: Which Genie Is Right for You?

Query Federation vs Lakeflow Connect in Databricks: When to Query and When to Ingest

Databricks Foreign Table

Databricks Genie Code Gets a Full Page Command Centre

DISTINCT is the major bottleneck because of the heavy shuffle.

Re: Catalog Managed Tables

Re: DISTINCT is the major bottleneck because of the heavy shuffle.

Re: DISTINCT is the major bottleneck because of the heavy shuffle.

Re: DISTINCT is the major bottleneck because of the heavy shuffle.

Re: DISTINCT is the major bottleneck because of the heavy shuffle.