Databricks Community

Nidhig631

As someone who spends hours every day building notebooks, debugging pipelines, and experimenting with AI agents in Databricks, there was always one thought in the back of my mind.“Why can’t Genie Code have a full-screen workspace where I can manage e...

Nidhig631

Need some advice from the community.I am processing around 100 million records using:df.select(required_cols).distinct().write.saveAsTable(...)The source has 1000+ columns, but I'm selecting only 20 columns before applying DISTINCT.I have already ena...

Nidhig631

Databricks LakeWatch- Security for the Agentic Era:What is Lakewatch?Lakewatch is a new security product, essentially a comprehensive security camera system for your entire company’s computer systems, powered by AI.Two significant developments occurr...

Nidhig631

Databricks Genie vs Databricks Genie Code – What's the Difference?Many people assume Genie and Genie Code are the same, but they are designed for completely different audiences and use cases.GenieWho it’s for: All employees, from Marketing Managers t...

Nidhig631

You can now embed a Databricks Genie Space directly into an external website or application.What does this mean?Instead of asking your business users to log into Databricks every time they want to query data, you can bring Genie to them right inside ...

Nidhig631

This is included with the Databricks managed table; there is no separate catalog managed table.

Nidhig631

From my analysis, the write phase is completing relatively quickly. The primary performance issue seems to be the DISTINCT operation, which is taking a significant amount of time due to the shuffle and deduplication overhead.

Nidhig631

Thanks for your response. However, I found it to be quite generic and more conversational than solution-oriented. I was expecting recommendations tailored to my specific problem statement, along with actionable steps or technical approaches to addres...

Nidhig631

I have already checked spark UI there is no data skewness, for such data load shuffle partition 400 is best.The third suggestion I will check but cost may increase in this.

Nidhig631

The data is overwrite mode anyways, just one day data.Distinct working is same as generating hashing key at the background for my use case.I have already checked on the solution suggested by you.

Databricks Community

User Stats

User Activity

Databricks Genie Code Gets a Full Page Command Centre

DISTINCT is the major bottleneck because of the heavy shuffle.

Claude Mythos & Databricks LakeWatch

Databricks Genie vs Databricks Genie Code

Embed a Databricks Genie space in an External Website or Application

Re: Catalog Managed Tables

Re: DISTINCT is the major bottleneck because of the heavy shuffle.

Re: DISTINCT is the major bottleneck because of the heavy shuffle.

Re: DISTINCT is the major bottleneck because of the heavy shuffle.

Re: DISTINCT is the major bottleneck because of the heavy shuffle.