Databricks Community

Avinash_Narala

Hello Databricks Community!I recently published a detailed breakdown on Medium about a real-world optimization nightmare we faced, and I wanted to share the core lessons learned with this group.We had a highly efficient Delta table pipeline handling ...

Avinash_Narala · 06-04-2026

Hey everyone,Have you ever opened Databricks Catalog Explorer to audit a table, only to find the downstream job listed as "Untitled"? Databricks Unity Catalog is incredibly powerful for automated lineage, but it quietly breaks the moment you orchestr...

Avinash_Narala · 06-03-2026

Hey everyone,We’ve all been there: a Delta Lake MERGE job that should take 20 minutes drags on for 90 minutes, while a full overwrite of the same table finishes in under 20. When an overwrite outpaces a selective merge, it's a massive red flag that y...

Avinash_Narala · 05-20-2026

Why Your Delta Lake Tables Are Quietly Ballooning (And How to Fix It)If your data pipeline only appends a few gigabytes a day, but your cloud storage footprint is skyrocketing into hundreds of gigabytes, you aren’t alone. We recently watched one of o...

Avinash_Narala · 05-06-2026

Hi everyone,I recently took a look into a silent cost driver in many data platforms: the default choice between managed and external tables in Unity Catalog.It is very common for teams to default to external tables, but this choice often leads to acc...

Avinash_Narala · 01-30-2025

i have a bunch of procedures to convert. so it is not possible to generalise the purpose of each procedure. I need a wrapper program to convert redshift sql procedures into python functions in databricks notebooks

Avinash_Narala · 01-27-2025

It looks good, but doesn't it take more time to execute , i.e, in case if we have a huge table, we need only table structure but scanning whole table for the condition 1=0?

Avinash_Narala · 01-26-2025

Is there any way to 1st create only metadata(table structure) and then migrate the data.

Avinash_Narala · 01-26-2025

Hi @dbx-user7354 ,OrderBy() should perform a global sort as showed in plot-2, but as per your problem it is sorting the data within the partitions which is the behavior of sortWithinPartitions(), so to solve this error. Please try with the latest DBR...

Avinash_Narala · 01-26-2025

Hi @SwathiChidurala ,The error is because you don't have the folder trip_zone inside the gold folder, so you can try by removing the trip_zone from the location or adding the folder trip_zone inside the gold folder in adls and then try it again.If th...

Databricks Community

User Stats

User Activity

How a Partitioning Mistake Turned a 12-Minute Databricks Job Into a 2-Hour Nightmare

Solving the "Untitled" Lineage Mystery in Unity Catalog

Why Your Delta MERGE is 5x Slower Than an Overwrite (And How to Fix It)

From 400GB to 35GB: Managing Delta Lake Storage Growth

Managed vs External Tables in Unity Catalog: The Decision That’s Silently Inflating Your Cloud Bill

Re: Redshift Stored Procedure Migration to Databricks

Re: SQL Server to Databricks Migration

Re: SQL Server to Databricks Migration

Re: Pyspark Dataframes orderby only orders within partition when having multiple worker

Re: deltaformat