Databricks Community

Avinash_Narala

Hi everyone,I am facing a data consistency issue in my Databricks incremental pipeline where records are being skipped because of a time gap between when a record is processed and when the physical file is finalized in Azure Blob Storage (ABFS).Our A...

Avinash_Narala

Hi everyone,I recently dealt with a frustrating scenario: a Databricks job that usually takes minutes ran for 18 hours without failing, quietly consuming compute and blocking downstream pipelines.The driver hadn't crashed, and the job hadn't failed—i...

Avinash_Narala · 01-29-2025

Hi,I want to migrate Redshift SQL Stored Procedures to databricks.As databricks doesn't support the concept of SQL Stored Procedures. How can I do so?

Avinash_Narala · 01-26-2025

Hi,I want a detailed plan steps to migrate my data from redshift to databricks.where to start, what to assess and what to migrate.It could really help me if you provide the detailed explaination on migration.Thanks in Advance.

Avinash_Narala · 01-24-2025

Hi,I want to build a python function to migrate SQL Server tables to Databricks.Is there any guide/ best practices on how to do so.It'll be really helpful if there is any.Regards,Avinash N

Avinash_Narala · 01-30-2025

i have a bunch of procedures to convert. so it is not possible to generalise the purpose of each procedure. I need a wrapper program to convert redshift sql procedures into python functions in databricks notebooks

Avinash_Narala · 01-27-2025

It looks good, but doesn't it take more time to execute , i.e, in case if we have a huge table, we need only table structure but scanning whole table for the condition 1=0?

Avinash_Narala · 01-26-2025

Is there any way to 1st create only metadata(table structure) and then migrate the data.

Avinash_Narala · 01-26-2025

Hi @dbx-user7354 ,OrderBy() should perform a global sort as showed in plot-2, but as per your problem it is sorting the data within the partitions which is the behavior of sortWithinPartitions(), so to solve this error. Please try with the latest DBR...

Avinash_Narala · 01-26-2025

Hi @SwathiChidurala ,The error is because you don't have the folder trip_zone inside the gold folder, so you can try by removing the trip_zone from the location or adding the folder trip_zone inside the gold folder in adls and then try it again.If th...

Databricks Community

User Stats

User Activity

Data Loss in Incremental Batch Jobs Due to Latency in delta file write to blob

How we solved the "18-Hour Running Job" problem with Data-Driven Timeouts

Redshift Stored Procedure Migration to Databricks

Redshift to Databricks Migration

SQL Server to Databricks Migration

Re: Redshift Stored Procedure Migration to Databricks

Re: SQL Server to Databricks Migration

Re: SQL Server to Databricks Migration

Re: Pyspark Dataframes orderby only orders within partition when having multiple worker

Re: deltaformat