Databricks Community

yeungcase · 07-03-2024

Hi All,I am currently trying to read data from a materialized view as a single dataframe which contains around 10M of rows and then write it into an Azure SQL database. However, I don't see the spark job moving a bit even an hour is passed. I have al...

yeungcase · 06-30-2024

Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

yeungcase · 07-03-2024

Hi @dbrx_user ,I've tried to rewrite the statement, but it does not help. However, I discover that it could successfully write 100k rows without having any issues. But once I further boost it to 1M rows, it just stuck there.

yeungcase · 07-03-2024

Hi @dbrx_user ,Thank you for your suggestion. Actually, I also notice that I should use repartition instead. However, I am now struggling on the optimal repartition number. I've tried make it as 16 but it seems the spill of memory and disk issue is v...

yeungcase · 07-01-2024

Hi @Retired_mod, highly appreciate for your help. May I know whether there is any official document explaining the details of RETRY_ON_FAILURE?

yeungcase · 06-25-2024

@Retired_mod , could you please elaborate more on how to "allow other tables to continue processing even if one table encounters an error"?

Databricks Community

User Stats

User Activity

Writing a single huge dataframe into Azure SQL Database using JDBC

Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

Re: Writing a single huge dataframe into Azure SQL Database using JDBC

Re: Writing a single huge dataframe into Azure SQL Database using JDBC

Re: Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

Re: DLT Pipeline Error Handling