Databricks Community

dikokob

We are experiencing the following issues.Description:I encountered an issue while executing a Spark SQL query in Databricks, and it seems to be related to the query optimization phase. The error message suggests an internal bug within Spark or the Sp...

dikokob

Hello Databricks Community,I'm encountering an issue with the Databricks Autoloader where, after running successfully for a period of time, it suddenly stops detecting new files in the source directory. This issue only gets resolved when I reset the ...

dikokob

@daniel_sahal,Any suggestion on a backfillinterval, that could work best for my use case?I will also look into the .awaitTermination() and test that out.I am mainly looking for a solution that will increase our ingestion's robustness.

dikokob

Hi @daniel_sahal ,Thanks for the response:cloudFiles.maxFileAge - isn't setcloudFiles.backfillInterval - Will it still be worth using it even though I am not using file notification mode. Also, I have autoloader running for a live system which runs o...

Databricks Community

User Stats

User Activity

Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime

Databricks Autoloader Checkpoint

Re: Databricks Autoloader Checkpoint

Re: Databricks Autoloader Checkpoint