@StephanieRivera
Auto Loader is primarily used for processing files automatically as they arrive in cloud storage.
You can refer to this document on AutoLoader - https://docs.databricks.com/en/ingestion/auto-loader/index.html
@Humi1245
You can view the event log of the cluster to understand the cause of the cluster termination. If it is due to gc issues, then we have to understand the reason behind high driver memory utilization which caused the GC.
If it's due to inact...
Hi @Gilg
You mentioned that micro-batch time is around 12 minutes recently. Do we also see jobs/stages with 12 minutes in the spark ui. If that is the case, then the processing of the file itself takes 12 minutes. If not, the 12 minutes is spent on ...
@Kaz
You can install these libraries using the Libraries section in the Compute.
All of the libraries mentioned here would be installed whenever the cluster is spun up.
