Databricks Community

hi @Shalabh Agarwal​ I am not able to find any official practice paper. it is still not available.

You can write your own code to fetch data from external API.Example: https://insightsndata.com/how-to-call-rest-api-store-data-in-databricks-8383f2458d7d

@Himanshi Patle​ in spark streaming there is one option maxFileAge using which you can control which files to process based on their timestamp.

@Harun Raseed Basheer​ usually for 1 worker node 1 executor will be there, if we need to split that executor within that worker node itself, we can do that based on memory that core has been assigned , below configs can be use spark.executor.coresspa...

Hi @Adam Rink​ Just checking for further info on your question. How are you deducing that the batch sizes are more than what you are providing as maxOffsetsPerTrigger ?

In Azure, you can use Cost Management to track your expenses incurred by Databricks instance.

You can try optimize and cache with the internal tables such as schema tables to fetch updated information.

I case of struct you can use (.) For extracting the value

I think you can use spark UI to see deep level details ​

You can restart the cluster and check once.​

Overwritte ​option will overwritte your data. If you want to change column name then you can first alter the delta table as per your need then you can append new data as well. So both problems you can resolve

Then can use.​