Hi @lprevost,
I don't think there's a way to "checkpoint partitions" as you said.
For the gzip files, probably your executor is running out of memory during the decompression process. One of the few solutions that doesn't require changing your sour...
Yes, First of all, open source spark already has a set of auto-tuning features denominated Adaptive Query Execution (AQE). Here are more details: https://spark.apache.org/docs/latest/sql-performance-tuning.html#adaptive-query-execution.
For even bett...
Hello @pjv ,
Python code will run solely on the driver node, for this case only the driver compute matters.
If you're submitting any pyspark transformations or actions through your code it will generate spark plans that will be later executed via t...
Hello @lprevost ,
By default, Autoloader will discover and read files according to their path lexical order. It doesn't matter how nested your folder structure is. if you're interested in loading all .csv files then a common cloudFiles (Autoloader) r...
@jyothib at the current moment, system tables are still under Public Preview stage (more details at: https://docs.databricks.com/en/admin/system-tables/index.html)We don’t offer data freshness SLOs for system tables at this point and there are no pla...
