Ability to submit a single job with cluster pools and init scriptsfor the following payload: {
"run_name": "A multitask job run",
"timeout_seconds": 86400,
"tasks": [
{
"task_key": "task_1",
"depends_on": ...
Hey, just trying out pool clusters and providing the instance_pool_type and driver_instance_pool_id configuration to the Airflow new_cluster fieldI also pass the init_scripts field with an s3 link as usual but it this case of pool clusters it doesn't...
Does anyone else experiencing lags on the Databricks notebook editor?The typing is sometimes very slow causes the cell to hang for a moment. (Using python)
I use the following option to write from multiple tasks to the same table with overwrite (in Pyspark).option("partitionOverwriteMode", "dynamic")The table was created with partition by so it works as expected.I read about liquid clustering and it's b...
