Pool clusters and init scripts
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-24-2024 10:51 PM
Hey, just trying out pool clusters and providing the
instance_pool_type and driver_instance_pool_id configuration to the Airflow new_cluster field
I also pass the init_scripts field with an s3 link as usual but it this case of pool clusters it doesn't seem to use it.
Am Im missing something or is it not supported for pool clusters, and what other options should I look at. (custom docker images or init scripts attached to policy, etc..)
thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2024 10:27 AM
When using a non pool cluster are you able to see the init script being deployed? You could set init script logging to see if it is being called or not at all https://docs.databricks.com/en/init-scripts/logs.html

