cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Pool clusters and init scripts

radix
New Contributor II

Hey, just trying out pool clusters and providing the 
instance_pool_type and driver_instance_pool_id configuration to the Airflow new_cluster field
I also pass the init_scripts field with an s3 link as usual but it this case of pool clusters it doesn't seem to use it.

Am Im missing something or is it not supported for pool clusters, and what other options should I look at. (custom docker images or init scripts attached to policy, etc..)

thanks

1 REPLY 1

Walter_C
Databricks Employee
Databricks Employee

When using a non pool cluster are you able to see the init script being deployed? You could set init script logging to see if it is being called or not at all https://docs.databricks.com/en/init-scripts/logs.html