cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Pool clusters and init scripts

radix
New Contributor II

Hey, just trying out pool clusters and providing the 
instance_pool_type and driver_instance_pool_id configuration to the Airflow new_cluster field
I also pass the init_scripts field with an s3 link as usual but it this case of pool clusters it doesn't seem to use it.

Am Im missing something or is it not supported for pool clusters, and what other options should I look at. (custom docker images or init scripts attached to policy, etc..)

thanks

1 REPLY 1

Walter_C
Databricks Employee
Databricks Employee

When using a non pool cluster are you able to see the init script being deployed? You could set init script logging to see if it is being called or not at all https://docs.databricks.com/en/init-scripts/logs.html 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group