Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-09-2025 01:14 AM
hi, you can also parametrize your job clusters ??
job_clusters:
- job_cluster_key: Job_cluster
new_cluster:
spark_version: ${var.spark_version}
spark_conf: ${var.spark_configuration}
azure_attributes:
first_on_demand: 1
availability: ON_DEMAND_AZURE
spot_bid_max_price: -1
node_type_id: ${var.cluster_node_type_id}
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
LOG_LEVEL: DEBUG
BUNDLE_ROOT_DIR: ${workspace.file_path}
enable_elastic_disk: true
data_security_mode: SINGLE_USER
num_workers: ${var.cluster_worker_nodes}
instance_pool_id: ${var.executor_instance_pool_id}
driver_instance_pool_id: ${var.driver_instance_pool_id}
