Integrate databrick cluster with aws using cloudformation

[Pratiksha1]

Hello community, I am trying to launch databrick cluster using AWS  cloudformation template. I have checked public extension option in cloudformation and databrick cluster extension is also available in third party extension however not getting how to map configuration. we have to provide configuration JSON to configure this extension. need help in "configuration json" file.

[Kaniz_Fatma]

Hi @Pratiksha1, When configuring a Databricks cluster using AWS CloudFormation, you can provide a JSON configuration to customize the cluster settings. 

Let’s break down the steps:

Cluster Configuration in JSON:

• The JSON configuration allows you to specify various parameters for your Databricks cluster. These parameters include things like instance types, autoscaling rules, libraries, and other cluster-specific settings.
• You can view the existing cluster configuration as JSON within the Databricks UI. Here’s how:
  1. Go to the Configuration tab of the cluster details page.
  2. Click JSON in the top right corner of the tab.
  3. Copy the JSON representation of the cluster configuration.
  4. You can then use this JSON when creating similar clusters programmatically via the Clusters API.

Editing Cluster Configuration:

• If you need to modify an existing cluster, you can edit its configuration. Keep in mind the following points:
  • Notebooks and jobs attached to the cluster remain attached even after editing.
  • Libraries installed on the cluster remain installed.
  • If you edit any attribute of a running cluster (except for size and permissions), you must restart it. This may disrupt users currently using the cluster.
  • You can only edit running or terminated clusters.
  • Permissions can be updated for clusters in other states on the cluster details page.

Cluster Types:

• Databricks offers two types of clusters:
  • All-purpose clusters: These can be shared by multiple users and are suitable for ad-hoc analysis, data exploration, or development.
  • Job clusters: These terminate when the job ends, reducing resource usage and cost.

Tradeoff Between Cost and Performance:

• When configuring your cluster, consider the tradeoff between cost and performance. Factors include Databricks Units (DBUs) consumed and the underlying resource costs.
• Secondary costs (such as SLA impact or resource waste) should also be considered.

Cluster Access Control:

• Fine-grained cluster access control allows admins to manage who can create clusters.
• There are two types of access control:
  • Cluster-creation permission: Admins can choose which users are allowed to create clusters.

Remember that your configuration decisions should align with your specific workload requirements, user types, and budget constraints. Feel free to explore the Databricks documentation for more detailed information on best practices an....