cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks set up in Prod environment

atul1146
New Contributor III

Hi! can anyone please help me with a documentation which can help me set up integration between data bricks with AWS without a QuickStart default cloud formation template.

I would want to use my own CFT rather than using the default due to security compliance.

1 ACCEPTED SOLUTION

Accepted Solutions

Pat
Honored Contributor III

Hi @Atul S​ ,

I think that terraform is recommended way to go with Databricks deployment. I mean it's also supported now by the Databricks support.

I haven't look much on the CloudFormation setup, because we decided to go with the Terraform in the company (CF was our IaC tool).

Here is some info about the IaC (tf, cdk).

https://docs.databricks.com/dev-tools/index-iac.html

I've checked this before:

https://aws-quickstart.github.io/quickstart-databricks-unified-data-analytics-platform/

but it's 2020, and now we have all those new things like Unity Catalog. TBH - terrafrom is seems to be way forward.

https://docs.databricks.com/administration-guide/account-api/templates.html :

imageDatabricks Terrafrom provider is not ideal, but it's adapting quickly to the newest Databricks API changes very quick.

There are some gotchas, like creating Unity Catalog tables (delta), that should be mentioned in the documentation in my opinion - not production ready or something like this.

I've noticed also lately more and more question in the forum about databricks/terraform. It looks like we are building here nice community :).

thanks,

Patryk.

View solution in original post

2 REPLIES 2

Pat
Honored Contributor III

Hi @Atul S​ ,

I think that terraform is recommended way to go with Databricks deployment. I mean it's also supported now by the Databricks support.

I haven't look much on the CloudFormation setup, because we decided to go with the Terraform in the company (CF was our IaC tool).

Here is some info about the IaC (tf, cdk).

https://docs.databricks.com/dev-tools/index-iac.html

I've checked this before:

https://aws-quickstart.github.io/quickstart-databricks-unified-data-analytics-platform/

but it's 2020, and now we have all those new things like Unity Catalog. TBH - terrafrom is seems to be way forward.

https://docs.databricks.com/administration-guide/account-api/templates.html :

imageDatabricks Terrafrom provider is not ideal, but it's adapting quickly to the newest Databricks API changes very quick.

There are some gotchas, like creating Unity Catalog tables (delta), that should be mentioned in the documentation in my opinion - not production ready or something like this.

I've noticed also lately more and more question in the forum about databricks/terraform. It looks like we are building here nice community :).

thanks,

Patryk.

atul1146
New Contributor III

Thank you @Pat Sienkiewicz​ for the information.... Appreciated.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.