Databricks set up in Prod environment

[atul1146]

Hi! can anyone please help me with a documentation which can help me set up integration between data bricks with AWS without a QuickStart default cloud formation template.

I would want to use my own CFT rather than using the default due to security compliance.

[Pat · Accepted solution]

Hi @Atul S​ ,

I think that terraform is recommended way to go with Databricks deployment. I mean it's also supported now by the Databricks support.

I haven't look much on the CloudFormation setup, because we decided to go with the Terraform in the company (CF was our IaC tool).

Here is some info about the IaC (tf, cdk).

https://docs.databricks.com/dev-tools/index-iac.html

I've checked this before:

https://aws-quickstart.github.io/quickstart-databricks-unified-data-analytics-platform/

but it's 2020, and now we have all those new things like Unity Catalog. TBH - terrafrom is seems to be way forward.

https://docs.databricks.com/administration-guide/account-api/templates.html :

Databricks Terrafrom provider is not ideal, but it's adapting quickly to the newest Databricks API changes very quick.

There are some gotchas, like creating Unity Catalog tables (delta), that should be mentioned in the documentation in my opinion - not production ready or something like this.

I've noticed also lately more and more question in the forum about databricks/terraform. It looks like we are building here nice community :).

thanks,

Patryk.

[atul1146]

Thank you @Pat Sienkiewicz​ for the information.... Appreciated.