Databricks set up in Prod environment

atul1146 — Sat, 05 Nov 2022 16:15:52 GMT

Hi! can anyone please help me with a documentation which can help me set up integration between data bricks with AWS without a QuickStart default cloud formation template.

I would want to use my own CFT rather than using the default due to security compliance.

Re: Databricks set up in Prod environment

Pat — Mon, 07 Nov 2022 09:35:24 GMT

Hi @Atul S ,

I think that terraform is recommended way to go with Databricks deployment. I mean it's also supported now by the Databricks support.

I haven't look much on the CloudFormation setup, because we decided to go with the Terraform in the company (CF was our IaC tool).

Here is some info about the IaC (tf, cdk).

https://docs.databricks.com/dev-tools/index-iac.html

I've checked this before:

https://aws-quickstart.github.io/quickstart-databricks-unified-data-analytics-platform/

but it's 2020, and now we have all those new things like Unity Catalog. TBH - terrafrom is seems to be way forward.

https://docs.databricks.com/administration-guide/account-api/templates.html :

Databricks Terrafrom provider is not ideal, but it's adapting quickly to the newest Databricks API changes very quick.

There are some gotchas, like creating Unity Catalog tables (delta), that should be mentioned in the documentation in my opinion - not production ready or something like this.

I've noticed also lately more and more question in the forum about databricks/terraform. It looks like we are building here nice community :).

thanks,

Patryk.

Re: Databricks set up in Prod environment

atul1146 — Mon, 07 Nov 2022 10:20:04 GMT

Thank you @Pat Sienkiewicz for the information.... Appreciated.

topic Re: Databricks set up in Prod environment in Data Engineering

Databricks set up in Prod environment

Re: Databricks set up in Prod environment

Re: Databricks set up in Prod environment