topic Re: Create workspace in Databricks deployed in Google Cloud using terraform in Data Engineering

Create workspace in Databricks deployed in Google Cloud using terraform

horatiug — Wed, 05 Oct 2022 16:31:44 GMT

In the documentation

https://registry.terraform.io/providers/databricks/databricks/latest/docs

https://docs.gcp.databricks.com/dev-tools/terraform/index.html

I could not find documentation on how to provision Databricks workspaces in GCP. Only creating Databricks workspaces in AWS and Azure is mentioned.

Is it possible to provision Databricks workspaces using terraform in Google Cloud Platform ?

Re: Create workspace in Databricks deployed in Google Cloud using terraform

horatiug — Mon, 10 Oct 2022 12:48:12 GMT

Hi @Debayan Mukherjee

thanks for the document. Really valuable resource. I have tried and got blocked into following error:

databricks_mws_workspaces.this: Creating...

╷

│ Error: cannot create mws workspaces: BAD_REQUEST: workspace creation REST API is not available to use. Please contact Databricks support

│

│ with databricks_mws_workspaces.this,

│ on main.tf line 35, in resource "databricks_mws_workspaces" "this":

│ 35: resource "databricks_mws_workspaces" "this" {

│

Any reason why this could happen ? Is this API maybe working from Databricks PREMIUM plan and not working in STANDARD plan ?

Thanks

Horatiu

Re: Create workspace in Databricks deployed in Google Cloud using terraform

Anonymous — Mon, 14 Nov 2022 04:01:04 GMT

Hi @horatiu guja

Does @Debayan Mukherjee response answer your question?

If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else, we can help you with more details.

Re: Create workspace in Databricks deployed in Google Cloud using terraform

horatiug — Mon, 14 Nov 2022 12:44:46 GMT

Hi @Vidula Khanna

solution works and will mark it as best. Just want to mention that Accounts API are needed and in GCP this is in preview. To use them the preview must be enabled for the corresponding account.

reggards

Horatiu

Re: Create workspace in Databricks deployed in Google Cloud using terraform

horatiug — Thu, 17 Nov 2022 15:08:13 GMT

Hi @Debayan Mukherjee

following the document I was able to create workspace and add some infrastructure in the workspace. Issue I face now is when running terraform destroy. Then I get following error:

╷

│ Error: cannot read user: cannot configure google-workspace auth: could not obtain OIDC token. impersonate: an audience must be provided Running 'gcloud auth application-default login' may help. Attributes used: google_credentials, google_service_account. Please check https://registry.terraform.io/providers/databricks/databricks/latest/docs#authentication for details

│

│ with databricks_user.me,

│ on main.tf line 70, in resource "databricks_user" "me":

│ 70: resource "databricks_user" "me" {

│

When the error occurs I am still logged in GCP using gcloud auth application-default login. Any hint what could be the reason for that ?

thanks

Horatiu

Re: Create workspace in Databricks deployed in Google Cloud using terraform

Debayan — Thu, 06 Oct 2022 13:02:55 GMT

Hi, @horatiu guja , as of now we do not have public document on the same. It is possible, but, there is no straight forward way to do it.

Re: Create workspace in Databricks deployed in Google Cloud using terraform

Debayan — Thu, 06 Oct 2022 13:27:21 GMT

In addition to the previous comment, attaching the document which is in private preview as of now.

Please let us know if you need further clarification on the same.

Re: Create workspace in Databricks deployed in Google Cloud using terraform

Debayan — Mon, 10 Oct 2022 15:25:34 GMT

Hi @horatiu guja , let me check on the error. This is still under development and does not work any many cases (hence the document is not published publicly).

Re: Create workspace in Databricks deployed in Google Cloud using terraform

Debayan — Mon, 10 Oct 2022 15:50:59 GMT

@horatiu guja , This feature works only for premium plan.