cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Create workspace in Databricks deployed in Google Cloud using terraform

horatiug
New Contributor III

In the documentation

https://registry.terraform.io/providers/databricks/databricks/latest/docs

https://docs.gcp.databricks.com/dev-tools/terraform/index.html

I could not find documentation on how to provision Databricks workspaces in GCP. Only creating Databricks workspaces in AWS and Azure is mentioned.

Is it possible to provision Databricks workspaces using terraform in Google Cloud Platform ?

8 REPLIES 8

Debayan
Esteemed Contributor III

Hi, @horatiu gujaโ€‹ , as of now we do not have public document on the same. It is possible, but, there is no straight forward way to do it.

Debayan
Esteemed Contributor III

In addition to the previous comment, attaching the document which is in private preview as of now.

Please let us know if you need further clarification on the same.

horatiug
New Contributor III

Hi @Debayan Mukherjeeโ€‹ 

thanks for the document. Really valuable resource. I have tried and got blocked into following error:

databricks_mws_workspaces.this: Creating...

โ•ท

โ”‚ Error: cannot create mws workspaces: BAD_REQUEST: workspace creation REST API is not available to use. Please contact Databricks support

โ”‚ 

โ”‚  with databricks_mws_workspaces.this,

โ”‚  on main.tf line 35, in resource "databricks_mws_workspaces" "this":

โ”‚  35: resource "databricks_mws_workspaces" "this" {

โ”‚ 

Any reason why this could happen ? Is this API maybe working from Databricks PREMIUM plan and not working in STANDARD plan ?

Thanks

Horatiu

Debayan
Esteemed Contributor III

Hi @horatiu gujaโ€‹ , let me check on the error. This is still under development and does not work any many cases (hence the document is not published publicly).

Debayan
Esteemed Contributor III

@horatiu gujaโ€‹ , This feature works only for premium plan.

horatiug
New Contributor III

Hi @Debayan Mukherjeeโ€‹ 

following the document I was able to create workspace and add some infrastructure in the workspace. Issue I face now is when running terraform destroy. Then I get following error:

โ•ท

โ”‚ Error: cannot read user: cannot configure google-workspace auth: could not obtain OIDC token. impersonate: an audience must be provided Running 'gcloud auth application-default login' may help. Attributes used: google_credentials, google_service_account. Please check https://registry.terraform.io/providers/databricks/databricks/latest/docs#authentication for details

โ”‚ 

โ”‚  with databricks_user.me,

โ”‚  on main.tf line 70, in resource "databricks_user" "me":

โ”‚  70: resource "databricks_user" "me" {

โ”‚ 

When the error occurs I am still logged in GCP using gcloud auth application-default login. Any hint what could be the reason for that ?

thanks

Horatiu

Anonymous
Not applicable

Hi @horatiu gujaโ€‹ 

Does @Debayan Mukherjeeโ€‹ response answer your question?

If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else, we can help you with more details.

horatiug
New Contributor III

Hi @Vidula Khannaโ€‹ 

solution works and will mark it as best. Just want to mention that Accounts API are needed and in GCP this is in preview. To use them the preview must be enabled for the corresponding account.

reggards

Horatiu

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group