topic Re: How to query existing storage and network configuration (AWS)? in Data Engineering

How to query existing storage and network configuration (AWS)?

ThisNoName — Mon, 19 Aug 2024 12:53:48 GMT

Trying to provision a simple workspace. All the code I can find look something like the following, where credential/storage/network resources are created, then referenced.

In my case, it's a separate repo and try to reuse existing configurations. I was able to query mws_credential and find credential_id, but there are no data source for storage_configuration and mws_networks, according to databricks documentation.

Any hints where I should look? Undocumented data sources? Wrapped up somewhere in other sources? I can see both configuration_id and network_id in tfstate but that's probably a bit too much

Thanks

data "databricks_mws_credentials" "all" { provider = databricks.mws }

resource "databricks_mws_workspaces" "this" { provider = databricks.mws credentials_id = databricks_mws_credentials.this.credentials_id storage_configuration_id = databricks_mws_storage_configurations.this.storage_configuration_id network_id = databricks_mws_networks.this.network_id ... ... } resource "databricks_mws_credentials" "this" { provider = databricks.mws ... ... } resource "databricks_mws_storage_configurations" "this" { provider = databricks.mws ... ... } resource "databricks_mws_networks" "this" { provider = databricks.mws security_group_ids = var.security_group_ids subnet_ids = var.vpc_private_subnets vpc_id = var.vpc_id ... ... }

Re: How to query existing storage and network configuration (AWS)?

raphaelblg — Mon, 19 Aug 2024 16:11:15 GMT

Hi @ThisNoName,

Based on your description it looks like you're trying to get Databricks account level information for networks and storage configurations. You can easily achieve that through the Databricks account API.

Here are the docs:

- Get all network configurations: https://docs.databricks.com/api/account/networks/list

- Get all storage configurations: https://docs.databricks.com/api/account/storage/list

Re: How to query existing storage and network configuration (AWS)?

ThisNoName — Mon, 19 Aug 2024 21:23:58 GMT

So there's no official terraform data sources for these? I guess maybe curl and grep or something? Beats fumbling through tfstate.

Will give a try. Thanks.