Union Data from two separate Databricks Workspaces

[626371]

Hi Everyone,

This may sound bit odd as I haven't used Databricks much.

I have a use case where I want to union data from two separate Databricks workspaces (under the same account) into a single SQL view. (Basically 2-2 tables from each workspaces and Data is not that big)

Which is the best way to do so? I believe unity catalog or Delta sharing are the options but I'm not sure which one is the best to proceed with?

Also, are there any additional details that I need to take care of while doing so?

p.s. - I'm researching the options for now, I haven't tested anything yet on the data.

Any leads are appreciated. Thanks in advance.

Thanks,

Bindi

[Kaniz]

Hi @Bindi P​, Databricks supports sharing feature tables across multiple workspaces. For example, from your workspace, you can create, write to, or read from a feature table in a centralized feature store. This is useful when multiple teams share access to feature tables or when your organization has numerous workspaces to handle different stages of development.

For a centralized feature store, Databricks recommends that you designate a single workspace to store all feature store metadata and create accounts for each user who needs access to the feature store.

Suppose your teams are also sharing models across workspaces. In that case, you may choose to dedicate the same centralized workspace for both feature tables and models or specify different centralized workspaces for each.

Access to the centralized feature store is controlled by tokens. Each user or script that needs access creates a personal access token in the centralized feature store and copies that token into the secret manager of their local workspace. Each API request sent to the centralized feature store workspace must include the access token; the Feature Store client provides a simple mechanism to specify the secrets to be used when performing cross-workspace operations.

For more information, please go through the documentation.

[626371]

@Kaniz Fatma​ 

Thanks for providing your inputs.

I have to use and maintain this for this single use case. we haven't set up feature store workspace at the moment.

Do you think, It's a good idea to create and maintain feature store for it?

[626371]

As suggested by Databricks support team, we are looking at the option of Unity Catalog to implement for our business case.

[Anonymous]

Hey there @Bindi P​ 

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? It would be really helpful for the other members too.

Cheers!