cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
cancel
Showing results for 
Search instead for 
Did you mean: 

Feature request: Run cluster in a different region

Erik
Valued Contributor II

We are one of those which unfortunately decided (earlier) to set up our infastructure in one of the sub-par Azure regions, and we happen to dont have any GPU clusters available.

We would like to test out some GPU enhanced ML in databricks, but at the moment our only solution is to create a completely new workspace in a new region and mount in our data there. This bypasses our CICD and our normal workflow, but works for testing.

I think it would be really usefull if we could create a cluster in another region from our normal workspace. I understand that it would cause extra network costs for both data transfer and managment, but it would allow us to run things in our normal workspace, and allow us to do the tradeoff between the extra cost of data-transfer vs the extra work of having multiple workspaces with multiple unity catalogs (since it can't span regions either).

It could very well be hidden behind a big red warning, maybe even a feature flag on the workspace level.

We need it for GPU clusters now, but honestly it would be quite usefull to also be able to use a serverless SQL warehouse seamlessly hosted in another region as well (with all the same caveats as above).

Alternatively, if unity catalog could span regions that would solve everything:-P

 

 

 

1 REPLY 1

Yeshwanth
Valued Contributor
Valued Contributor

@Erik 

I think it would be really useful if we could create a cluster in another region from our normal workspace

This does not seem like a possible thing for me as the Control plane resources and the Data plane resources are bound to that region. If it happens it would be great 🙂

Alternatively, if unity catalog could span regions that would solve everything:-P

Spot on!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.