Public exposure for clusters in SCC enabled workspaces

[NadithK]

Hi,
We are facing a requirement where we need to somehow expose one of our Databricks clusters to an external service. Our organization's cyber team is running a security audit of all of the resource we use and they have some tools which they use to run scans on VMs we use for our work.

Databricks clusters have also come up as our sensitive data goes through them. We have workspaces created with SCC enabled.

Is there a way for us to expose our Databricks clusters so these external tools could access the clusters to run their scans. At least some public IP or hostname that can be exposed. It looks like this is not possible at first glance.
Wondering if anyone else has run into the same requirement and if they have done a workaround or something.

We have our setup in Azure cloud and these cyber tools are running in AWS.

Thanks in advance.

[Kaniz]

Hi @NadithK, Exposing Databricks clusters to external services while maintaining security is a common challenge.

Let’s explore a couple of approaches you can consider:

1. Access via Databricks User Groups and Table Access Control:

   • Create a Databricks User Group for each external consumer (e.g., your cyber team).
   • Enable Table Access Control for your Databricks account.
   • Set up a High-Concurrency cluster and enable Table Access Control for the cluster.
   • Grant the user group permission to attach to the cluster.
   • Create an external master database where you create tables for your data lake files (and optionally create views to transform data into a consumer-friendly schema).
   • Create a separate database with filtered data for each consumer and grant the user group access to objects in that database.
   • Connect to the consumer database from external tools (e.g., Redash, Power BI, Grafana) to access the...¹.

2. Using Databricks-Connect Package:

   • If you need to connect to Databricks clusters from your local machine or an external non-Spark cluster, consider using the databricks-connect package from PyPI.
   • This package provides an alternative way to establish a connection based on a Config object from the...².

3. Azure Databricks with Secure Cluster Connectivity (SCC):

   • Deploy Azure Databricks with SCC enabled in a spoke virtual network using VNet injection and Private Link.
   • Set up Private Link endpoints for your Azure Data Services in a separate subnet within the Azure Databricks spoke virtual network.
   • This approach ensures secure connectivity while allowing you to manage access to your Databricks clu...³.

Remember to evaluate these options based on your specific requirements and security policies. Each approach has its trade-offs, so choose the one that best aligns with your organization’s needs. If you have any further questions or need additional guidance, feel free to ask! 😊

[NadithK]

Hi @Kaniz ,

Thank you very much for the reply. But I don't think this actually resolves our concern.
All these solutions talk about utilizing the databricks cluster to access/read data in Databricks. They focus on getting to the Databricks data through databricks cli or REST APIs.

We am not concerned with getting to the data.
What we want to achieve is connect to the underlying individual Virtual machines of the clusters and run some scans on those virtual machine. Scan the operating system, open ports, installed libraries and linux configurations of those cluster virtual machines.

We want to consider the databricks clusters as yet another set of virtual machines used in our organization and assess the security of those virtual machines.

I am not sure if this is achievable, but what we are looking for is a way to somehow expose these cluster virtual machines to external tools. For example, then we could may be give the public IPs of the cluster virtual machines to our tools so then can run port scans and such. May be there is another way to do it using privatelinks. Not sure.

Would love to hear if this is possible.

Thanks for the feedback. Really appreciate it.