Hi @Kaniz_Fatma ,
Thank you very much for the reply. But I don't think this actually resolves our concern.
All these solutions talk about utilizing the databricks cluster to access/read data in Databricks. They focus on getting to the Databricks data through databricks cli or REST APIs.
We am not concerned with getting to the data.
What we want to achieve is connect to the underlying individual Virtual machines of the clusters and run some scans on those virtual machine. Scan the operating system, open ports, installed libraries and linux configurations of those cluster virtual machines.
We want to consider the databricks clusters as yet another set of virtual machines used in our organization and assess the security of those virtual machines.
I am not sure if this is achievable, but what we are looking for is a way to somehow expose these cluster virtual machines to external tools. For example, then we could may be give the public IPs of the cluster virtual machines to our tools so then can run port scans and such. May be there is another way to do it using privatelinks. Not sure.
Would love to hear if this is possible.
Thanks for the feedback. Really appreciate it.