topic Re: Connect from Databricks to Hbase HDinsight cluster. in Data Engineering

Connect from Databricks to Hbase HDinsight cluster.

Vadim1 — Mon, 30 May 2022 12:57:10 GMT

Hi, I have Databricks installation in Azure. I want to run a job that connects to HBase in a separate HDinsight cluster.

What I tried:

Created a peering between base cluster and Databricks vNets.
I can ping IPs of Hbase zookeeper nodes but I cannot access them by domain name (zk1.abc.net, etc). That's why I'm getting timeout and host-not-found errors.

As I understand I need to set up a DNS to be able to communicate with Hbase but I cannot change Databricks DNS (locked by Databricks security policy).

Can you help me with this, please?

Maybe you can refer to the guide that I can use?

Thanks.

Re: Connect from Databricks to Hbase HDinsight cluster.

-werners- — Tue, 31 May 2022 12:00:15 GMT

@Vadim Z, Could it be a permission setting for your user?

Because you can change DNS settings on Databricks (f.e. https://kb.databricks.com/cloud/custom-dns-routing.html)

Re: Connect from Databricks to Hbase HDinsight cluster.

Vadim1 — Thu, 02 Jun 2022 12:18:15 GMT

Thanks for the response, werners. I found that my Databricks cluster was created with a private virtual network that cannot be changed, so I deployed a new Databricks cluster with my own virtual network and this resolved all issues with connectivity.

Re: Connect from Databricks to Hbase HDinsight cluster.

User16764241763 — Mon, 06 Jun 2022 03:37:30 GMT

Vadim,

Thank you for the response. Appreciate it.