Allowing Azure Databricks to query a local/private database

AskMe55 — Wed, 21 Aug 2024 12:27:24 GMT

Hello,

I am trying to set up a simple machine learning pipeline where I want to generate example data on my computer, save this data into a MariaDB database on my computer, and then allow Azure Databricks to access my local database to train a model with it, etc. What is the best and preferred way to allow Azure Databricks to access my database since I don't want to migrate my data into Azure.

Thanks!

Re: Allowing Azure Databricks to query a local/private database

-werners- — Wed, 21 Aug 2024 13:01:36 GMT

To do that, Databricks needs access to your local LAN.
This means configuring network security groups or a firewall.
Setting up a private endpoint is also a good idea.
You also have to make sure that your databricks cluster can connect to your on-prem database.

Now, assuming all this is working, what will actually happen is this:
databricks connects to your database, fetches the data you defined (in your spark script) and moves it to the databricks workers which reside in the cloud.
Databricks does the transformationts etc and finally moves the data back to your on-prem system.
Depending on the sizing of the underlying on-prem database this can take a while.
So your data will be sent to the cloud, no matter what (best case = worker RAM only).
I am not sure if that is what you want.
Most of the times, on-prem data is copied to cloud storage, and processed from there.

topic Re: Allowing Azure Databricks to query a local/private database in Data Engineering

Allowing Azure Databricks to query a local/private database

Re: Allowing Azure Databricks to query a local/private database