Querying Mysql db from Azure databricks where public access is disabled

ashraf1395 — Fri, 19 Jul 2024 07:21:17 GMT

Hi there,

We are trying to setup a infra that ingest data from MySQL hosted on awa EC2 instance with pyspark and azure databricks and dump to the adls storage.

Since databases has public accessibility disabled and how can I interact with MySQL from azure databricks?

Major challenge is public accessiblity disabled. Querying MySQL normally is doable

Re: Querying Mysql db from Azure databricks where public access is disabled

-werners- — Fri, 19 Jul 2024 07:35:25 GMT

you will need some kind of tunnel that opens the db server to external access.
Perhaps a vpn is an option?
If not: won't be possible.
An alternative way would be to have some local LAN tool extract the data and then move it to S3/... and afterwards let Databricks pick up those files.
If I'm not mistaken, Databricks is integrating ETL functionality in their platform (I think it is in private preview), so it might be possible in the future (although you would still need some kind of opening to the public facing internet using a gateway/vpn whatever).

topic Querying Mysql db from Azure databricks where public access is disabled in Data Engineering

Querying Mysql db from Azure databricks where public access is disabled

Re: Querying Mysql db from Azure databricks where public access is disabled