topic Re: Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport in Data Engineering

Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport=ON

TimB — Tue, 19 Sep 2023 19:52:40 GMT

I have added a connection to a MySql database in Azure, and I have created a foreign catalog in Databricks. But when I go to query the database I get the following error;

Connections using insecure transport are prohibited while --require_secure_transport=ON

I appreciate that I can turn off this setting on my database, but I would rather do it properly and configure it from the Databricks side. Is it possible to configure Databricks so it uses secure transport on external connections?

Re: Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport

TimB — Wed, 20 Sep 2023 08:10:54 GMT

Thanks Kaniz, but if you view the screenshot below, you will see the options I get, and I'm not sure where I should be adding those extra details. Can you advise?

Re: Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport

TimB — Wed, 20 Sep 2023 08:39:18 GMT

Hi Kaniz,

That is the way I work when I normally query data from the azure SQLdatabase in DBX. But I am trying to use the new Lakehouse Federation from within DBX itself which should abstract away those connections properties.

https://learn.microsoft.com/en-us/azure/databricks/query-federation/mysql

Re: Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport

J-Bradlee — Mon, 09 Sep 2024 00:33:22 GMT

Hi,

Is there an update on this? I have the same issue where I try using Lakehouse federation but I cannot input the additional connection information or manually edit the url created when defining a "connection" on the databrick UI required for Lakehouse federation.

Re: Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport

TimB — Mon, 09 Sep 2024 07:22:08 GMT

I was trying to connect to our database (Azure MySQL) from DBX, but we wanted require_secure_transport to be set to ON, and we didn't want to turn it off. We ended up moving DBX within a VNET and setting up a private link to get around this.

Re: Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport

J-Bradlee — Tue, 10 Sep 2024 02:20:27 GMT

Thanks Tim. That seems like the way to go. You prompted me to find this section of the docs: Networking recommendations for Lakehouse Federation - Azure Databricks | Microsoft Learn which mentions exactly what you just said.

On a side note: If you have set up databricks serverless compute to access your Azure mySQL? Did you do the same thing with the serverless compute?

Re: Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport

TimB — Tue, 10 Sep 2024 08:02:05 GMT

I was unable to use serverless compute with Azure MySQL. I did have a meeting with tech support at databricks and the conclusion at the time was - if your Azure MySql is not publically accessible, you cannot connect to it from serverless compute.

I'm mainly using serverless compute to access Dashboards, so I'm using jobs in the background to populate delta tables so the serverless compute can access those instead as a work around.