Exposing Data for Consumers in non-UC ADB

[fly_high_five]

Hi,

I want to expose data to consumers from our non-UC ADB. Consumers would be consuming data mainly using SQL client like DBeaver.  I tried SQL endpoint of Interactive Cluster and connected via DBeaver however when I try to fetch/export all rows of table it fails but succeds only for small number of rows upto 8k. 

What is the best way to expose consumer layer?

[szymon_dybczak]

Hi @fly_high_five ,

Which version of JDBC driver are you using? How you connection string looks like?

[fly_high_five]

Hi @szymon_dybczak 

I am using latest JDBC driver 2.7.3 https://www.databricks.com/spark/jdbc-drivers-archive

And my JDBC url comes from JDBC endpoint of Interactive Cluster.

jdbc:databricks://adb-{workspace_id}.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/{workspace_id}/{cluster_id};AuthMech=11;Auth_Flow=2;TokenCachePassPhrase=U2MToken;EnableTokenCache=0

[szymon_dybczak · Accepted solution]

Looking at the error I noticed the weird name of the storage account. So now I'm thinking that your jdbc connection is using CloudFetch.

"Cloud Fetch, a capability that fetches query results through the cloud storage that is set up in your Azure Databricks deployment.

Query results are uploaded to an internal DBFS storage location as Arrow-serialized files of up to 20 MB. When the driver sends fetch requests after query completion, Azure Databricks generates and returns shared access signatures to the uploaded files. The JDBC driver then uses the URLs to download the results directly from DBFS."

Can you add following option to your JDBC url? It will disable cloudFetch. Once you set this option try again.

EnableQueryResultDownload=0

[fly_high_five]

Hi @szymon_dybczak ,
Your recommendation worked. I modified JDBC url to 

jdbc:databricks://adb-{workspace_id}.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/{workspace_id}/{cluster_id};AuthMech=11;Auth_Flow=2;TokenCachePassPhrase=U2MToken;EnableTokenCache=0;EnableQueryResultDownload=0.

I tried another solution which looks essentially same what @szymon_dybczak  you recommended. Solution is attached.