Databricks-connect 11.3 works with Unity Catalog ?

[DataEngineer92]

Databricks-connect 11.3 works for unity catalog enabled cluster or not ? I am trying to utilize the databricks-connect in Azure DevOps pipeline using Microsoft hosted agent. But getting the below error when running the databricks-connect test command. 

Error :

org.apache.spark.SparkException: There is no Credential Scope.  

ValueError: Scala command failed to produce correct result

[Anonymous]

@Rey Jhon​ :

Databricks-connect should work with Unity catalog enabled clusters as well. The error you are seeing could be caused by an issue with your credentials. Here are some steps you can take to troubleshoot the issue:

1. Double-check that you have provided the correct Databricks instance URL, Personal Access Token (PAT), and organization ID when configuring databricks-connect.
2. Make sure that the credentials you are using have the correct permissions to access the cluster. You may need to check with your Databricks administrator to ensure that the correct access controls have been configured.
3. Verify that the cluster is running and accessible. You can do this by checking the cluster status in the Databricks workspace or by running a simple Spark job against the cluster using the Databricks CLI.
4. Try running the databricks-connect test command again with the --verbose flag to see if any additional error messages are printed. This may provide more insight into what is causing the issue.

[marvin1]

Hi, I am having the exact same problem. I am using databricks-connect 11.3 with runtime 11.3 on a Unity-enabled single-node cluster. I am the Databricks admin and have manage permissions on the cluster. I have generated a new PAT and receive this message.

I do not see an option to run databricks-connect in verbose mode. Please elaborate?

❯ databricks-connect test --verbose

usage: databricks-connect.exe [-h] {configure,test,get-jar-dir,get-spark-home}

databricks-connect.exe: error: unrecognized arguments: --verbose

Thanks!

[Anonymous]

Hi @Rey Jhon​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

[bastih · Accepted solution]

Hi @Rey Jhon​ 

Sadly, databricks-connect for DBR versions < 13 does not support UC, as outlined in the limitations section: https://docs.databricks.com/dev-tools/databricks-connect-legacy.html#limitations.

To use UC, we recommend using DBR version >= 13 with databricks-connect, which supports UC - and has many additional quality-of-life improvements over previous versions.

Best,

Sebastian