cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks-connect 11.3 works with Unity Catalog ?

DataEngineer92
New Contributor II

Databricks-connect 11.3 works for unity catalog enabled cluster or not ? I am trying to utilize the databricks-connect in Azure DevOps pipeline using Microsoft hosted agent. But getting the below error when running the databricks-connect test command. 

Error :

org.apache.spark.SparkException: There is no Credential Scope.  

ValueError: Scala command failed to produce correct result

1 ACCEPTED SOLUTION

Accepted Solutions

bastih
New Contributor III

Hi @Rey Jhon​ 

Sadly, databricks-connect for DBR versions < 13 does not support UC, as outlined in the limitations section: https://docs.databricks.com/dev-tools/databricks-connect-legacy.html#limitations.

To use UC, we recommend using DBR version >= 13 with databricks-connect, which supports UC - and has many additional quality-of-life improvements over previous versions.

Best,

Sebastian

View solution in original post

3 REPLIES 3

Anonymous
Not applicable

@Rey Jhon​ :

Databricks-connect should work with Unity catalog enabled clusters as well. The error you are seeing could be caused by an issue with your credentials. Here are some steps you can take to troubleshoot the issue:

  1. Double-check that you have provided the correct Databricks instance URL, Personal Access Token (PAT), and organization ID when configuring databricks-connect.
  2. Make sure that the credentials you are using have the correct permissions to access the cluster. You may need to check with your Databricks administrator to ensure that the correct access controls have been configured.
  3. Verify that the cluster is running and accessible. You can do this by checking the cluster status in the Databricks workspace or by running a simple Spark job against the cluster using the Databricks CLI.
  4. Try running the databricks-connect test command again with the --verbose flag to see if any additional error messages are printed. This may provide more insight into what is causing the issue.

Anonymous
Not applicable

Hi @Rey Jhon​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

bastih
New Contributor III

Hi @Rey Jhon​ 

Sadly, databricks-connect for DBR versions < 13 does not support UC, as outlined in the limitations section: https://docs.databricks.com/dev-tools/databricks-connect-legacy.html#limitations.

To use UC, we recommend using DBR version >= 13 with databricks-connect, which supports UC - and has many additional quality-of-life improvements over previous versions.

Best,

Sebastian

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.