cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

itamarwe
by New Contributor II
  • 128 Views
  • 3 replies
  • 1 kudos

Resolved! Google PubSub for DLT - Error

I'm trying to create a delta live table from a Google PubSub stream.Unfortunately I'm getting the following error:org.apache.spark.sql.streaming.StreamingQueryException: [PS_FETCH_RETRY_EXCEPTION] Task in pubsub fetch stage cannot be retried. Partiti...

  • 128 Views
  • 3 replies
  • 1 kudos
Latest Reply
itamarwe
New Contributor II
  • 1 kudos

Hi @Kaniz_Fatma, it was indeed a permissions issue. Nevertheless, I must admit that the error message is slightly misleading.Thanks.

  • 1 kudos
2 More Replies
phguk
by New Contributor III
  • 247 Views
  • 4 replies
  • 1 kudos

Using Azure Key Vault secret to access Azure Storage

I am trying to configure access to Azure Storage Account (ADLS2) using OAUTH.  The doc here gives an example of how to specify a secret in a cluster's spark configuration{{secrets/<secret-scope>/<service-credential-key>}}I can see how this works for ...

  • 247 Views
  • 4 replies
  • 1 kudos
Latest Reply
Slash
New Contributor II
  • 1 kudos

Hi @phguk ,To make it work you need to assign role of Key Vault Secret User to built in databricks service principal. Key vault administrator role that you picked is about control plane in azure. (If you are interested in this topic type in Google az...

  • 1 kudos
3 More Replies
Linda19
by New Contributor
  • 98 Views
  • 1 replies
  • 1 kudos

What is the Best Postman Alternative?

Hey guys, I have been using Postman for quite some time now and getting disappointed recently and want to make a swtich. Is there something better than Postman? I've heard about that APIDog is much easier to use with a much better UI, and support all...

  • 98 Views
  • 1 replies
  • 1 kudos
Latest Reply
Slash
New Contributor II
  • 1 kudos

Hi @Linda19 ,I can recommend insomnia.rest and hopscotch. Or CURL

  • 1 kudos
Karthikext
by New Contributor
  • 90 Views
  • 1 replies
  • 0 kudos

Databricks connectivity issue when it is premium account

Hello Team,Our application is running in an on-prem VM, not in Azure, and it is trying to connect to Azure Databricks. The Databricks instance on Azure is backed by ADLS Gen 2 storage. It is a Premium tier instance with private web and API endpoint U...

  • 90 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Karthikext,  Ensure that the service principal you’ve created has the necessary permissions to access Databricks. Specifically, it should be either a Contributor or Owner role on the Databricks workspace resource in Azure.You can add the service ...

  • 0 kudos
Arnold_Souza
by New Contributor III
  • 4906 Views
  • 6 replies
  • 3 kudos

How to move a metastore to a new Storage Account in unity catalog?

Hello, I would like to change the Metastore location in Databricks Account Console. I have one metastore created that is in an undesired container/storage account. I could see that it's not possible to edit a metastore that is already created. I coul...

1.JPG
  • 4906 Views
  • 6 replies
  • 3 kudos
Latest Reply
Bakkie
New Contributor II
  • 3 kudos

Facing the same challenge. Any update on this?The ADLS Gen 2 path seems rather definitive when setting up a metastore:

  • 3 kudos
5 More Replies
Prashanthkumar
by New Contributor III
  • 88 Views
  • 1 replies
  • 0 kudos

Issue with Databricks repo patch

I have a strange issue when i am trying to do Databricks repo PATCH rest API using Postman it works however if i run the same curl command from Github actions it doesn't do the refresh. What could be the reason? any suggestions.

  • 88 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Prashanthkumar, Could you please ensure that the access token used in the GitHub Actions workflow has the necessary permissions to perform the PATCH operation on the Databricks repo? You can check the token's scope in the Databricks user settings...

  • 0 kudos
Phani1
by Valued Contributor
  • 115 Views
  • 1 replies
  • 0 kudos

Denodo with Databricks

Hi Team,How can I make a SQL Endpoint web api in Databricks that can be used in denodo?Regards,Janga

  • 115 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Phani1,  Databricks provides an API called the SQL Statement Execution API. It allows you to submit standard SQL statements over HTTP and retrieve results. You can use this API to integrate Databricks with various applications, including Denodo.Y...

  • 0 kudos
Anusha95
by New Contributor II
  • 137 Views
  • 2 replies
  • 0 kudos

Filtering output dataset using parameter values from different dataset

Hi Team, I am trying to build dynamic dashboard for which I have created 2 dataset and I am passing values from 1st dataset to 2nd dataset as parameter. When an user selects parameter values of 1st dataset, 2nd dataset should use those user input val...

  • 137 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

Here are a few suggestions: Ensure that your query in the second dataset is correctly set up to accept and use the parameters from the first dataset. The parameters should be correctly mapped to the catalog, schema, and table name in your "select * ...

  • 0 kudos
1 More Replies
ShankarM
by New Contributor III
  • 133 Views
  • 1 replies
  • 0 kudos

Resolved! Hadoop Hive migration to Databricks

Hi,Can you let me know what are the challenges and how to mitigate while migrating Hive objects to Databricks. I could not get any information on this. Can you please provide.

  • 133 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @ShankarM, Here are the main considerations and mitigation strategies:   Hive Workload Adaptation: Challenge: Adapting existing Hadoop workloads to Databricks’ advanced analytics framework.Mitigation: Rework and fine-tune your Hadoop workload...

  • 0 kudos
Data_Engineer3
by Contributor II
  • 1529 Views
  • 5 replies
  • 0 kudos

Delete databricks community post

Hi All,If in case if I make any mistake in my previous post in the databricks community, how can delete the post which I was posted and got replied on the same. Is this possible to delete the old post which was already replied by someone else.Thanks,

  • 1529 Views
  • 5 replies
  • 0 kudos
Latest Reply
Data_Engineer3
Contributor II
  • 0 kudos

I need to delete my old post which contain the details which should not be shared.https://community.databricks.com/t5/data-engineering/need-to-define-the-struct-and-array-of-struct-field-colum-in-the/m-p/58131#M31022 

  • 0 kudos
4 More Replies
jose_db_aws
by New Contributor II
  • 108 Views
  • 1 replies
  • 1 kudos

Resolved! unable to perform CD using github actions for MLops

Hi, I am new to databricks and MLOPS as well. I am trying out the databricks asset bundles mlops tutorial at https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.htmlmy cloud account is using AWS. I had asked to setup FS and Model Registry i...

jose_db_aws_0-1721042139398.png jose_db_aws_0-1721043189198.png jose_db_aws_1-1721042263883.png jose_db_aws_2-1721042559490.png
  • 108 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_db_aws
New Contributor II
  • 1 kudos

The problem was with the secrets at github actions were different from the Personal access token that I had set as my environment variable, DATABRICKS_TOKEN locally. After I copied this over the github's STAGING_WORKSPACE_TOKEN it is able to run the ...

  • 1 kudos
phguk
by New Contributor III
  • 1957 Views
  • 4 replies
  • 0 kudos

Adding NFS storage as external volume (Unity)

Can anyone share experience (or point me to another reference) that describes how to configure Azure Blob storage which has NFS enabled as an external volume to Databricks ?I've succeeded in adding SMB storage to Databricks but (if I understand prope...

  • 1957 Views
  • 4 replies
  • 0 kudos
Latest Reply
walter
New Contributor II
  • 0 kudos

hi @phguk could you share how you managed to create an external volume referencing to an azure fileshare ?are you using Unity catalog for this ? it was my understanding this is not possible.

  • 0 kudos
3 More Replies
Rajani
by Contributor II
  • 332 Views
  • 2 replies
  • 2 kudos

Resolved! How to pass a dynamic query to source server from databricks

I have this usecase wherein i am supposed to pass a dynamic query to get data from source I have tried the query option but its giving error SparkConnectGrpcException: (com.microsoft.sqlserver.jdbc.SQLServerException) Incorrect syntax near the keywor...

  • 332 Views
  • 2 replies
  • 2 kudos
Latest Reply
Rajani
Contributor II
  • 2 kudos

Hi  thanks for your reply,I have used foreign catalouge to fetch required data from information schema  then i am creating the dynamic query in databricks and then passing in query this is working for me! @Kaniz_Fatma

  • 2 kudos
1 More Replies
rameshkumar610
by New Contributor
  • 498 Views
  • 5 replies
  • 0 kudos

S60 Eliminate SPN secrets - Connect Azure Databricks to ADLS Gen2 , Gen1 via custom AD token

Hi Team,In Azure Databricks, we currently use Service Principal when creating Mount Points to Azure storage ( ADLS Gen1, ADLS Gen 2 and Azure Blob Storage).As part of S360 action to eliminate SPN secrets, we were asked to move to SPN+certificate / MS...

  • 498 Views
  • 5 replies
  • 0 kudos
Latest Reply
iakshaykr
New Contributor II
  • 0 kudos

@ramesitexp Yes @Slash is correct for now only valid option is below : OAuth 2.0 with a Microsoft Entra ID service principalShared access signatures (SAS)Account keys For now we are using OAuth 2.0 with a Microsoft Entra ID service principal  with cl...

  • 0 kudos
4 More Replies
Spyro_3
by New Contributor
  • 99 Views
  • 1 replies
  • 0 kudos

Free trail account

Hi Can I able to create unity catalog using free trail account?

  • 99 Views
  • 1 replies
  • 0 kudos
Latest Reply
Slash
New Contributor II
  • 0 kudos

Hi @Spyro_3 ,Yeah, you should be able to create unity catalog. Trial version allows you to create premium workspace which is required for unity catalog. Notice, that to setup metastore you also need to have Global Administrator permission (if we are ...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors