- 1219 Views
- 2 replies
- 3 kudos
User is running a job triggered from ADF in Databricks. In this job they need to use custom libraries that are in jars. Most of the times jobs are running fine, however sometimes it fails with:java.lang.NoClassDefFoundError: Could not initializeAny s...
- 1219 Views
- 2 replies
- 3 kudos
Latest Reply
Can you please check if there are more than one jar containing this class . If multiple jars of the same type are available on the cluster, then there is no guarantee of JVM picking the proper classes for processing, which results in the intermittent...
1 More Replies
- 1797 Views
- 3 replies
- 0 kudos
I would like to set the permissions to jobs such as granting "CAN_VIEW" or "CAN_MANAGE" to specific groups that run from ADF. It appears that we need to set permissions in pipe line where job runs from ADF, But I could not figure it out.
- 1797 Views
- 3 replies
- 0 kudos
Latest Reply
Thank you @Debayan Mukherjee and @Vidula Khanna for getting back to me. But, it didn't help my case. I am specifically looking for setting permissions to the job so that our team can see the job cluster including Spark UI with that privilege. ...
2 More Replies
- 3048 Views
- 2 replies
- 3 kudos
Hi Team ,I am using job cluster while setting Linked Service in ADF to call Data bricks Notebook activity .Cluster Detail - Policy - UnrestrictedAccess Mode - Single userUnity Catalog Enabled.databrick run time - 12.2 LTS (includes Apache Spark 3.3.2...
- 3048 Views
- 2 replies
- 3 kudos
Latest Reply
Hi @Akshay Patni We haven't heard from you since the last response from @Debayan Mukherjee . Kindly share the information with us, and in return, we will provide you with the necessary solution. Thanks and Regards
1 More Replies
- 484 Views
- 0 replies
- 0 kudos
In my particular use case, the creation of the mount is initiated through a notebook activity in Azure Data Factory (ADF). This activity utilizes a job cluster for the current execution. However, it has come to my attention that the mounts generated ...
- 484 Views
- 0 replies
- 0 kudos
by
KVNARK
• Honored Contributor II
- 1579 Views
- 2 replies
- 1 kudos
Notebook activity is getting timed out after certain time of running (5 hours) in ADF pipeline and getting timeout error.Its just simply getting timed out error. Problem is this will process TB of data daily. can anyone have any idea to fix this.
- 1579 Views
- 2 replies
- 1 kudos
by
Jkb
• New Contributor II
- 1338 Views
- 2 replies
- 2 kudos
- 1338 Views
- 2 replies
- 2 kudos
Latest Reply
Hi @J. G., Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does. Your feedback will...
1 More Replies
- 644 Views
- 2 replies
- 0 kudos
hi , I want date for runtime from ADF as @utcnow() -- base paramater of notebook activity in ADF and take the data in ADB using widgets as runtime_date, further i want that column to be added in my table X with the populated value from the widget.Eve...
- 644 Views
- 2 replies
- 0 kudos
Latest Reply
sher
Valued Contributor II
you can use as current_timestamp() or now()refer link: https://docs.databricks.com/sql/language-manual/functions/current_timestamp.html
1 More Replies
by
KVNARK
• Honored Contributor II
- 1612 Views
- 4 replies
- 6 kudos
how can we parameterize key of the spark-config in the job cluster linked service from Azure datafactory, we can parameterize the values but any idea how can we parameterize the key so that when deploying to further environment it takes the PROD/QA v...
- 1612 Views
- 4 replies
- 6 kudos
Latest Reply
@KVNARK . You can use Databricks Secrets (create a Secret scope from AKV https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes) and then reference a secret in spark configuration (https://learn.microsoft.com/en-us/azure/d...
3 More Replies
- 1480 Views
- 4 replies
- 4 kudos
How to run the databricks notebook through ADF ???
- 1480 Views
- 4 replies
- 4 kudos
Latest Reply
Hi @Punit Chauhan you can use databricks notebook activity in ADF to trigger you databricks notebook via ADF-
3 More Replies
by
g96g
• New Contributor III
- 2956 Views
- 8 replies
- 0 kudos
I have project where I have to read the data from NETSUITE using API. Databricks Notebook runs perfectly when I manually insert the table names I want to read from the source. I have dataset (csv) file in adf with all the table names that I need to r...
- 2956 Views
- 8 replies
- 0 kudos
Latest Reply
Have you tried do debug the json payload of adf trigger ? maybe it wrongly conveys tables names
7 More Replies
- 345 Views
- 0 replies
- 0 kudos
Hi Everyone ,My Business requirement is to schedule single job from 1st to 10th of the month on 12 AM , 3AM , 12PM , 4PM , 8PM and from 10th to Month End 1AM , 12PM , 4PM , 8PM right now we have created 2 schedular to meet the requirement and using c...
- 345 Views
- 0 replies
- 0 kudos
- 3800 Views
- 4 replies
- 1 kudos
I have noticed that my orchestrated pipelines (in ADF) sometimes fail due to this error:ErrorCode=FailedToReadFromAzureDatabricksDeltaLake,Failed to read from azure databricks delta lake.
Error message : Failed to send request to Azure Databricks Clu...
- 3800 Views
- 4 replies
- 1 kudos
Latest Reply
Hi @Oscar Dyremyhr Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
3 More Replies
- 7492 Views
- 10 replies
- 10 kudos
Hi,I would like to capture notebook custom log exceptions(python) from ADF pipeline based on the exceptions pipeline should got succeed or failed.Is there any mechanism to implement it. In my testing ADF pipeline is successful irrespective of the log...
- 7492 Views
- 10 replies
- 10 kudos
Latest Reply
Hi SailajaB,Try this out.Notebook, once executed successfully return a long JSON formatted output. We need to specify appropriate nodes to fetch the output. In below screenshot we can see that when notebook ran it returns empName & empCity as output....
9 More Replies
- 980 Views
- 2 replies
- 1 kudos
Hi Team,How do we check the existence of a table in ADF container using SQL query in Databricks?Thanks in advance.
- 980 Views
- 2 replies
- 1 kudos
Latest Reply
Hi, please elaborate on the issue for us to help you resolve it.
1 More Replies
- 4512 Views
- 2 replies
- 3 kudos
I am trying to execute a local PySpark script on a Databricks cluster via dbx utility to test how passing arguments to python works in Databricks when developing locally. However, the test arguments I am passing are not being read for some reason. Co...
- 4512 Views
- 2 replies
- 3 kudos
Latest Reply
You can pass parameters using dbx launch --parametersIf you want to define it in the deployment template please try to follow exactly databricks API 2.1 schema https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate (for examp...
1 More Replies
