cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Mohit_m
by • Valued Contributor II
  • 4074 Views
  • 2 replies
  • 3 kudos

Resolved! Could not initialize class error

User is running a job triggered from ADF in Databricks. In this job they need to use custom libraries that are in jars. Most of the times jobs are running fine, however sometimes it fails with:java.lang.NoClassDefFoundError: Could not initializeAny s...

Data Engineering
Reply
  • 4074 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Mohit_m
Valued Contributor II
  • 3 kudos

Can you please check if there are more than one jar containing this class . If multiple jars of the same type are available on the cluster, then there is no guarantee of JVM picking the proper classes for processing, which results in the intermittent...

  • 3 kudos
1 More Replies
dbx_8451
by • New Contributor II
  • 4005 Views
  • 3 replies
  • 0 kudos

How to set the permissions to databricks jobs that created and run from Azure Data Factory(ADF)?

I would like to set the permissions to jobs such as granting "CAN_VIEW" or "CAN_MANAGE" to specific groups that run from ADF. It appears that we need to set permissions in pipe line where job runs from ADF, But I could not figure it out. ​​

Data Engineering
Reply
  • 4005 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
dbx_8451
New Contributor II
  • 0 kudos

Thank you @Debayan Mukherjee​  and @Vidula Khanna​  for getting back to me. But, it didn't help my case. I am specifically looking for setting permissions to the job so that our team can see the job cluster including Spark UI with that privilege. ...

  • 0 kudos
2 More Replies
akshay_patni228
by • New Contributor II
  • 9034 Views
  • 2 replies
  • 3 kudos

Missing Credential Scope - Unable to call databrick(Scala) notebook from ADF

Hi Team ,I am using job cluster while setting Linked Service in ADF to call Data bricks Notebook activity .Cluster Detail - Policy - UnrestrictedAccess Mode - Single userUnity Catalog Enabled.databrick run time - 12.2 LTS (includes Apache Spark 3.3.2...

Data Engineering
Reply
  • 9034 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Akshay Patni​ We haven't heard from you since the last response from @Debayan Mukherjee​ ​ . Kindly share the information with us, and in return, we will provide you with the necessary solution. Thanks and Regards

  • 3 kudos
1 More Replies
KVNARK
by • Honored Contributor II
  • 3389 Views
  • 2 replies
  • 1 kudos

Resolved! Notebook activity is getting timed out in ADF pipeline.

Notebook activity is getting timed out after certain time of running (5 hours) in ADF pipeline and getting timeout error.Its just simply getting timed out error. Problem is this will process TB of data daily. can anyone have any idea to fix this.

Data Engineering
Reply
  • 3389 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
KVNARK
Honored Contributor II
  • 1 kudos

@Daniel Sahal​ - Noted. Thanks Daniel!

  • 1 kudos
1 More Replies
Jkb
by • New Contributor II
  • 3555 Views
  • 0 replies
  • 1 kudos

Workflow triggered by CLI shown "manually" triggered

We trigger different Worflows by ADF.These workflows will be shown triggered "manually".Is this behaviour intentional? At least for users, this is confusing.ADF-triggered Run: Databricks-Workflows: 

ADF_Monitor manually1 manually2
Data Engineering
Reply
  • 3555 Views
  • 0 replies
  • 1 kudos
sonali1996
by • New Contributor
  • 1233 Views
  • 2 replies
  • 0 kudos

adding Widget as a column and populating its value every-time in that column in a table.

hi , I want date for runtime from ADF as @utcnow() -- base paramater of notebook activity in ADF and take the data in ADB using widgets as runtime_date, further i want that column to be added in my table X with the populated value from the widget.Eve...

Data Engineering
Reply
  • 1233 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sher
Valued Contributor II
  • 0 kudos

you can use as current_timestamp() or now()refer link: https://docs.databricks.com/sql/language-manual/functions/current_timestamp.html

  • 0 kudos
1 More Replies
KVNARK
by • Honored Contributor II
  • 3356 Views
  • 4 replies
  • 6 kudos

Resolved! How to parameterize key of spark config in the job clusterlinked service from ADF

how can we parameterize key of the spark-config in the job cluster linked service from Azure datafactory, we can parameterize the values but any idea how can we parameterize the key so that when deploying to further environment it takes the PROD/QA v...

Data Engineering
Reply
  • 3356 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 6 kudos

@KVNARK .​ You can use Databricks Secrets (create a Secret scope from AKV https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes) and then reference a secret in spark configuration (https://learn.microsoft.com/en-us/azure/d...

  • 6 kudos
3 More Replies
Prototype998
by • New Contributor III
  • 3476 Views
  • 4 replies
  • 4 kudos

Resolved! Databricks notebook run

How to run the databricks notebook through ADF ??? 

Data Engineering
Reply
  • 3476 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 4 kudos

Hi @Punit Chauhan​ you can use databricks notebook activity in ADF to trigger you databricks notebook via ADF-

  • 4 kudos
3 More Replies
g96g
by • New Contributor III
  • 6024 Views
  • 8 replies
  • 0 kudos

Resolved! ADF pipeline fails when passing the parameter to databricks

I have project where I have to read the data from NETSUITE using API. Databricks Notebook runs perfectly when I manually insert the table names I want to read from the source. I have dataset (csv) file in adf with all the table names that I need to r...

Data Engineering
Reply
  • 6024 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
mcwir
Contributor
  • 0 kudos

Have you tried do debug the json payload of adf trigger ? maybe it wrongly conveys tables names

  • 0 kudos
7 More Replies
ossinova
by • Contributor II
  • 6649 Views
  • 4 replies
  • 1 kudos

ADF - Failed To Send Request to Databricks Cluster

I have noticed that my orchestrated pipelines (in ADF) sometimes fail due to this error:ErrorCode=FailedToReadFromAzureDatabricksDeltaLake,Failed to read from azure databricks delta lake. Error message : Failed to send request to Azure Databricks Clu...

Data Engineering
Reply
  • 6649 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
3 More Replies
SailajaB
by • Valued Contributor III
  • 11951 Views
  • 10 replies
  • 10 kudos

Resolved! Is there a way to capture the notebook logs from ADF pipeline?

Hi,I would like to capture notebook custom log exceptions(python) from ADF pipeline based on the exceptions pipeline should got succeed or failed.Is there any mechanism to implement it. In my testing ADF pipeline is successful irrespective of the log...

Data Engineering
Reply
  • 11951 Views
  • 10 replies
  • 10 kudos
View Replies
Latest Reply
GurpreetSethi
Databricks Employee
  • 10 kudos

Hi SailajaB,Try this out.Notebook, once executed successfully return a long JSON formatted output. We need to specify appropriate nodes to fetch the output. In below screenshot we can see that when notebook ran it returns empName & empCity as output....

  • 10 kudos
9 More Replies
Bharath_1610
by • New Contributor
  • 1852 Views
  • 2 replies
  • 1 kudos

Resolved! Check Existence of table

Hi Team,How do we check the existence of a table in ADF container using SQL query in Databricks?Thanks in advance.

Data Engineering
Reply
  • 1852 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 1 kudos

Hi, please elaborate on the issue for us to help you resolve it.

  • 1 kudos
1 More Replies
sage5616
by • Valued Contributor
  • 6721 Views
  • 2 replies
  • 3 kudos

Resolved! Running local python code with arguments in Databricks via dbx utility.

I am trying to execute a local PySpark script on a Databricks cluster via dbx utility to test how passing arguments to python works in Databricks when developing locally. However, the test arguments I am passing are not being read for some reason. Co...

Data Engineering
Reply
  • 6721 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

You can pass parameters using dbx launch --parametersIf you want to define it in the deployment template please try to follow exactly databricks API 2.1 schema https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate (for examp...

  • 3 kudos
1 More Replies
Labels