cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Atul_Sharan
by New Contributor II
  • 3366 Views
  • 3 replies
  • 3 kudos

Resolved! Error Code: 3206 - Processed HTTP request failed.

The ADF(Azur Data Factory) pipelines jobs executing several Databricks Notebook activities in parallel have been failing regularly with the following error "Error Code: 3206 - Processed HTTP request failed." The issue gets resolved on its own upon re...

  • 3366 Views
  • 3 replies
  • 3 kudos
Latest Reply
willjoe
New Contributor III
  • 3 kudos

Method 1 - Close Conflicting Programsdown voteWhen you get a runtime error, keep in mind that it is happening due to programs that are conflicting with each other. The first thing you can do to resolve the problem is to stop these conflicting program...

  • 3 kudos
2 More Replies
sparkstreaming
by New Contributor III
  • 4348 Views
  • 7 replies
  • 6 kudos

Resolved! Rest API invocation for databricks notebook fails while invoking from ADF pipeline

In the current implementation a streaming databricks notebook needs to be started based on the configuration passed. Since the rest of databricks notebooks are being invoked by using ADF,it was decided to use ADF for starting these notebooks. Since t...

  • 4348 Views
  • 7 replies
  • 6 kudos
Latest Reply
Kaniz
Community Manager
  • 6 kudos

Hi @Prasanth KP​ , Just a friendly follow-up. Do you still need help, or @Hubert Dudek (Customer)​ and @Werner Stinckens​ 's responses help you to find the solution? Please let us know.

  • 6 kudos
6 More Replies
my_community2
by New Contributor III
  • 4005 Views
  • 10 replies
  • 1 kudos

Running notebooks on DataBricks in Azure blowing up all over since morning of Apr 5 (MST). Was there another poor deployment at DataBricks? This reall...

Running notebooks on DataBricks in Azure blowing up all over since morning of Apr 5 (MST). Was there another poor deployment at DataBricks? This really needs to stop. We are running premium DataBricks on Azure and calling notebooks from ADF.10.2 (inc...

image
  • 4005 Views
  • 10 replies
  • 1 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 1 kudos

@Maciej G​ try using the below init script to increase the repl timeout.-------------------------------------- #!/bin/bash cat > /databricks/common/conf/set_repl_timeout.conf << EOL {  databricks.daemon.driver.launchTimeout = 150 }EOL----------------...

  • 1 kudos
9 More Replies
irfanaziz
by Contributor II
  • 3918 Views
  • 4 replies
  • 0 kudos

Resolved! If two Data Factory pipelines are run at the same time or share a window of execution do they share the Databricks spark cluster(if both have the same linked service)? ( job clusters are those that are create on the go, defined in the linked service).

Continuing the above case, does that mean if i have several like 5 ADF pipelines scheduled regularly at the same time, its better to use an existing cluster as all of the ADF pipelines would share the same cluster and hence the cost will be lower?

  • 3918 Views
  • 4 replies
  • 0 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 0 kudos

for adf or job run we always prefer job cluster. but for streaming, you may consider using interactive cluster . but anyway you need to monitor the cluster load, if loads are high there will be chance to job slowness as well as failure. also data siz...

  • 0 kudos
3 More Replies
MattM
by New Contributor III
  • 1137 Views
  • 3 replies
  • 2 kudos

Resolved! Pricing Spot Instance vs New Job Cluster

We are running multiple Databricks job via ADF. I was wondering which option out of the below is a cheaper route for databricks notebook processing from ADF. When I create a ADF linked service, which should I use to lower my cost.New Job Cluster opti...

  • 1137 Views
  • 3 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

the instance pool will be cheaper if you use spot instances. But only if you size your instance pool correctly. (number of workers and scale down time)AFAIK you cannot use spot instances for job clusters in ADF

  • 2 kudos
2 More Replies
MoJaMa
by Valued Contributor II
  • 2540 Views
  • 1 replies
  • 1 kudos
  • 2540 Views
  • 1 replies
  • 1 kudos
Latest Reply
MoJaMa
Valued Contributor II
  • 1 kudos

Yes, you can pass parameters from ADF —> Azure Databricks.https://docs.microsoft.com/en-us/azure/data-factory/solution-template-databricks-notebook#how-to-use-this-templateYou can also pass values back from the Notebook --> ADF via the dbutils.notebo...

  • 1 kudos
MallikSunkara
by New Contributor II
  • 7169 Views
  • 4 replies
  • 0 kudos

how to pass arguments and variables to databricks python activity from azure data factory

how to pass arguments and variables to databricks python activity from azure data factory

  • 7169 Views
  • 4 replies
  • 0 kudos
Latest Reply
CristianIspan
New Contributor II
  • 0 kudos

try importing argv from sys. Then if you have the parameter added correctly in DataFactory you could get it in your python script typing argv[1] (index 0 is the file path).

  • 0 kudos
3 More Replies
Labels