Databricks Community

Deepak_Goldwyn · 08-18-2022

We are investigating how to pass parameter from Databricks Job to DLT pipeline. Our process orchestrator is Azure Data Factory from where we trigger the Databricks Job using Jobs API. As part of the 'run-now' request, we would like to pass a paramete...

Deepak_Goldwyn · 07-08-2022

We have written few python functions(methods within a class) and packaged them as a wheel library.In the as-is situation we use to install that wheel library in All-Purpose cluster that we already have created. It works fine.In the to-be situtation(D...

Deepak_Goldwyn · 06-17-2022

I am trying to create Databricks Jobs and Delta live table(DLT) pipelines by using Databricks API.I would like to have the JSON code of Jobs and DLT in the repository(to configure the code as per environment) and execute the Databricks API by passing...

Deepak_Goldwyn · 08-16-2022

Hi Jose,Yes it answered my question. I am indeed using JSON file to create Jobs and pipelinesThanks.

Deepak_Goldwyn · 07-11-2022

It said "it could not find the whl file"Upon investigation we found our library sits in nexus and the cluster environment variable should be setup.And when added the below in DLT pipeline settings json,"spark_env_vars": { "PIP_INDEX_URL": "<URL for ...

Deepak_Goldwyn · 07-11-2022

@Tomasz Bacewicz Thanks for your reply !We are using the below command as a fist cmd (cell) in the DLT notebook,%pip install /dbfs/dist/abnamro_acdpt_centraldatapoint-0.12.0.dev24-py3-none-any.whlFyi,When we try to manually install the same on the J...

Databricks Community

User Stats

User Activity

Pass parameter value from Job to DLT pipeline

DLT Pipeline and Job Cluster

Create Jobs and Pipelines in Workflows using API

Re: Create Jobs and Pipelines in Workflows using API

Re: DLT Pipeline and Job Cluster

Re: DLT Pipeline and Job Cluster