- 3978 Views
- 3 replies
- 2 kudos
How to proper use Databricks MLFlow Managed tracker/register with Databricks Workflow
Hey.I'm building a DevOps/MLOps pipeline to train/register simple scikit learn model.I created a simple Databricks Workflow to execute training and register task on specific .git branch. (Workflow is setup with Databricks Repo on specifc branch, with...
- 3978 Views
- 3 replies
- 2 kudos
- 2 kudos
I had same issue while trying to call notebook from workflow. I was able to do what you did. But it needs new experiment name for each run, so I had to do this:# Set the experimentexperiment_name = f"/Workspace/MLOps/{env}/experiment/{experiment}_{ti...
- 2 kudos
- 6517 Views
- 2 replies
- 4 kudos
Dynamic variable and multi-instance tasks.
1. How to pass dynamic variable values like "sysdate" to a job parameters, so that it will automatically take the updated values on the fly.2. How to run multiple instance of set of tasksin a job (for different parameters). For e.g the same pipeline ...
- 6517 Views
- 2 replies
- 4 kudos
- 4 kudos
Hey Maverick1,Did you find a solution for your second question?I have also same approach. In databricks, it has workflows, job clusters, tasks etc.I'm trying to do creating one job cluster with one configuration or specification which has a workflow ...
- 4 kudos
- 12904 Views
- 5 replies
- 2 kudos
Run one workflow dynamically with different parameter and schedule time.
Can we run one workflow for different parameters and different schedule time. so that only one workflow can executed for different parameters we do not have to create that workflow again and again. or we can say Is there any possibility to drive work...
- 12904 Views
- 5 replies
- 2 kudos
- 2 kudos
Update / Solved: Using CLI on Linux/MacOS: Send in the sample json with job_id in it. databricks jobs run-now --json '{ "job_id":<job-ID>, "notebook_params": { <key>:<value>, <key>:<value> }}' Using CLI on Windows: Send in the sample json w...
- 2 kudos
- 2610 Views
- 4 replies
- 0 kudos
Why is there a limit in /2.1/jobs/list?
I detected that there ist a limit of 25 in /2.1/jobs/list. While from what i know /2.0/jobs/list had no limit? Why is this the case? Is it planned to increase the limit at some point?I know that the offset concept exist, but from my standpoint that i...
- 2610 Views
- 4 replies
- 0 kudos
- 0 kudos
Jobs API 2.1 jobs list responses will be capped at a limit of 25. With the introduction of pagination in Jobs API 2.1, and to stay in-line with providing increased stability, a limit was introduced on the amount Jobs API 2.1 jobslist responses.
- 0 kudos
- 1991 Views
- 2 replies
- 3 kudos
- 1991 Views
- 2 replies
- 3 kudos
- 3 kudos
No error, just seeing the EXPAND DISK in cluster event logs. This is just a regular spark application. I am not sure if the cloud storage matters - a spark application uses it as input and output.
- 3 kudos
- 3684 Views
- 1 replies
- 32 kudos
Databricks Roadmap Azure There are a lot of excitement new features coming in 2022. I tried to put them all on one list: Unity catalog (seems that it ...
Databricks Roadmap AzureThere are a lot of excitement new features coming in 2022. I tried to put them all on one list:Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)Control metastore, unity creatio...
- 3684 Views
- 1 replies
- 32 kudos