- 1144 Views
- 4 replies
- 1 kudos
Data AI Summit 2024
My first Data + AI summit and it's been a great experience
- 1144 Views
- 4 replies
- 1 kudos
- 2503 Views
- 4 replies
- 0 kudos
Sharing Output between different tasks for MLOps pipeline as a Databricks Jobs
Hello EveryoneWe are trying to create an ML pipeline on Databricks using the famous Databricks workflows. Currently our pipeline includes having 3 major components: Data Ingestion, Model Training and Model Testing. My question is whether it is possib...
- 2503 Views
- 4 replies
- 0 kudos
- 1174 Views
- 0 replies
- 0 kudos
Register Model mounted in S3
Hello!I'm having an issue registering a model saved in a mounted S3 bucket using mlflow.Let me give a little bit more context:1. First I mounted my S3 with all the corresponding IAM permissions:s3_bucket_name = f"s3a://{s3_bucket}"dbutils.fs.mount(so...
- 1174 Views
- 0 replies
- 0 kudos
- 3550 Views
- 3 replies
- 0 kudos
Resolved! Llm
Are LLMs really ready for production deployment?
- 3550 Views
- 3 replies
- 0 kudos
- 0 kudos
You should be careful while putting them to production without guardrails, perhaps using Mosaic AI gateway announced today that would aggregate these functionalities, it should be something to start. These are not the only things you should worry abo...
- 0 kudos
- 4718 Views
- 4 replies
- 4 kudos
Generate and export dbt documentation from the Workflow dbt task to S3
I'm testing the Databricks Jobs feature with a dbt task and wanted to know if you had any advice for me for managing dbt documentation.I can use "dbt run" commands to run my models then "dbt docs generate" to generate the documentation. But is it pos...
- 4718 Views
- 4 replies
- 4 kudos
- 4 kudos
How can I access these target files from the task itself ? I am trying to use dbt's state modifiers for detecting models that changed and only running models when the source freshness changed. Is there an easy way to store and use these state files i...
- 4 kudos
- 1501 Views
- 1 replies
- 1 kudos
ML Model
What's the best option to store your trained ML models
- 1501 Views
- 1 replies
- 1 kudos
- 1 kudos
Depending on how many you have, different solutions may be appropriate - and conveniently, you can use MLflow as a front end for most of these if you're working in Python. If you're working on personal projects, a local MLflow instance might be the r...
- 1 kudos
- 1190 Views
- 2 replies
- 0 kudos
Can view Model Registry using a Service Principal, but cannot load the model for inference.
I have a Service Principal (for M2M auth) with read access to a Databricks Model Registry. I can successfully search the registry (via the `WorkspaceClient`) and find the model that I want to load using (Python) APIs, but I cannot load the model for ...
- 1190 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @JC3, Thank you for posting your question in the Databricks community. Is it possible to share with us the minimum reproducible code?
- 0 kudos
- 1158 Views
- 2 replies
- 0 kudos
Download model artifact with HTTP
Hi, I want to pass a link for Kserve to download a model registered in Mlflow, which uses an HTTP request method to do that (it can be downloaded directly from GitHub or HuggingFace). Will setting up an artifact store in S3 or other public storage se...
- 1158 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @leolmz, You can refer to the doc for downloading the model artifacts
- 0 kudos
- 1895 Views
- 1 replies
- 0 kudos
EasyOcr Endpoint not accepting inputs
Hi all! I am trying to create an endpoint for Easy OCR. I was able to create the experiment using a wrapper class with the code below: # import libraries import mlflow import mlflow.pyfunc import cloudpickle import cv2 import re import easyocr impo...
- 1895 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @John22, Thank you for posting your question on the Databricks community. First, are you able to infer the output within the notebook itself? Which cloud are you on AWS or Azure?
- 0 kudos
- 783 Views
- 0 replies
- 0 kudos
AutoML models not completing
Hello, Whilst using a cluster set-up running 14.3 LTS ML, 2-10 workers, worker and driver type of r5d.xlarge I am having issues creating a regression model on 700k rows and 80 factors (no high cardinality in any factor shown).The first phase of the e...
- 783 Views
- 0 replies
- 0 kudos
- 1863 Views
- 1 replies
- 0 kudos
DBRX - Serving endpoint failed - update timed out.
Hi,https://notebooks.databricks.com/demos/llm-rag-chatbot/index.htmlFollowing this tutorial I'm trying to serve an endpoint with DBRX model connected to my data in Vector Db.Without any problem I can log my model in Databricks with MLFlow and call th...
- 1863 Views
- 1 replies
- 0 kudos
- 919 Views
- 1 replies
- 0 kudos
mlops-stacks workflow testing vs staging
I'm a newbie to MLOps and abit confused about the use and the implementation of staging and testing environment in the mlops-stack template. as far as I understand the staging environment is where we run the integration test. But in the ci-cd pipelin...
- 919 Views
- 1 replies
- 0 kudos
- 0 kudos
@MohsenJOfficial Site wrote:I'm a newbie to MLOps and abit confused about the use and the implementation of staging and testing environment in the mlops-stack template. as far as I understand the staging environment is where we run the integration te...
- 0 kudos
- 9024 Views
- 5 replies
- 5 kudos
DLT with unity catalog and ML
We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog. When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User se...
- 9024 Views
- 5 replies
- 5 kudos
- 5 kudos
No updates as far as I am aware.You could make the workflow copying the data smart though and try to only do incremental updates, seems like a lot of effort though.
- 5 kudos
- 1151 Views
- 0 replies
- 0 kudos
Conditional email notification based on if statement - dbutils
Is it possible to trigger an email notification based on a conditional statement in Python without exiting the process?Specifically, I have a robustness check in my prediction pipeline that performs simple imputation when encountering missing data. T...
- 1151 Views
- 0 replies
- 0 kudos
- 4583 Views
- 4 replies
- 2 kudos
Best practice for model promotion so that models are not removed from previous stage
Hi,Using Model Registry to promote models is great. However, I am facing an issue, where multiple Databricks workspaces (SIT / UAT / Prod) use a model at various stages (Staging for SIT and UAT, Production for Prod workspace).We have a workflow runni...
- 4583 Views
- 4 replies
- 2 kudos
