cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Science & Machine Learning

Forum Posts

Start a conversation
CharlesFlores
by • New Contributor II
  • 311 Views
  • 2 replies
  • 0 kudos

Serializing custom SparkMLlib Evaluator

Hi guys,We're facing a weird behavior or we're missing some configuration in our code. I've tried to find some information unsuccessfully. Let me try to explain our case, we have implemented a custom Evaluator in python using PySpark API, something l...

Machine Learning
Reply
  • 311 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @CharlesFlores,  The error message indicates that there’s an issue with the configuration value for fs.azure.account.key.To resolve this, you need to ensure that you’re using the correct authentication method for accessing your Azure Data Lake Sto...

  • 0 kudos
1 More Replies
larsr
by • New Contributor II
  • 292 Views
  • 1 replies
  • 0 kudos

Authentication model serving endpoint

Hi, I was wondering whether model serving endpoints support authentication with Azure Managed Identities.

Machine Learning
Reply
  • 292 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@larsr Databricks itself supports authentication through Managed Identity and Model Serving Endpoint requires bearer token, so yeah - i suppose it's doable.

  • 0 kudos
enkefalos-commu
by • New Contributor III
  • 1177 Views
  • 3 replies
  • 1 kudos

Unable to deploy phi-3 model due to packaging library

I am trying to deploy phi-3 model in databricks but getting below error while creating serving endpoint. Help us on this as soon as possible.

Machine Learning
Reply
  • 1177 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Erwanlc
New Contributor II
  • 1 kudos

Hello, I'm facing the same issue. No matter what I am trying, I end up with dependencies issues...

  • 1 kudos
2 More Replies
Miki
by • New Contributor II
  • 534 Views
  • 2 replies
  • 0 kudos

Logging signature slows down inference to a crawl

I am having a similar issue thislog signature and input data for Spark LinearRegression using mlflow v2.13.0 and using mlflow.pyfunc.log_model to log my model. Starting a new post here since there doesn't seem to be any follow up from the community o...

Machine Learning
mlflow
signature
Reply
  • 534 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
MohsenJ
Contributor
  • 0 kudos

@Miki can you please share you code for logging the signature with array types

  • 0 kudos
1 More Replies
rahuja
by • New Contributor III
  • 778 Views
  • 5 replies
  • 0 kudos

Sharing Output between different tasks for MLOps pipeline as a Databricks Jobs

Hello EveryoneWe are trying to create an ML pipeline on Databricks using the famous Databricks workflows. Currently our pipeline includes having 3 major components: Data Ingestion, Model Training and Model Testing. My question is whether it is possib...

Machine Learning
Reply
  • 778 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
rahuja
New Contributor III
  • 0 kudos

@Kaniz_Fatma @Hkesharwani  any updates?

  • 0 kudos
4 More Replies
SOlivero
by • New Contributor III
  • 245 Views
  • 0 replies
  • 0 kudos

Register Model mounted in S3

Hello!I'm having an issue registering a model saved in a mounted S3 bucket using mlflow.Let me give a little bit more context:1. First I mounted my S3 with all the corresponding IAM permissions:s3_bucket_name = f"s3a://{s3_bucket}"dbutils.fs.mount(so...

Machine Learning
Reply
  • 245 Views
  • 0 replies
  • 0 kudos
Giuseppina
by • New Contributor III
  • 2183 Views
  • 3 replies
  • 0 kudos

Resolved! Llm

Are LLMs really ready for production deployment?

Machine Learning
Reply
  • 2183 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
ryojikn
New Contributor III
  • 0 kudos

You should be careful while putting them to production without guardrails, perhaps using Mosaic AI gateway announced today that would aggregate these functionalities, it should be something to start. These are not the only things you should worry abo...

  • 0 kudos
2 More Replies
Anonymous
by • Not applicable
  • 2892 Views
  • 5 replies
  • 4 kudos

Generate and export dbt documentation from the Workflow dbt task to S3

I'm testing the Databricks Jobs feature with a dbt task and wanted to know if you had any advice for me for managing dbt documentation.I can use "dbt run" commands to run my models then "dbt docs generate" to generate the documentation. But is it pos...

Machine Learning
Reply
  • 2892 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
136039
New Contributor II
  • 4 kudos

How can I access these target files from the task itself ? I am trying to use dbt's state modifiers for detecting models that changed and only running models when the source freshness changed. Is there an easy way to store and use these state files i...

  • 4 kudos
4 More Replies
GGG_P
by • New Contributor III
  • 2188 Views
  • 2 replies
  • 2 kudos

How to proper use Databricks MLFlow Managed tracker/register with Databricks Workflow

Hey.I'm building a DevOps/MLOps pipeline to train/register simple scikit learn model.I created a simple Databricks Workflow to execute training and register task on specific .git branch. (Workflow is setup with Databricks Repo on specifc branch, with...

Machine Learning
Reply
  • 2188 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
BernardoC
New Contributor II
  • 2 kudos

Nice contribution!

  • 2 kudos
1 More Replies
UnniKAnat
by • New Contributor
  • 1042 Views
  • 1 replies
  • 1 kudos

ML Model

What's the best option to store your trained ML models

Machine Learning
Reply
  • 1042 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
mheffernan
New Contributor II
  • 1 kudos

Depending on how many you have, different solutions may be appropriate - and conveniently, you can use MLflow as a front end for most of these if you're working in Python. If you're working on personal projects, a local MLflow instance might be the r...

  • 1 kudos
JC3
by • New Contributor II
  • 536 Views
  • 2 replies
  • 0 kudos

Can view Model Registry using a Service Principal, but cannot load the model for inference.

I have a Service Principal (for M2M auth) with read access to a Databricks Model Registry. I can successfully search the registry (via the `WorkspaceClient`) and find the model that I want to load using (Python) APIs, but I cannot load the model for ...

Machine Learning
Reply
  • 536 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kumaran
Valued Contributor III
  • 0 kudos

Hello @JC3, Thank you for posting your question in the Databricks community. Is it possible to share with us the minimum reproducible code?

  • 0 kudos
1 More Replies
MightyMasdo
by • New Contributor
  • 237 Views
  • 1 replies
  • 0 kudos

Spark context not implemented Error when using Databricks connect

I am developing an application using databricks connect and when I try to use VectorAssembler I get the Error sc is not none Assertion Error. is there a workaround for this ?

Machine Learning
Reply
  • 237 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

@MightyMasdo could you please share the screenshot of the error along with the command?

  • 0 kudos
leolmz
by • New Contributor II
  • 490 Views
  • 3 replies
  • 0 kudos

Download model artifact with HTTP

Hi, I want to pass a link for Kserve to download a model registered in Mlflow, which uses an HTTP request method to do that (it can be downloaded directly from GitHub or HuggingFace). Will setting up an artifact store in S3 or other public storage se...

Machine Learning
Reply
  • 490 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Kumaran
Valued Contributor III
  • 0 kudos

Hi @leolmz, You can refer to the doc for downloading the model artifacts

  • 0 kudos
2 More Replies
c3
by • New Contributor II
  • 171 Views
  • 0 replies
  • 0 kudos

AutoML workflows will no longer run with job compute

We have a few workflows that have been running fine with job compute (runtime 14x).  They started failing on 6/3 with the following error: The cluster [xxx] is not an all-purpose cluster. existing_cluster_id only supports all-purpose cluster IDs. I w...

Machine Learning
Reply
  • 171 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels