cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Science & Machine Learning

Forum Posts

yopbibo
by Contributor II
  • 1887 Views
  • 2 replies
  • 5 kudos

Deploy a ML model, trained and registered in Databricks to AKS

Hi,I can train, registered a ML Model in my Datbricks Workspace.Then, to deploy it on AKS, I need to register the model in Azure ML, and then, deploy to AKS.Is it possible to skip the Azure ML step?I would like to deploy directly into my AKS instance...

  • 1887 Views
  • 2 replies
  • 5 kudos
Latest Reply
Debayan
Databricks Employee
  • 5 kudos

Hi, Thanks for reaching out to Databricks. Registering a model can be done, and it is not mentioned if it is optional or not in Microsoft documents. Reference : https://docs.microsoft.com/en-gb/azure/databricks/applications/mlflow/models#register-mod...

  • 5 kudos
1 More Replies
Somi
by New Contributor III
  • 6571 Views
  • 10 replies
  • 0 kudos

Resolved! How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...best_hyperparam = fmin(fn=CNN_HOF,                  space=space,                ...

  • 6571 Views
  • 10 replies
  • 0 kudos
Latest Reply
Dooley
Valued Contributor II
  • 0 kudos

This can happen when you try to serialize a keras model with an unserializable layer. What does your model look like? Also what is in that search space variable? What are you trying to optimize on?

  • 0 kudos
9 More Replies
Somi
by New Contributor III
  • 1267 Views
  • 3 replies
  • 0 kudos

No saved model after stopping the cluster.

I have saved a keras model in some directories in dbfs to load and retrain that with more data, etc. The problem is that when cluster stops and restarts, seems those directories and model are no longer available there and it starts training a new mod...

  • 1267 Views
  • 3 replies
  • 0 kudos
Latest Reply
Somi
New Contributor III
  • 0 kudos

Hi @Vidula Khanna​ I figured it out by replacing OS library module with dbutils utilities. It looks like mre compatible with DBFS.

  • 0 kudos
2 More Replies
Ashley1
by Contributor
  • 4620 Views
  • 5 replies
  • 5 kudos

Feature table: merge very slow

Hi All, We're just started to look at the feature store capabilities of Databricks. Our first attempt to create a feature table has resulted in very slow write. To avoid the time incurred by the feature functions I generated a dataframe with same...

Historical Spark UI for cluster 0622-013318-zoqth84b, driver 332737051535251367 - Details for Query 352
  • 4620 Views
  • 5 replies
  • 5 kudos
Latest Reply
Vidula
Honored Contributor
  • 5 kudos

Hi @Ashley Betts​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 5 kudos
4 More Replies
Giorgi
by Contributor
  • 1228 Views
  • 2 replies
  • 4 kudos

Resolved! Azure Data Factory: allocate resources per Notebook

I'm using Azure Data Factory to create pipeline of Databricks notebooks, something like this:[Notebook 1 - data pre-processing ] -> [Notebook 2 - model training ] -> [Notebook 3 - performance evaluation].Can I write some config file, that would allow...

  • 1228 Views
  • 2 replies
  • 4 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

I understand that, in your case, auto-scaling will take too much time.The simplest option is to use a different cluster for another notebook (and be sure that the previous cluster is terminated instantly).Another option is to use REST API 2.0/cluster...

  • 4 kudos
1 More Replies
confusedIntern
by New Contributor III
  • 3340 Views
  • 4 replies
  • 2 kudos

Uploaded Docker image into cluster. Used cluster for MLFlow experiment, but no experiment is logged/there are no experiment runs. Why is this?

Hi! So I used this MLFlow experiment I found from the databricks website: https://docs.databricks.com/_static/notebooks/machine-learning-with-unity-catalog.htmlAnd I created this cluster using a custom Docker image I created myself:  Usually when I c...

Screen Shot 2022-08-02 at 4.13.14 PM Screen Shot 2022-08-02 at 4.17.10 PM Screen Shot 2022-08-02 at 4.17.47 PM
  • 3340 Views
  • 4 replies
  • 2 kudos
Latest Reply
Debayan
Databricks Employee
  • 2 kudos

Have you tried the steps mentioned in the below URL:https://docs.databricks.com/clusters/custom-containers.html#step-3-launch-your-cluster

  • 2 kudos
3 More Replies
THIAM_HUATTAN
by Valued Contributor
  • 2341 Views
  • 7 replies
  • 6 kudos

Why this Databricks ML code gets stuck?

I could not paste the code here because of the some word not allowed, so I have to paste it elsewhere.Below is OK:https://justpaste.it/8xcr9But below gets stuck:https://justpaste.it/8nydtand it keeps looping and running...

  • 2341 Views
  • 7 replies
  • 6 kudos
Latest Reply
Vidula
Honored Contributor
  • 6 kudos

Hey @THIAM HUAT TAN​ Hope all is well! Just wanted to check in if you were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 6 kudos
6 More Replies
matebreeze
by New Contributor
  • 1224 Views
  • 0 replies
  • 0 kudos

MLflow model serving: KeyError: 'python_function'

Hello, I am training a logistic regression on text with the help of an tf-idf vectorizer.This is done with MLflow and sklearn in databricks.The model itself is trained successfully in databricks and it is possible to accomplish predictions within the...

  • 1224 Views
  • 0 replies
  • 0 kudos
Mirko
by Contributor
  • 2238 Views
  • 4 replies
  • 0 kudos

Why is there a limit in /2.1/jobs/list?

I detected that there ist a limit of 25 in /2.1/jobs/list. While from what i know /2.0/jobs/list had no limit? Why is this the case? Is it planned to increase the limit at some point?I know that the offset concept exist, but from my standpoint that i...

  • 2238 Views
  • 4 replies
  • 0 kudos
Latest Reply
User16873043099
Contributor
  • 0 kudos

Jobs API 2.1 jobs list responses will be capped at a limit of 25. With the introduction of pagination in Jobs API 2.1, and to stay in-line with providing increased stability, a limit was introduced on the amount Jobs API 2.1 jobslist responses.

  • 0 kudos
3 More Replies
sroychow
by New Contributor
  • 1826 Views
  • 2 replies
  • 0 kudos

Unable to create model version using rest api on Managed MLFlow on GCP. Getting a Failed Registration.

I am trying to use Managed MLFlow as tracking server on GCP. I use rest apis to connect with the MLFLOW using Databricks token.I can create experiment and even the model but what when I try to create a model version I run into this following error. ...

image.png
  • 1826 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Shounak Roychowdhury​, Just a friendly follow-up. Do you still need help or you were able to find the solution to this question? please let us know

  • 0 kudos
1 More Replies
jhonw901227
by New Contributor II
  • 2158 Views
  • 4 replies
  • 2 kudos

Save VM cost when using Rest API deploying models for online inference

ADB allows us to deploy the models for online inference through a REST API. To that aim ADB creates a VM dedicated to serve a specific model. Data Scientist can create and deploy several models for testing online inference, thus the cost can rapidly ...

  • 2158 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey @John Wilmar Herrera Gil​ Thank you so much for getting back to us. We really appreciate your time.Wish you a great Databricks journey ahead!

  • 2 kudos
3 More Replies
labromb
by Contributor
  • 4573 Views
  • 4 replies
  • 5 kudos

Submitting multiple parallel jobs to the same job cluster causes Azure vCPU quota manager to count the clusters vCPUs on each invocation

I have an ADF pipeline which invokes a Databricks job six times in parallel. My assumption is all jobs get routed to the same job cluster which then deals with all the invocations in parallel. This was working fine when I had five sources, when I add...

  • 4573 Views
  • 4 replies
  • 5 kudos
Latest Reply
labromb
Contributor
  • 5 kudos

Many thanks both for confirming.

  • 5 kudos
3 More Replies
vaver_3
by New Contributor III
  • 15250 Views
  • 1 replies
  • 5 kudos

Resolved! ingest a .csv file with spaces in column names using Delta Live into a streaming table

How do I ingest a .csv file with spaces in column names using Delta Live into a streaming table? All of the fields should be read using the default behavior .csv files for DLT autoloader - as strings. Running the pipeline gives me an error about in...

  • 15250 Views
  • 1 replies
  • 5 kudos
Latest Reply
vaver_3
New Contributor III
  • 5 kudos

After additional googling on "withColumnRenamed", I was able to replace all spaces in column names with "_" all at once by using select and alias instead:@dlt.view( comment="" ) def vw_raw(): return ( spark.readStream.format("cloudF...

  • 5 kudos
mayank347
by New Contributor II
  • 1416 Views
  • 1 replies
  • 3 kudos

Feature Store - Feature Lookup with Filter

I am working with feature store to save the engineered features. However, for the specific case we have lots of feature table and lot of separate target variables on which we want to train separate models. Now for each of these model, we can leverage...

  • 1416 Views
  • 1 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Thanks for taking the time to let us know how to make Databricks even better! @Mayank Srivastava​  I love that you included a real-life example as well. I think I know the right PM at Databricks that will be interested in this input. Thanks again for...

  • 3 kudos
Gopi0403
by New Contributor III
  • 901 Views
  • 1 replies
  • 0 kudos

hi Team, I am facing an issue when deploying the databricks model into AWS Sagemaker. Kindly check the below error and advice me on this. Traceback (...

hi Team, I am facing an issue when deploying the databricks model into AWS Sagemaker. Kindly check the below error and advice me on this.Traceback (most recent call last): File "<string>", line 1, in <module> File "/miniconda/lib/python3.9/site-pack...

  • 901 Views
  • 1 replies
  • 0 kudos
Latest Reply
Gopi0403
New Contributor III
  • 0 kudos

Any update on the above issue?

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels