- 4351 Views
- 4 replies
- 2 kudos
Uploaded Docker image into cluster. Used cluster for MLFlow experiment, but no experiment is logged/there are no experiment runs. Why is this?
Hi! So I used this MLFlow experiment I found from the databricks website: https://docs.databricks.com/_static/notebooks/machine-learning-with-unity-catalog.htmlAnd I created this cluster using a custom Docker image I created myself: Usually when I c...
- 4351 Views
- 4 replies
- 2 kudos
- 2 kudos
Have you tried the steps mentioned in the below URL:https://docs.databricks.com/clusters/custom-containers.html#step-3-launch-your-cluster
- 2 kudos
- 3056 Views
- 7 replies
- 6 kudos
Why this Databricks ML code gets stuck?
I could not paste the code here because of the some word not allowed, so I have to paste it elsewhere.Below is OK:https://justpaste.it/8xcr9But below gets stuck:https://justpaste.it/8nydtand it keeps looping and running...
- 3056 Views
- 7 replies
- 6 kudos
- 6 kudos
Hey @THIAM HUAT TAN​ Hope all is well! Just wanted to check in if you were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 6 kudos
- 1495 Views
- 0 replies
- 0 kudos
MLflow model serving: KeyError: 'python_function'
Hello, I am training a logistic regression on text with the help of an tf-idf vectorizer.This is done with MLflow and sklearn in databricks.The model itself is trained successfully in databricks and it is possible to accomplish predictions within the...
- 1495 Views
- 0 replies
- 0 kudos
- 2668 Views
- 4 replies
- 0 kudos
Why is there a limit in /2.1/jobs/list?
I detected that there ist a limit of 25 in /2.1/jobs/list. While from what i know /2.0/jobs/list had no limit? Why is this the case? Is it planned to increase the limit at some point?I know that the offset concept exist, but from my standpoint that i...
- 2668 Views
- 4 replies
- 0 kudos
- 0 kudos
Jobs API 2.1 jobs list responses will be capped at a limit of 25. With the introduction of pagination in Jobs API 2.1, and to stay in-line with providing increased stability, a limit was introduced on the amount Jobs API 2.1 jobslist responses.
- 0 kudos
- 2215 Views
- 2 replies
- 0 kudos
Unable to create model version using rest api on Managed MLFlow on GCP. Getting a Failed Registration.
I am trying to use Managed MLFlow as tracking server on GCP. I use rest apis to connect with the MLFLOW using Databricks token.I can create experiment and even the model but what when I try to create a model version I run into this following error. ...
- 2215 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Shounak Roychowdhury​, Just a friendly follow-up. Do you still need help or you were able to find the solution to this question? please let us know
- 0 kudos
- 2558 Views
- 4 replies
- 2 kudos
Save VM cost when using Rest API deploying models for online inference
ADB allows us to deploy the models for online inference through a REST API. To that aim ADB creates a VM dedicated to serve a specific model. Data Scientist can create and deploy several models for testing online inference, thus the cost can rapidly ...
- 2558 Views
- 4 replies
- 2 kudos
- 2 kudos
Hey @John Wilmar Herrera Gil​ Thank you so much for getting back to us. We really appreciate your time.Wish you a great Databricks journey ahead!
- 2 kudos
- 5436 Views
- 4 replies
- 5 kudos
Submitting multiple parallel jobs to the same job cluster causes Azure vCPU quota manager to count the clusters vCPUs on each invocation
I have an ADF pipeline which invokes a Databricks job six times in parallel. My assumption is all jobs get routed to the same job cluster which then deals with all the invocations in parallel. This was working fine when I had five sources, when I add...
- 5436 Views
- 4 replies
- 5 kudos
- 16486 Views
- 1 replies
- 5 kudos
Resolved! ingest a .csv file with spaces in column names using Delta Live into a streaming table
How do I ingest a .csv file with spaces in column names using Delta Live into a streaming table? All of the fields should be read using the default behavior .csv files for DLT autoloader - as strings. Running the pipeline gives me an error about in...
- 16486 Views
- 1 replies
- 5 kudos
- 5 kudos
After additional googling on "withColumnRenamed", I was able to replace all spaces in column names with "_" all at once by using select and alias instead:@dlt.view( comment="" ) def vw_raw(): return ( spark.readStream.format("cloudF...
- 5 kudos
- 1704 Views
- 1 replies
- 3 kudos
Feature Store - Feature Lookup with Filter
I am working with feature store to save the engineered features. However, for the specific case we have lots of feature table and lot of separate target variables on which we want to train separate models. Now for each of these model, we can leverage...
- 1704 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for taking the time to let us know how to make Databricks even better! @Mayank Srivastava​ I love that you included a real-life example as well. I think I know the right PM at Databricks that will be interested in this input. Thanks again for...
- 3 kudos
- 1158 Views
- 1 replies
- 0 kudos
hi Team, I am facing an issue when deploying the databricks model into AWS Sagemaker. Kindly check the below error and advice me on this. Traceback (...
hi Team, I am facing an issue when deploying the databricks model into AWS Sagemaker. Kindly check the below error and advice me on this.Traceback (most recent call last): File "<string>", line 1, in <module> File "/miniconda/lib/python3.9/site-pack...
- 1158 Views
- 1 replies
- 0 kudos
- 0 kudos
Any update on the above issue?
- 0 kudos
- 851 Views
- 0 replies
- 2 kudos
Unity Catalog Webinar: Join us to learn what's new, and what’s coming in Unity Catalog Governance for Data and AI is complex. Databricks Unity Cat...
Unity Catalog Webinar: Join us to learn what's new, and what’s coming in Unity CatalogGovernance for Data and AI is complex. Databricks Unity Catalog provides a unified governance solution for all data and AI assets on any cloud, empowering data team...
- 851 Views
- 0 replies
- 2 kudos
- 2340 Views
- 0 replies
- 0 kudos
How to identify S3 object type (directory or file) created by Databricks?
The issue context is Delta Lake connector in Trino https://github.com/trinodb/trino/issues/13017Trino identifies S3 object as a directory or a file using Content-Type header. Other query engines set application/x-directory in case of directories, bu...
- 2340 Views
- 0 replies
- 0 kudos
- 2611 Views
- 1 replies
- 2 kudos
Resolved! Store a secret only accessible to the current user
During an interactive notebook session, I want a user to be able to retrieve a secret specific to that user. I haven't decided on storage mechanisms, but I'm open to storage mechanisms that can scalably authorize access to a single user and that I ca...
- 2611 Views
- 1 replies
- 2 kudos
- 2 kudos
I ended up using Databricks Secrets as the storage mechanism after learning from my account rep that the limit is soft and we can request a higher scope limit. In this case, each user gets a dedicated scope and no other users have access.
- 2 kudos
- 3355 Views
- 4 replies
- 1 kudos
Resolved! ML Practioner | ml 09 - automl notebook | error on importing databricks.automl
executing the following code...from databricks import automlsummary = automl.regress(train_df, target_col="price", primary_metric="rmse", timeout_minutes=5, max_trials=10)generates the error...ImportError: cannot import name 'automl' from 'databricks...
- 3355 Views
- 4 replies
- 1 kudos
- 2316 Views
- 2 replies
- 1 kudos
How to input initial centroids to K-Means or GMM Clustering in SparkML ?
Hi, I want to use KMeans Model or Gaussian Mixture Model algorithm for clustering using the SparkML library, in which I want to specify the initial centroids. The option of giving initial centroids is there in the Spark MLlib (RDD based APIs) however...
- 2316 Views
- 2 replies
- 1 kudos
