- 11206 Views
- 10 replies
- 0 kudos
Resolved! How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object
I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...best_hyperparam = fmin(fn=CNN_HOF, space=space, ...
- 11206 Views
- 10 replies
- 0 kudos
- 0 kudos
This can happen when you try to serialize a keras model with an unserializable layer. What does your model look like? Also what is in that search space variable? What are you trying to optimize on?
- 0 kudos
- 2572 Views
- 3 replies
- 0 kudos
No saved model after stopping the cluster.
I have saved a keras model in some directories in dbfs to load and retrain that with more data, etc. The problem is that when cluster stops and restarts, seems those directories and model are no longer available there and it starts training a new mod...
- 2572 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Vidula Khanna​ I figured it out by replacing OS library module with dbutils utilities. It looks like mre compatible with DBFS.
- 0 kudos
- 8515 Views
- 5 replies
- 5 kudos
Feature table: merge very slow
Hi All, We're just started to look at the feature store capabilities of Databricks. Our first attempt to create a feature table has resulted in very slow write. To avoid the time incurred by the feature functions I generated a dataframe with same...
- 8515 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Ashley Betts​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 5 kudos
- 2672 Views
- 2 replies
- 4 kudos
Resolved! Azure Data Factory: allocate resources per Notebook
I'm using Azure Data Factory to create pipeline of Databricks notebooks, something like this:[Notebook 1 - data pre-processing ] -> [Notebook 2 - model training ] -> [Notebook 3 - performance evaluation].Can I write some config file, that would allow...
- 2672 Views
- 2 replies
- 4 kudos
- 4 kudos
I understand that, in your case, auto-scaling will take too much time.The simplest option is to use a different cluster for another notebook (and be sure that the previous cluster is terminated instantly).Another option is to use REST API 2.0/cluster...
- 4 kudos
- 7267 Views
- 4 replies
- 2 kudos
Uploaded Docker image into cluster. Used cluster for MLFlow experiment, but no experiment is logged/there are no experiment runs. Why is this?
Hi! So I used this MLFlow experiment I found from the databricks website: https://docs.databricks.com/_static/notebooks/machine-learning-with-unity-catalog.htmlAnd I created this cluster using a custom Docker image I created myself: Usually when I c...
- 7267 Views
- 4 replies
- 2 kudos
- 2 kudos
Have you tried the steps mentioned in the below URL:https://docs.databricks.com/clusters/custom-containers.html#step-3-launch-your-cluster
- 2 kudos
- 4480 Views
- 7 replies
- 6 kudos
Why this Databricks ML code gets stuck?
I could not paste the code here because of the some word not allowed, so I have to paste it elsewhere.Below is OK:https://justpaste.it/8xcr9But below gets stuck:https://justpaste.it/8nydtand it keeps looping and running...
- 4480 Views
- 7 replies
- 6 kudos
- 6 kudos
Hey @THIAM HUAT TAN​ Hope all is well! Just wanted to check in if you were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 6 kudos
- 2017 Views
- 0 replies
- 0 kudos
MLflow model serving: KeyError: 'python_function'
Hello, I am training a logistic regression on text with the help of an tf-idf vectorizer.This is done with MLflow and sklearn in databricks.The model itself is trained successfully in databricks and it is possible to accomplish predictions within the...
- 2017 Views
- 0 replies
- 0 kudos
- 3843 Views
- 4 replies
- 0 kudos
Why is there a limit in /2.1/jobs/list?
I detected that there ist a limit of 25 in /2.1/jobs/list. While from what i know /2.0/jobs/list had no limit? Why is this the case? Is it planned to increase the limit at some point?I know that the offset concept exist, but from my standpoint that i...
- 3843 Views
- 4 replies
- 0 kudos
- 0 kudos
Jobs API 2.1 jobs list responses will be capped at a limit of 25. With the introduction of pagination in Jobs API 2.1, and to stay in-line with providing increased stability, a limit was introduced on the amount Jobs API 2.1 jobslist responses.
- 0 kudos
- 3327 Views
- 2 replies
- 0 kudos
Unable to create model version using rest api on Managed MLFlow on GCP. Getting a Failed Registration.
I am trying to use Managed MLFlow as tracking server on GCP. I use rest apis to connect with the MLFLOW using Databricks token.I can create experiment and even the model but what when I try to create a model version I run into this following error. ...
- 3327 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Shounak Roychowdhury​, Just a friendly follow-up. Do you still need help or you were able to find the solution to this question? please let us know
- 0 kudos
- 3637 Views
- 4 replies
- 2 kudos
Save VM cost when using Rest API deploying models for online inference
ADB allows us to deploy the models for online inference through a REST API. To that aim ADB creates a VM dedicated to serve a specific model. Data Scientist can create and deploy several models for testing online inference, thus the cost can rapidly ...
- 3637 Views
- 4 replies
- 2 kudos
- 2 kudos
Hey @John Wilmar Herrera Gil​ Thank you so much for getting back to us. We really appreciate your time.Wish you a great Databricks journey ahead!
- 2 kudos
- 6956 Views
- 4 replies
- 5 kudos
Submitting multiple parallel jobs to the same job cluster causes Azure vCPU quota manager to count the clusters vCPUs on each invocation
I have an ADF pipeline which invokes a Databricks job six times in parallel. My assumption is all jobs get routed to the same job cluster which then deals with all the invocations in parallel. This was working fine when I had five sources, when I add...
- 6956 Views
- 4 replies
- 5 kudos
- 18566 Views
- 1 replies
- 5 kudos
Resolved! ingest a .csv file with spaces in column names using Delta Live into a streaming table
How do I ingest a .csv file with spaces in column names using Delta Live into a streaming table? All of the fields should be read using the default behavior .csv files for DLT autoloader - as strings. Running the pipeline gives me an error about in...
- 18566 Views
- 1 replies
- 5 kudos
- 5 kudos
After additional googling on "withColumnRenamed", I was able to replace all spaces in column names with "_" all at once by using select and alias instead:@dlt.view( comment="" ) def vw_raw(): return ( spark.readStream.format("cloudF...
- 5 kudos
- 2355 Views
- 1 replies
- 3 kudos
Feature Store - Feature Lookup with Filter
I am working with feature store to save the engineered features. However, for the specific case we have lots of feature table and lot of separate target variables on which we want to train separate models. Now for each of these model, we can leverage...
- 2355 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for taking the time to let us know how to make Databricks even better! @Mayank Srivastava​ I love that you included a real-life example as well. I think I know the right PM at Databricks that will be interested in this input. Thanks again for...
- 3 kudos
- 1685 Views
- 1 replies
- 0 kudos
hi Team, I am facing an issue when deploying the databricks model into AWS Sagemaker. Kindly check the below error and advice me on this. Traceback (...
hi Team, I am facing an issue when deploying the databricks model into AWS Sagemaker. Kindly check the below error and advice me on this.Traceback (most recent call last): File "<string>", line 1, in <module> File "/miniconda/lib/python3.9/site-pack...
- 1685 Views
- 1 replies
- 0 kudos
- 0 kudos
Any update on the above issue?
- 0 kudos
- 1422 Views
- 0 replies
- 2 kudos
Unity Catalog Webinar: Join us to learn what's new, and what’s coming in Unity Catalog Governance for Data and AI is complex. Databricks Unity Cat...
Unity Catalog Webinar: Join us to learn what's new, and what’s coming in Unity CatalogGovernance for Data and AI is complex. Databricks Unity Catalog provides a unified governance solution for all data and AI assets on any cloud, empowering data team...
- 1422 Views
- 0 replies
- 2 kudos
