- 2871 Views
- 4 replies
- 0 kudos
How do I distribute machine learning process in my spark data frame
Hi,I'm trying to use around 5 numerical features on 3.5 million rows to train and test my model with a spark data frame.My cluster has 60 nodes available but is only using 2. How can I distribute the process or make it for efficient and faster.My cod...
- 2871 Views
- 4 replies
- 0 kudos
- 0 kudos
@mohaimen_syed - can you please try using pyspark.ml implementation of randomForestClassifier instead of sklearn and see if it works. Below is an example - https://github.com/apache/spark/blob/master/examples/src/main/python/ml/random_forest_classif...
- 0 kudos
- 986 Views
- 1 replies
- 0 kudos
AutoMl experiment not start
Hi!Last week I was able to perform some experiments in autoML. Today even after completing all the values in the form, the start button stay greyed. (I try train with the same dataset and parameters that last week train ok, with no luck) have any con...
- 986 Views
- 1 replies
- 0 kudos
- 0 kudos
@Hypatia - can you please detailed out the issue further? What was changed between previous good run and now when entering the details on AutoML experiment page?
- 0 kudos
- 7535 Views
- 5 replies
- 3 kudos
Issues running Chroma function from Dolly Demo
Hello,I'm working through the Dolly demo that was released a few days ago. There are a few lines of code that I have had to alter in order to make it work in my environment. However, I've been trying to run similarity_search() on the generated Chroma...
- 7535 Views
- 5 replies
- 3 kudos
- 3 kudos
As written in the command output when pip installing the python packages, you should then run a `dbutils.library.restartPython()`. Please try the following commands:%pip install -U transformers langchain chromadb accelerate bitsandbytesdbutils.librar...
- 3 kudos
- 1334 Views
- 1 replies
- 3 kudos
Resolved! Not possible to start AutoML experiment because start button not clickable
Hey everyone,we were trying to start a new AutoML experiment but the 'Start AutoML' button won't activate. We've also noticed that the number of selected trainingset columns does'nt change even if some columns are unselected.Has anybody encountered t...
- 1334 Views
- 1 replies
- 3 kudos
- 3 kudos
This seems to be a Databricks-wide problem. Please see related ticket on https://community.databricks.com/t5/machine-learning/can-t-run-an-automl-experiment-because-button-is-greyed-out/td-p/57388
- 3 kudos
- 2036 Views
- 1 replies
- 0 kudos
inference table not working
Hi,I'm trying to enable inference table for my llama_2_7b_hf serving endpoint, however I'm getting the following error:"Inference tables are currently not available with accelerated inference." Anyone one have an idea on how to overcome this issue? C...
- 2036 Views
- 1 replies
- 0 kudos
- 0 kudos
From the information you provided, it seems like you are trying to enable inference tables for an existing endpoint. However, the error message suggests that this feature may not be supported with accelerated inference.If you have previously disabled...
- 0 kudos
- 2435 Views
- 1 replies
- 0 kudos
Model Serving via Unity Catalog
Hi everyone! Has anyone successfully deployed a model saved on Unity Catalog to Model Serving? I get:Event Log: Served model creation failed for served model 'model', config version 15. Error message: Container creation failed. Please see build logs ...
- 2435 Views
- 1 replies
- 0 kudos
- 0 kudos
We are facing the same issue.
- 0 kudos
- 4999 Views
- 3 replies
- 5 kudos
Resolved! Access denied error to S3 bucket while running Kinesis spark streaming.
I get this below error while trying to simulate kinesis streams as mentioned in Databricks documentation at https://docs.databricks.com/getting-started/streaming.htmlError:java.nio.file.AccessDeniedException:Amazon S3; Status Code: 403; Error Code: A...
- 4999 Views
- 3 replies
- 5 kudos
- 5 kudos
If you do spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY_ID) + secret with any other secret that has less access than your default one this sometimes happens, so running those commands but with your normal secre...
- 5 kudos
- 1666 Views
- 1 replies
- 1 kudos
DLT UC for ML use cases
Hi Team,we have use case to run ML use case using DLT UC, we are facing library issues Error: INVALID ARGUMENT: no module named ‘importlib_metadata’we installed them manually by passing import in notebooks, but we have lot of missing libraries and we...
- 1666 Views
- 1 replies
- 1 kudos
- 1 kudos
@Retired_mod do we have any support of shared cluster for ML in near roadmap please
- 1 kudos
- 920 Views
- 1 replies
- 0 kudos
Unable to create an endpoint serving for transformer model (hugginface)
Hello I am trying to create a text classification model based on this blog https://www.databricks.com/blog/rapid-nlp-development-databricks-delta-and-transformers and the notebook accelerator. I just changed the model to take a french bert but i cann...
- 920 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Anasse, the LLM landscape has changed drastically since that blog was released in mid-2022. We have new, updated guidance which you can find here (make sure you check out the Next Steps section for the RAG Demo link as well). Additionally, if y...
- 0 kudos
- 1746 Views
- 1 replies
- 0 kudos
Custom deployment of LLM model in Databricks
Can we deploy our own Custom LLM model in Databricks? If anyone has any material or link, please share with me.
- 1746 Views
- 1 replies
- 0 kudos
- 1132 Views
- 0 replies
- 0 kudos
Save ML model in the registry fails
Using `mlflow.pyfunc.log_model` to save a model in the model registry fails with the following error: `PicklingError: Can't pickle <built-in function input>: it's not the same object as builtins.input`I've reproduced it locally and got to the conclus...
- 1132 Views
- 0 replies
- 0 kudos
- 1909 Views
- 0 replies
- 0 kudos
serving-endpoints: setting non secret environment variables
I want to set environment variables for serving-endpoints that are not being read as secrets by using the DB API. As an example I want to set the OPENAI_API_BASE for a LangChain model. Is this possible?
- 1909 Views
- 0 replies
- 0 kudos
- 2048 Views
- 3 replies
- 2 kudos
Databricks Observability - Sample / Pre-defined queries / notebooks for capturing the needed metrics from the Overwatch data model.
We are exploring Overwatch for data bricks environment monitoring. While we understand that different types of metrics (Audit and cluster) can be fetched from the overwatch data model at an high level. Are there any pre-defined queries / jump start n...
- 2048 Views
- 3 replies
- 2 kudos
- 2 kudos
Overwatch has 44+ prebuilt Dashboards you can get started just ask for it in https://github.com/databrickslabs/overwatch as an user question you will get an response.
- 2 kudos
- 1450 Views
- 0 replies
- 0 kudos
ML Flow until January 24
Hi! When i was creating a new endpoint a have this alert CREATE A MODEL SERVING ENDPOINT TO SERVE YOUR MODEL BEHIND A REST API INTERFACE. YOU CAN STILL USE LEGACY ML FLOW MODEL SERVING UNTIL JANUARY 2024 I don't understand if my Legacy MLFlow Model ...
- 1450 Views
- 0 replies
- 0 kudos
- 1205 Views
- 1 replies
- 0 kudos
how to use databricks foundation models outside databricks environment?
i want to use below code outside databricks environment without using cluster .getting issue while i am using chat_model.predict().how to authenticate this outside databricks or any other way to this? # Test Databricks Foundation LLM modelfrom langc...
- 1205 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you perhaps talking about Model Serving ?https://docs.databricks.com/en/machine-learning/model-serving/index.html#requirementsThis is how you configure it https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoin...
- 0 kudos
