- 1439 Views
- 1 replies
- 0 kudos
ML for predictive maintenence use cases?!
How are you using ML to help determine predictive maintenance needs for your systems or operations?
- 1439 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @LDogg, You can do predictive maintenance something like this:Start by streaming sensor or IoT data like temperature, pressure, vibration, etc. into Delta Lake using tools like Structured Streaming or Delta Live Tables.Next, we can process and eng...
- 0 kudos
- 9149 Views
- 6 replies
- 1 kudos
How can I use the feature store for time series out of sample prediction?
For instance, have a new model trained every Saturday with training data up to the previous Fri, and use such model to predict daily the following week?In the same context, if the features are keyed by date, could I create a training set with a diffe...
- 9149 Views
- 6 replies
- 1 kudos
- 1 kudos
Hello, I just came across this and I have a similar question. I am quite new to Databricks and the feature store, but I wanted to use it, however, I am having some difficulty figuring out what specifically I can do.In my case I am using XGBoost regre...
- 1 kudos
- 5462 Views
- 3 replies
- 0 kudos
spark_session invocation from executor side error, when using sparkXGBregressor and fe client
Hi I have created a model and pipeline using xgboost.spark's sparkXGBregressor and pyspark.ml's Pipeline instance. However, i run into a "RuntimeError: _get_spark_session should not be invoked from executor side." when i try to save the predictions i...
- 5462 Views
- 3 replies
- 0 kudos
- 0 kudos
I have tried the recommended solution logging the model with the parameter flavor = mlflow.pyfunc but it returns the following error when logging the model using FeatureEngineeringClient.log_model function:_validate_function_python_model(python_mode...
- 0 kudos
- 3448 Views
- 3 replies
- 0 kudos
filedescriptor out of range in select()
Hi AllI am running a trining job using Mlflow and Databricks recipe. In the recipe.train step the training starts an experiment and runs for 350 epochs. After the 350 epochs are completed and I try to log the artifacts, the process gets stuck for a l...
- 3448 Views
- 3 replies
- 0 kudos
- 0 kudos
Than you for replying. I don't log any artifact during every epoch. I only log metrics every epoch. I try to log all artifacts at the end of training. Which is why I see the experiment finishing successfully and then these errors happening. How can I...
- 0 kudos
- 3480 Views
- 3 replies
- 0 kudos
Not able to run Pipeline Model load functions unity catalog cluster
ISSUE -- Not able to run PipelineModel load functions unity catalog clusterERROR --[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `sparkContext` is not supported in Spark Connect as it depends on the JVM. If you need to use this attribute, do not use Spark ...
- 3480 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks for your reply LRALVA, When i tried to run mlflow.spark.log_model(pipeline_model, "spark_pipeline_model") on my already saved model which was saved using random forest a long back. log_model gives me error that model is not a spark flavor. So ...
- 0 kudos
- 4223 Views
- 2 replies
- 1 kudos
Resolved! Serving Endpoint: Container Image Creation Fails
For my RAG use case, I've registered my langchain chain as a model to Unity Catalog. When I'm trying to serve the model, container image creation fails with the following error in the build log:[...] #16 178.1 Downloading langchain_core-0.3.17-py3-no...
- 4223 Views
- 2 replies
- 1 kudos
- 1 kudos
I was able to solve the problem by adding python-snappy==0.7.3 to the requirements.
- 1 kudos
- 12954 Views
- 8 replies
- 4 kudos
Possible to use `params` argument of `mlflow.pyfunc.PythonModel` deployed to Databricks endpoint?
I'm deploying a custom model using the `mlflow.pyfunc.PythonModel` class as described here. My model uses the `params` argument in the `predict` method to allow the user to choose some aspects of the model at inference time. For example:class CustomM...
- 12954 Views
- 8 replies
- 4 kudos
- 4 kudos
Haven't found a way how to make the parameters show up on the model page in databricks or be served as part of the "use now" button on the deployed endpoint.
- 4 kudos
- 2596 Views
- 1 replies
- 0 kudos
How to export genie externally
Is it possible to deploy the Genie model externally from Databricks and integrate it as a standalone chatbot through an API interface?
- 2596 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shivsingh , This is possible through the Genie API. You can find documentation here: https://docs.databricks.com/api/workspace/genie , and here is a blog post with some best practices: https://www.databricks.com/blog/genie-conversation-apis-publi...
- 0 kudos
- 4244 Views
- 2 replies
- 2 kudos
workflow not pickingup correct host value (While working with MLflow model registry URI)
Exception: mlflow.exceptions.MlflowException: An API request to https://canada.cloud.databricks.com/api/2.0/mlflow/model-versions/list-artifacts failed due to a timeout. The error message was: HTTPSConnectionPool(host='canada.cloud.databricks.com', p...
- 4244 Views
- 2 replies
- 2 kudos
- 5503 Views
- 2 replies
- 0 kudos
Model Serving Endpoint: Cuda-OOM for Custom Model
Hello all,I am tasked to evaluate a new LLM for some use-cases. In particular, I need to build a POC for a chat bot based on that model. To that end, I want to create a custom Serving Endpoint for an LLM pulled from huggingfaces. The model itself is...
- 5503 Views
- 2 replies
- 0 kudos
- 0 kudos
Here are some suggestions: 1. Update coda.yaml. Replace the current config with this optimized version: channels: - conda-forge dependencies: - python=3.10 # 3.12 may cause compatibility issues - pip - pip: - mlflow==2.21.3 - torch...
- 0 kudos
- 3701 Views
- 1 replies
- 0 kudos
Latest model in unity catalog model
I'm trying to train multiple models in one unity catalog. However, there is no way to automatically choose the latest version of these models when running the project? Do I always need to choose based on the alias or version number that I already kno...
- 3701 Views
- 1 replies
- 0 kudos
- 0 kudos
When using Databricks Unity Catalog for managing multiple models and their versions, there is no built-in automatic mechanism that dynamically selects the "latest" version of a model when running a project. To automatically choose the latest versio...
- 0 kudos
- 4840 Views
- 3 replies
- 0 kudos
Resolved! Error in creating a serving endpoint: registered model not found
I have registered a custom model which loads another model in the load_context method. Everything works fine when I load (with mlflow.pyfunc.load_model) and use the model in a notebook. When I try to create a serving endpoint for it I keep becoming t...
- 4840 Views
- 3 replies
- 0 kudos
- 0 kudos
It is registered in the Unity Catalog. I have found a complete other solution now. With the help of TransformedTargetRegressor I don't need a separate normalisation step anymore and therefore don't load a model in load_context anymore.
- 0 kudos
- 3088 Views
- 3 replies
- 1 kudos
Resolved! How to install Tensorflow 1 based compute or packages in Databricks
I want to install Tensorflow 1 based packages along with python 3.7 etc. I tried multiple ways including using a custom docker image. But nothing seems to workAlso I know that the minimum runtime version available in Databricks is 10.4So is it possib...
- 3088 Views
- 3 replies
- 1 kudos
- 1 kudos
@aswinkks You're right to be cautious — as of 2025, using TensorFlow 1.x in modern environments likeDatabricks has become increasingly difficult, if not practically unsupported, due to the combination of:- Deprecation of Python 3.7- TensorFlow 1.x be...
- 1 kudos
- 30377 Views
- 6 replies
- 7 kudos
Resolved! Access the environment variable from the custom container base cluster
Hi Databricks Community, I want to set environment variables for all clusters in my workspace. The goal is to the have environment variable, available in all notebooks executed on the cluster.The environment variable is generated in global init scrip...
- 30377 Views
- 6 replies
- 7 kudos
- 7 kudos
Thanks @Lukasz Lu​ - that worked for me as well. When I used the following script:#!/bin/bash echo MY_TEST_VAR=value1 | tee -a /etc/environment >> /databricks/spark/conf/spark-env.shfor non-docker clusters, MY_TEST_VAR shows up twice in ` /databrick...
- 7 kudos
- 7379 Views
- 2 replies
- 3 kudos
Resolved! Using Datbricks Connect with serverless compute and MLflow
Hi all,I have been using databricks-connect with serverless compute to develop and debug my databricks related code. It worked great so far. Now I started integrating ML-Flow in my workflow, and I am encountering an issue. When I run the following co...
- 7379 Views
- 2 replies
- 3 kudos
- 3 kudos
The error you are encountering, pyspark.errors.exceptions.connect.AnalysisException: [CONFIG_NOT_AVAILABLE] Configuration spark.mlflow.modelRegistryUri is not available. SQLSTATE: 42K0I, is a known issue when using MLflow with serverless clusters in ...
- 3 kudos
