- 1249 Views
- 1 replies
- 0 kudos
Latest model in unity catalog model
I'm trying to train multiple models in one unity catalog. However, there is no way to automatically choose the latest version of these models when running the project? Do I always need to choose based on the alias or version number that I already kno...
- 1249 Views
- 1 replies
- 0 kudos
- 0 kudos
When using Databricks Unity Catalog for managing multiple models and their versions, there is no built-in automatic mechanism that dynamically selects the "latest" version of a model when running a project. To automatically choose the latest versio...
- 0 kudos
- 3597 Views
- 3 replies
- 0 kudos
Resolved! Error in creating a serving endpoint: registered model not found
I have registered a custom model which loads another model in the load_context method. Everything works fine when I load (with mlflow.pyfunc.load_model) and use the model in a notebook. When I try to create a serving endpoint for it I keep becoming t...
- 3597 Views
- 3 replies
- 0 kudos
- 0 kudos
It is registered in the Unity Catalog. I have found a complete other solution now. With the help of TransformedTargetRegressor I don't need a separate normalisation step anymore and therefore don't load a model in load_context anymore.
- 0 kudos
- 2020 Views
- 3 replies
- 1 kudos
Resolved! How to install Tensorflow 1 based compute or packages in Databricks
I want to install Tensorflow 1 based packages along with python 3.7 etc. I tried multiple ways including using a custom docker image. But nothing seems to workAlso I know that the minimum runtime version available in Databricks is 10.4So is it possib...
- 2020 Views
- 3 replies
- 1 kudos
- 1 kudos
@aswinkks You're right to be cautious — as of 2025, using TensorFlow 1.x in modern environments likeDatabricks has become increasingly difficult, if not practically unsupported, due to the combination of:- Deprecation of Python 3.7- TensorFlow 1.x be...
- 1 kudos
- 27525 Views
- 6 replies
- 7 kudos
Resolved! Access the environment variable from the custom container base cluster
Hi Databricks Community, I want to set environment variables for all clusters in my workspace. The goal is to the have environment variable, available in all notebooks executed on the cluster.The environment variable is generated in global init scrip...
- 27525 Views
- 6 replies
- 7 kudos
- 7 kudos
Thanks @Lukasz Lu​ - that worked for me as well. When I used the following script:#!/bin/bash echo MY_TEST_VAR=value1 | tee -a /etc/environment >> /databricks/spark/conf/spark-env.shfor non-docker clusters, MY_TEST_VAR shows up twice in ` /databrick...
- 7 kudos
- 5500 Views
- 2 replies
- 3 kudos
Resolved! Using Datbricks Connect with serverless compute and MLflow
Hi all,I have been using databricks-connect with serverless compute to develop and debug my databricks related code. It worked great so far. Now I started integrating ML-Flow in my workflow, and I am encountering an issue. When I run the following co...
- 5500 Views
- 2 replies
- 3 kudos
- 3 kudos
The error you are encountering, pyspark.errors.exceptions.connect.AnalysisException: [CONFIG_NOT_AVAILABLE] Configuration spark.mlflow.modelRegistryUri is not available. SQLSTATE: 42K0I, is a known issue when using MLflow with serverless clusters in ...
- 3 kudos
- 2380 Views
- 3 replies
- 2 kudos
Error when uploading MLFlow artifacts to DBFS
Hi everyone,I'm attempting to use MLFlow experiment tracking from a local machine, but I'm encountering difficulties in uploading artifacts.I've tried a sample code as simple as the following.import mlflow import os os.environ["DATABRICKS_HOST"] = "...
- 2380 Views
- 3 replies
- 2 kudos
- 2 kudos
It is considered best practice not to store any production data or assets in DBFS (Databricks File System). The primary reason is that DBFS does not provide robust security controls-anyone with workspace access can potentially access items stored the...
- 2 kudos
- 5507 Views
- 4 replies
- 0 kudos
Resolved! FeatureEngineeringClient workspace id error
Hi, I am working from local notebook using vscode databricks extension.I am trying to use FeatureEngineeringClient, when I create data set training_set = fe.create_training_set( df=filtered_data_train, feature_lookups=payments_feat...
- 5507 Views
- 4 replies
- 0 kudos
- 0 kudos
I’ve done some additional research and found that the FeatureStoreClient is not officially supported when accessing a managed Databricks environment from an external IDE, even when using Databricks Connect. The client library is designed to operate w...
- 0 kudos
- 3131 Views
- 2 replies
- 2 kudos
"error_code":"INVALID_PARAMETER_VALUE","message":"INVALID_PARAMETER_VALUE: Failed to generate access
Hello everyone,I have an Azure Databricks subscription with my company, and I want to use external LLMs in databricks, like claude-3 or gemini. I managed to create a serving endpoint for Anthropic and I am able to use claude 3.But I want to use a Gem...
- 3131 Views
- 2 replies
- 2 kudos
- 2672 Views
- 1 replies
- 0 kudos
Resolved! Enabled the AI Builder Preview but unable to see the feature on the menu even after 3-4 hours
I am an account admin and enabled the beta feature. Does any additional permissions need to be added before I can see the feature on the workspace.
- 2672 Views
- 1 replies
- 0 kudos
- 0 kudos
Realized that our workspace is hosted in a different region. AI Builder is available for only couple of regions at the moment. I was able to spin up a new workspace and it works. Can close this thread
- 0 kudos
- 1228 Views
- 1 replies
- 0 kudos
How to paralellize using R in Databricks notebook?
Hi!I'm using an R library, but it is only using one node, is there a way to paralellize it?Thanks in advance!
- 1228 Views
- 1 replies
- 0 kudos
- 0 kudos
To parallelize computations in R while using a Databricks environment, you can utilize two main approaches: SparkR or sparklyr. Both allow you to run R code in a distributed manner across multiple nodes in a cluster. Hope this helps. Louis.
- 0 kudos
- 947 Views
- 1 replies
- 0 kudos
Not able to run end to end ML project on Databricks Trial
I started using Databricks trial version from today. I want to explore full end to end ML lifecycle on the databricks. I observed for the compute only 'serverless' option is available. I was trying to execute the notebook posted on https://docs.datab...
- 947 Views
- 1 replies
- 0 kudos
- 0 kudos
I can take up to 15 minutes for the serving endpoint to be created. Once you initiate the "create endpoint" chunk of code go and grab a cup of coffee and wait 15 minutes. Then, before you use it verify it is running (bottom left menu "Serving") by g...
- 0 kudos
- 1726 Views
- 1 replies
- 0 kudos
Resolved! Exploring Serverless Features in Databricks for ML Use Cases
Hello, I need to develop some ML use case. I would like to understand if the serverless functionality unlocks any additional features or if it is mandatory for certain capabilities.Thank you!
- 1726 Views
- 1 replies
- 0 kudos
- 0 kudos
Serverless functionality in Databricks is not mandatory for utilizing machine learning (ML) capabilities. However, it does unlock specific benefits and features that can enhance certain workflows. Here’s how serverless compute can add value, based on...
- 0 kudos
- 6718 Views
- 1 replies
- 4 kudos
Error when reading Excel file: "org.apache.poi.ooxml.POIXMLException: Strict OOXML isn't currently supported, please see bug #57699"
Hi,I want to read an Excel "xlsx" file. The excel file has several sheets and multi-row header. The original file format was "xlsm" and I changed the extension to "xlsx". I try the following code:filepath_xlsx = "dbfs:/FileStore/Sample_Excel/data.xl...
- 6718 Views
- 1 replies
- 4 kudos
- 4 kudos
copying the data onto a newer file solved my issue. Likely issue related to files metadata!
- 4 kudos
- 32165 Views
- 9 replies
- 6 kudos
Spark with LSTM
I am still lost on the Spark and Deep Learning model.If I have a (2D) time series that I want to use for e.g. an LSTM model. Then I first convert it to a 3D array and then pass it to the model. This is normally done in memory with numpy. But what hap...
- 32165 Views
- 9 replies
- 6 kudos
- 6 kudos
Same problem as @imgaboy here, is the solution was to save into table our inputs after formating them ready to feed the lstm and just turn 2d to 3d via datagenerator??
- 6 kudos
- 5005 Views
- 6 replies
- 3 kudos
Resolved! Nested runs don't group correctly in MLflow
How do I get MLflow child runs to appear as children of their parent run in the MLflow GUI, if I'm choosing my own experiment location instead of letting everything be written to the default experiment location?If I run the standard tutorial (https:/...
- 5005 Views
- 6 replies
- 3 kudos
- 3 kudos
OK, here's more info about what's wrong, and a solution.I used additional parameter logging to determine that no matter how I adjust the parameters of the inner call to ```mlflow.start_run()```the `experiment_id` parameter of the child runs differs f...
- 3 kudos
