- 2818 Views
- 4 replies
- 2 kudos
Problem serving a langchain model on Databricks
Hi, I've encountered a problem of serving a langchain model I just created successfully on Databricks.I was using the following code to set up a model in unity catalog:from mlflow.models import infer_signatureimport mlflowimport langchainmlflow.set_r...
- 2818 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi,The warnings/errors in the logs of the langchain model log process can give you a good hint, although it may be not that evident at first sight.It happened something similar to me - same error message, and the cause was having used an OpenAI model...
- 2 kudos
- 2408 Views
- 2 replies
- 0 kudos
Accessing Databricks Volumes from a Serving Endpoint Using a Custom Model Class in Unity Catalog
Hi everyone,I’m looking for accessing Unity Catalog (UC) Volumes from a Databricks Serving Endpoint. Here’s my current setup:I have a custom AI model class for inference, which I logged into Unity Catalog using mlflow.pyfunc.log_model.I’ve created a ...
- 2408 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey VELU1122,did you find a solution for it. We are struggling with the same problem currently. Thanks
- 0 kudos
- 1669 Views
- 4 replies
- 0 kudos
Resolved! FeatureEngineeringClient workspace id error
Hi, I am working from local notebook using vscode databricks extension.I am trying to use FeatureEngineeringClient, when I create data set training_set = fe.create_training_set( df=filtered_data_train, feature_lookups=payments_feat...
- 1669 Views
- 4 replies
- 0 kudos
- 0 kudos
I’ve done some additional research and found that the FeatureStoreClient is not officially supported when accessing a managed Databricks environment from an external IDE, even when using Databricks Connect. The client library is designed to operate w...
- 0 kudos
- 2361 Views
- 2 replies
- 2 kudos
"error_code":"INVALID_PARAMETER_VALUE","message":"INVALID_PARAMETER_VALUE: Failed to generate access
Hello everyone,I have an Azure Databricks subscription with my company, and I want to use external LLMs in databricks, like claude-3 or gemini. I managed to create a serving endpoint for Anthropic and I am able to use claude 3.But I want to use a Gem...
- 2361 Views
- 2 replies
- 2 kudos
- 1777 Views
- 1 replies
- 0 kudos
Resolved! Enabled the AI Builder Preview but unable to see the feature on the menu even after 3-4 hours
I am an account admin and enabled the beta feature. Does any additional permissions need to be added before I can see the feature on the workspace.
- 1777 Views
- 1 replies
- 0 kudos
- 0 kudos
Realized that our workspace is hosted in a different region. AI Builder is available for only couple of regions at the moment. I was able to spin up a new workspace and it works. Can close this thread
- 0 kudos
- 647 Views
- 0 replies
- 0 kudos
When does everyone utilize the model register?
Hi, I'm Yuki,I'm considering when I should use register_model.In my case, I'm running the training batch once a week and if the model is good, I want to update the champion.I have created the code to register the model if the score is the best.# star...
- 647 Views
- 0 replies
- 0 kudos
- 1503 Views
- 2 replies
- 0 kudos
spark_session invocation from executor side error, when using sparkXGBregressor and fe client
Hi I have created a model and pipeline using xgboost.spark's sparkXGBregressor and pyspark.ml's Pipeline instance. However, i run into a "RuntimeError: _get_spark_session should not be invoked from executor side." when i try to save the predictions i...
- 1503 Views
- 2 replies
- 0 kudos
- 0 kudos
Did you ever find a resolution to this? I've been running into the same error with a Spark XGBoost classification model, and haven't had any success in finding a solution. Setting it to a pyfunc model in logging resulted in an error, and clearly you ...
- 0 kudos
- 582 Views
- 1 replies
- 0 kudos
How to paralellize using R in Databricks notebook?
Hi!I'm using an R library, but it is only using one node, is there a way to paralellize it?Thanks in advance!
- 582 Views
- 1 replies
- 0 kudos
- 0 kudos
To parallelize computations in R while using a Databricks environment, you can utilize two main approaches: SparkR or sparklyr. Both allow you to run R code in a distributed manner across multiple nodes in a cluster. Hope this helps. Louis.
- 0 kudos
- 458 Views
- 1 replies
- 0 kudos
Not able to run end to end ML project on Databricks Trial
I started using Databricks trial version from today. I want to explore full end to end ML lifecycle on the databricks. I observed for the compute only 'serverless' option is available. I was trying to execute the notebook posted on https://docs.datab...
- 458 Views
- 1 replies
- 0 kudos
- 0 kudos
I can take up to 15 minutes for the serving endpoint to be created. Once you initiate the "create endpoint" chunk of code go and grab a cup of coffee and wait 15 minutes. Then, before you use it verify it is running (bottom left menu "Serving") by g...
- 0 kudos
- 564 Views
- 0 replies
- 0 kudos
Nested experiments and UC
Í have a general problem. I run a nested experiment in ML FLow, training and logging several models in a loop. Then I want to register the best in UC. No problem so far. But when I load the model I register and run prediction it dosen't work. If I o...
- 564 Views
- 0 replies
- 0 kudos
- 685 Views
- 0 replies
- 0 kudos
Databricks app and R shiny
Hello,I've been testing the Databricks app and have the follow questions:1. My organization currently uses Catalog Explorer instead of Unity Catalog. I want to develop a Shiny app and was able to run code from the template under New > App. However, t...
- 685 Views
- 0 replies
- 0 kudos
- 713 Views
- 0 replies
- 0 kudos
Experiences with CatBoost Spark Integration in Production on Databricks?
Hi Community,I am currently evaluating various gradient boosting options on Databricks using production-level data, including the CatBoost Spark integration (ai.catboost:catboost-spark).I would love to hear from others who have successfully used this...
- 713 Views
- 0 replies
- 0 kudos
- 594 Views
- 1 replies
- 0 kudos
Resolved! Exploring Serverless Features in Databricks for ML Use Cases
Hello, I need to develop some ML use case. I would like to understand if the serverless functionality unlocks any additional features or if it is mandatory for certain capabilities.Thank you!
- 594 Views
- 1 replies
- 0 kudos
- 0 kudos
Serverless functionality in Databricks is not mandatory for utilizing machine learning (ML) capabilities. However, it does unlock specific benefits and features that can enhance certain workflows. Here’s how serverless compute can add value, based on...
- 0 kudos
- 5555 Views
- 1 replies
- 4 kudos
Error when reading Excel file: "org.apache.poi.ooxml.POIXMLException: Strict OOXML isn't currently supported, please see bug #57699"
Hi,I want to read an Excel "xlsx" file. The excel file has several sheets and multi-row header. The original file format was "xlsm" and I changed the extension to "xlsx". I try the following code:filepath_xlsx = "dbfs:/FileStore/Sample_Excel/data.xl...
- 5555 Views
- 1 replies
- 4 kudos
- 4 kudos
copying the data onto a newer file solved my issue. Likely issue related to files metadata!
- 4 kudos
- 21311 Views
- 9 replies
- 6 kudos
Spark with LSTM
I am still lost on the Spark and Deep Learning model.If I have a (2D) time series that I want to use for e.g. an LSTM model. Then I first convert it to a 3D array and then pass it to the model. This is normally done in memory with numpy. But what hap...
- 21311 Views
- 9 replies
- 6 kudos
- 6 kudos
Same problem as @imgaboy here, is the solution was to save into table our inputs after formating them ready to feed the lstm and just turn 2d to 3d via datagenerator??
- 6 kudos
