- 1544 Views
- 1 replies
- 1 kudos
Resolved! No Spark Session Available Within Model Serving Environment
Hi,Is it possible to have a Spark session, that can be to used query the Unity Catalog etc, available within a Model Serving?I have an MLFlow Pyfunc model that needs to get data from a Feature Table as part of its `.predict()` method. See my earlier ...
- 1544 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @mharrison Creating a Spark session within a Model Serving environment is not directly supported, which is why you are encountering the Exception: No SparkSession Available! error. This limitation arises because the serving environment does not a...
- 1 kudos
- 26836 Views
- 3 replies
- 0 kudos
dbfs not found
Hi, I've saved a custom pyfunc and now I'm trying to load it in a pandas_udf. It works on small samples or if I repartition everything to 1 partition, but when I try to run it on a larger sample and distribute it across my cluster it fails repeatably...
- 26836 Views
- 3 replies
- 0 kudos
- 0 kudos
This problem can often be attributed to the model artifacts not being available on all the executors, especially in a distributed environment. Can you try using the dbutils.fs.refreshMounts() in your code? If the model is small enough, broadcast it t...
- 0 kudos
- 832 Views
- 2 replies
- 0 kudos
Feature Lookup Help
Hi,ContextI'm looking for help trying to get Unity Catalog Feature Lookup to work with my model how I need it to.I have a trained darts time series model that takes as input to its `.predict()` method both the history of the variable in question, and...
- 832 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for your response. It sounds like the 2nd approach is best for me, modifying the `predict()` method to perform the required history lookup.Is it possible to do this via the Feature Engineering client within that method, or should I simply quer...
- 0 kudos
- 903 Views
- 4 replies
- 1 kudos
How to serve a Unity Catalog ML model to external usage
Hello everyone I am following this notebook tutorial https://docs.databricks.com/en/machine-learning/manage-model-lifecycle/index.html#example-notebook Now I can register a machine learning model in Unity Catalog, but the tutorial only shows how to u...
- 903 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @johndoe99012 If the answer resolved your question, please consider marking it as the solution. It helps others in the community find answers more easily.
- 1 kudos
- 3568 Views
- 3 replies
- 0 kudos
Maximum wait time Databricks Model Serving
hi, hope you are fineI deployed a model 3 or 2 months ago using Databricks Serving and MLFlow. The model worked good using GPU from model serving.I stopped using it for some months and when I tried again deploying it, it has some errors.1. [FIXED] A ...
- 3568 Views
- 3 replies
- 0 kudos
- 5569 Views
- 3 replies
- 1 kudos
Resolved! Tuning `CrossValidator` spark job performance
I am running a 3-fold cross validation of an ML pipeline that utilizes `GBTClassifier` as the final step. It takes 18 hours to run and I am looking for feedback into how to improve the performance as I expect this to go faster.For context here is the...
- 5569 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @Assaad Mrad​ , So this looks like trying to decide between putting the pipeline in the cross validator or the cross validator in the pipeline. Since you are doing the polynomial expansion as part of the pipeline you might want to consider putt...
- 1 kudos
- 2647 Views
- 4 replies
- 0 kudos
Training Job Failure (Driver Error)
We have a new model training job that was running fine for a few days and then started failing. I have attached images for more details.I am wondering if 'can't reach driver cluster' is a red herring. It says the driver is healthy right before execut...
- 2647 Views
- 4 replies
- 0 kudos
- 0 kudos
In our case, we needed to correct our dependent libraries. We had an incorrect path referenced.
- 0 kudos
- 1261 Views
- 0 replies
- 0 kudos
Rolling predictions with FeatureEngineeringClient
I am performing a time series analysis, using a XGBoostRegressor with rolling predictions. I am doing so using the FeatureEngineeringClient (in combination with Unity Catalog), where I create and load in my features during training and inference, as ...
- 1261 Views
- 0 replies
- 0 kudos
- 871 Views
- 2 replies
- 0 kudos
Resolved! Vector search index stops at 45406
I am trying to create a vector search index for a table, but it stops at 45406 rows. I can see that the writeback table has all the records but the indexing stops. Is there a hard limit on index?
- 871 Views
- 2 replies
- 0 kudos
- 0 kudos
There are some limits that you can be hitting: Row Size for Delta Sync Index: The maximum row size is 100KB.Embedding Source Column Size for Delta Sync Index: The maximum size is 32764 bytes.Bulk Upsert Request Size Limit for Direct Vector Index: The...
- 0 kudos
- 3793 Views
- 2 replies
- 1 kudos
AutoML Runs Failing
After the Data Exploration notebook runs successfully, all AutoML trials fail without providing a source notebook. I have ensured that the training data labels have no null values or any labels with 16 or less occurrences associated with them. I cann...
- 3793 Views
- 2 replies
- 1 kudos
- 1014 Views
- 2 replies
- 0 kudos
Python running far slower than locally, even with large cluster and multiple workers
I have a notebook that is running extremely slowly even when I try to do pretty basic python functions. It is running far slower than locally no matter what I try, this is in spite of using a 32gb 4 core cluster with 4-8 workers. For context, my data...
- 1014 Views
- 2 replies
- 0 kudos
- 0 kudos
Please share more information, for example: Type of data sourceType of operations being executed (sharing code if possible)Timings of local runs and Databricks runs
- 0 kudos
- 3313 Views
- 2 replies
- 0 kudos
What's the recommended way to scale XGBoost/LGBM to datasets that don't fit in memory ?
I'm looking to scale xgboost to large datasets which won't fit in memory on a single large EC2 instance (billions to tens of billions of rows scale). I also require many of the bells & whistles of regular in-memory xgboost slash lightgbm including:Th...
- 3313 Views
- 2 replies
- 0 kudos
- 1238 Views
- 0 replies
- 0 kudos
AutoML "need to sample" not working as expected
tl; dr:When the AutoML run realizes it needs to do sampling because the driver / worker node memory is not enough to load / process the entire dataset, it fails. A sample weight column is NOT provided by me, but I believe somewhere in the process the...
- 1238 Views
- 0 replies
- 0 kudos
- 2889 Views
- 3 replies
- 5 kudos
Deploy a ML model, trained and registered in Databricks to AKS
Hi,I can train, registered a ML Model in my Datbricks Workspace.Then, to deploy it on AKS, I need to register the model in Azure ML, and then, deploy to AKS.Is it possible to skip the Azure ML step?I would like to deploy directly into my AKS instance...
- 2889 Views
- 3 replies
- 5 kudos
- 5 kudos
Is it still the case, can't we serve the model in Databricks. I am new to this, so I am just wondering the capabilities.
- 5 kudos
- 1160 Views
- 1 replies
- 0 kudos
Resolved! Serving Endpoint: Container Image Creation Fails
For my RAG use case, I've registered my langchain chain as a model to Unity Catalog. When I'm trying to serve the model, container image creation fails with the following error in the build log:[...] #16 178.1 Downloading langchain_core-0.3.17-py3-no...
- 1160 Views
- 1 replies
- 0 kudos
- 0 kudos
I was able to solve the problem by adding python-snappy==0.7.3 to the requirements.
- 0 kudos
