- 1368 Views
- 2 replies
- 0 kudos
Feature Lookup Help
Hi,ContextI'm looking for help trying to get Unity Catalog Feature Lookup to work with my model how I need it to.I have a trained darts time series model that takes as input to its `.predict()` method both the history of the variable in question, and...
- 1368 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for your response. It sounds like the 2nd approach is best for me, modifying the `predict()` method to perform the required history lookup.Is it possible to do this via the Feature Engineering client within that method, or should I simply quer...
- 0 kudos
- 1757 Views
- 4 replies
- 1 kudos
How to serve a Unity Catalog ML model to external usage
Hello everyone I am following this notebook tutorial https://docs.databricks.com/en/machine-learning/manage-model-lifecycle/index.html#example-notebook Now I can register a machine learning model in Unity Catalog, but the tutorial only shows how to u...
- 1757 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @johndoe99012 If the answer resolved your question, please consider marking it as the solution. It helps others in the community find answers more easily.
- 1 kudos
- 4077 Views
- 3 replies
- 0 kudos
Maximum wait time Databricks Model Serving
hi, hope you are fineI deployed a model 3 or 2 months ago using Databricks Serving and MLFlow. The model worked good using GPU from model serving.I stopped using it for some months and when I tried again deploying it, it has some errors.1. [FIXED] A ...
- 4077 Views
- 3 replies
- 0 kudos
- 6790 Views
- 3 replies
- 1 kudos
Resolved! Tuning `CrossValidator` spark job performance
I am running a 3-fold cross validation of an ML pipeline that utilizes `GBTClassifier` as the final step. It takes 18 hours to run and I am looking for feedback into how to improve the performance as I expect this to go faster.For context here is the...
- 6790 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @Assaad Mrad​ , So this looks like trying to decide between putting the pipeline in the cross validator or the cross validator in the pipeline. Since you are doing the polynomial expansion as part of the pipeline you might want to consider putt...
- 1 kudos
- 3790 Views
- 4 replies
- 0 kudos
Training Job Failure (Driver Error)
We have a new model training job that was running fine for a few days and then started failing. I have attached images for more details.I am wondering if 'can't reach driver cluster' is a red herring. It says the driver is healthy right before execut...
- 3790 Views
- 4 replies
- 0 kudos
- 0 kudos
In our case, we needed to correct our dependent libraries. We had an incorrect path referenced.
- 0 kudos
- 3641 Views
- 0 replies
- 2 kudos
Rolling predictions with FeatureEngineeringClient
I am performing a time series analysis, using a XGBoostRegressor with rolling predictions. I am doing so using the FeatureEngineeringClient (in combination with Unity Catalog), where I create and load in my features during training and inference, as ...
- 3641 Views
- 0 replies
- 2 kudos
- 1494 Views
- 2 replies
- 0 kudos
Resolved! Vector search index stops at 45406
I am trying to create a vector search index for a table, but it stops at 45406 rows. I can see that the writeback table has all the records but the indexing stops. Is there a hard limit on index?
- 1494 Views
- 2 replies
- 0 kudos
- 0 kudos
There are some limits that you can be hitting: Row Size for Delta Sync Index: The maximum row size is 100KB.Embedding Source Column Size for Delta Sync Index: The maximum size is 32764 bytes.Bulk Upsert Request Size Limit for Direct Vector Index: The...
- 0 kudos
- 4417 Views
- 2 replies
- 1 kudos
AutoML Runs Failing
After the Data Exploration notebook runs successfully, all AutoML trials fail without providing a source notebook. I have ensured that the training data labels have no null values or any labels with 16 or less occurrences associated with them. I cann...
- 4417 Views
- 2 replies
- 1 kudos
- 1301 Views
- 2 replies
- 0 kudos
Python running far slower than locally, even with large cluster and multiple workers
I have a notebook that is running extremely slowly even when I try to do pretty basic python functions. It is running far slower than locally no matter what I try, this is in spite of using a 32gb 4 core cluster with 4-8 workers. For context, my data...
- 1301 Views
- 2 replies
- 0 kudos
- 0 kudos
Please share more information, for example: Type of data sourceType of operations being executed (sharing code if possible)Timings of local runs and Databricks runs
- 0 kudos
- 3584 Views
- 0 replies
- 0 kudos
AutoML "need to sample" not working as expected
tl; dr:When the AutoML run realizes it needs to do sampling because the driver / worker node memory is not enough to load / process the entire dataset, it fails. A sample weight column is NOT provided by me, but I believe somewhere in the process the...
- 3584 Views
- 0 replies
- 0 kudos
- 3610 Views
- 3 replies
- 5 kudos
Deploy a ML model, trained and registered in Databricks to AKS
Hi,I can train, registered a ML Model in my Datbricks Workspace.Then, to deploy it on AKS, I need to register the model in Azure ML, and then, deploy to AKS.Is it possible to skip the Azure ML step?I would like to deploy directly into my AKS instance...
- 3610 Views
- 3 replies
- 5 kudos
- 5 kudos
Is it still the case, can't we serve the model in Databricks. I am new to this, so I am just wondering the capabilities.
- 5 kudos
- 1317 Views
- 2 replies
- 1 kudos
Endpoint creation without scale-to-zero
Hi, I've got a question about deploying an endpoint for Llama 3.1 8b. The following code should create the endpoint without scale-to-zero. The endpoint is being created, but with scale-to-zero, although scale_to_zero_enabled is set to False. Instead ...
- 1317 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks for the reply @Walter_C. This didn't quite work, since it used a CPU and didn't consider the max_provisioned_throughput, but I finally got it to work like this: from mlflow.deployments import get_deploy_client client = get_deploy_client("data...
- 1 kudos
- 8700 Views
- 5 replies
- 2 kudos
Issue with Multi-column In predicates are not supported in the DELETE condition.
I'm trying to delete rows from a table with the same date or id as records in another table. I'm using the below query and get the error 'Multi-column In predicates are not supported in the DELETE condition'. delete from cost_model.cm_dispatch_consol...
- 8700 Views
- 5 replies
- 2 kudos
- 2 kudos
I seem to get this error on some DeltaTables and not others:df.createOrReplaceTempView("channels_to_delete") spark.sql(""" delete from lake.something.earnings where TenantId = :tenantId and ChannelId = in ( select ChannelId ...
- 2 kudos
- 2938 Views
- 3 replies
- 1 kudos
Resolved! Extracting Topics From Text Data Using PySpark
Hi EveryoneI tried to follow the same steps in Topic from Text on similar data as example. However, when I tri to fit the model with data I get this error.IllegalArgumentException: requirement failed: Column features must be of type equal to one of t...
- 2938 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @amirA ,The LDA model expects the features column to be of type Vector from the pyspark.ml.linalg module, specifically either a SparseVector or DenseVector, whereas you have provided Row type.You need to convert your Row object to SparseVector.Che...
- 1 kudos
- 7501 Views
- 15 replies
- 2 kudos
Serving Endpoint Container Image Creation Fails
Hello, I trained a model using MLFlow, and saved the model as an artifact. I can load the model from a notebook and it works as expected (i.e. I can load the model using its URI).However, when I want to deploy it using Databricks endpoints, container...
- 7501 Views
- 15 replies
- 2 kudos
- 2 kudos
@ivan_calvo The problem still exists. Surely there has to be some other option than downgrading the ML cluster to DBR 14.3 LTS ML?
- 2 kudos
