- 1169 Views
- 4 replies
- 2 kudos
Unable to register Scikit-learn or XGBoost model to unity catalog
Hello, I'm following the tutorial provided here https://docs.databricks.com/aws/en/notebooks/source/mlflow/mlflow-classic-ml-e2e-mlflow-3.html for ML model management process using ML FLow, in a unity-catalog enabled workspace, however I'm facing an ...
- 1169 Views
- 4 replies
- 2 kudos
- 2 kudos
You need to ensure that your Unity Catalog catalog and schema already exist, that you have the necessary permissions to use them, and that you update the code to reference your own catalog and schema names. You must also run on a classic cluster with...
- 2 kudos
- 4215 Views
- 3 replies
- 0 kudos
DAB - Add/remove task depending on workspace.
I use DAB for deploying Jobs, I want to add a specific Task in dev only but not in staging or prod. Is there any way to achieve this using DAB ?
- 4215 Views
- 3 replies
- 0 kudos
- 0 kudos
I know it's a bit old, but if someone is looking into a solution, then I was able to resolve the issue where I need to deploy some jobs only into the DEV target:https://github.com/databricks/bundle-examples/tree/main/knowledge_base/target_includes.Us...
- 0 kudos
- 864 Views
- 2 replies
- 3 kudos
Resolved! Population stability index (PSI) calculation in Lakehouse monitor
Hi! We are using Lakehouse monitoring for detecting data drift in our metrics. However, the exact calculation of metrics is not documented anywhere (I couldnt find it) and it raises questions on how they are done, in our case especially - PSI. I woul...
- 864 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Danik , I have reviewed this. 1) Is there documentation for PSI and other metrics?Public docs list PSI in the drift table and give thresholds, but don’t detail the exact algorithm.Internally, numeric PSI uses ~1000 quantiles, equal‑height binning...
- 3 kudos
- 14538 Views
- 5 replies
- 3 kudos
Resolved! How to PREVENT mlflow's autologging from logging ALL runs?
I am logging runs from jupyter notebook. the cells which has `mlflow.sklearn.autlog()` behaves as expected. but, the cells which has .fit() method being called on sklearn's estimators are also being logged as runs without explicitly mentioning `mlflo...
- 14538 Views
- 5 replies
- 3 kudos
- 431 Views
- 4 replies
- 1 kudos
MLFlow Detailed Trace view doesn't work in some workspaces
I've created a Databricks Model Serving Endpoint which serves an MLFlow Pyfunc model. The model uses langchain and I'm using mlflow.langchain.autolog().At my company we have some production(-like) workspaces where users cannot e.g. run Notebooks and ...
- 431 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Jahnavi,Thanks for your reply. I think the issues you mentioned are not the cause of the discrepancy though. I have attached a screenshot of the same trace ID when displayed in the Experiments UI (where I cannot get a detailed trace view) and in t...
- 1 kudos
- 955 Views
- 2 replies
- 3 kudos
Resolved! Is Delta Lake deeply tested in Professional Data Engineer Exam?
I wanted to ask people who have already taken the Databricks Certified Professional Data Engineer exam whether Delta Lake is tested in depth or not. While preparing, I’m currently using the Databricks Certified Professional Data Engineer sample quest...
- 955 Views
- 2 replies
- 3 kudos
- 3 kudos
Yes, Delta Lake concepts are an important part of the Databricks Professional Data Engineer exam, but they aren’t tested in extreme depth compared to core Spark transformations and data pipeline design. The exam mainly focuses on practical understand...
- 3 kudos
- 1895 Views
- 3 replies
- 2 kudos
Model Serving - Shadow Deployment - Azure
Hey,I'm composing an architecture within the usage of Model Serving Endpoints and one of the needs that we're aiming to resolve is Shadow Deployment.Currently, it seems that the traffic configurations available in model serving do not allow this type...
- 1895 Views
- 3 replies
- 2 kudos
- 2 kudos
@ryojikn and @irtizak , you’re right. Databricks Model Serving allows splitting traffic between model versions, but it doesn’t have a true shadow deployment where live production traffic is mirrored to a new model for monitoring without affecting use...
- 2 kudos
- 346 Views
- 2 replies
- 3 kudos
Getting error when running databricks deploy bundle command
HI all,I am trying to implement MLOps project using https://github.com/databricks/mlops-stacks repo.I have created azure databricks with Premium (+ Role-based access controls) (Click to change) and following bundle creation and deploy using uRL: http...
- 346 Views
- 2 replies
- 3 kudos
- 3 kudos
This is expected behavior with mlops-stacks and not an issue with your Terraform version or the CLI. The main problem is that your Azure Databricks workspace does not have Unity Catalog enabled or assigned. The mlops-stacks templates assume Unity Cat...
- 3 kudos
- 434 Views
- 2 replies
- 2 kudos
Why does my MLflow model training job fail on Databricks with an out‑of‑memory error for large datas
I am trying to train a machine learning model using MLflow on Databricks. When my dataset is very large, the training stops and gives an ‘out-of-memory’ error. Why does this happen and how can I fix it?
- 434 Views
- 2 replies
- 2 kudos
- 2 kudos
+1 to what @mukul1409 has told. Please follow the guides below to distribute the training: https://docs.databricks.com/aws/en/machine-learning/train-model/distributed-training/spark-pytorch-d... https://docs.databricks.com/aws/en/notebooks/source/dee...
- 2 kudos
- 973 Views
- 1 replies
- 1 kudos
Resolved! Full list of serving endpoint metrics returned by api/2.0/serving-endpoints/[ENDPOINT_NAME]/metrics
Hello! Looking at the documentation for this metric endpoint: https://docs.databricks.com/aws/en/machine-learning/model-serving/metrics-export-serving-endpointIt does not include a sample API response, and the code examples given don't have the full ...
- 973 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @KyraHinnegan , I did some digging and here is what I found: Based on the Databricks documentation, GPU metrics exposed by the Serving Endpoint Metrics API follow a clear and consistent naming convention. Once you know the pattern, the response i...
- 1 kudos
- 742 Views
- 1 replies
- 0 kudos
Getting error when running databricks deploy bundle command
HI all,I am trying to implement MLOps project using https://github.com/databricks/mlops-stacks repo.I have created azure databricks with Premium (+ Role-based access controls) (Click to change) and following bundle creation and deploy using uRL: http...
- 742 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, first things to check is that you have the correct permissions on the user or service principal you're running the job with, the user needs to have workspace access and cluster creation access toggled on. Next you need to check you have a metast...
- 0 kudos
- 1502 Views
- 3 replies
- 1 kudos
Resolved! What are the practical differences between bagging and boosting algorithms?
How are bagging and boosting different when you use them in real machine-learning projects?
- 1502 Views
- 3 replies
- 1 kudos
- 1 kudos
The practical differences between bagging and boosting mostly come down to how they build models and how they handle errors:Model Training Approach:Bagging (Bootstrap Aggregating): Builds multiple models in parallel using random subsets of the data. ...
- 1 kudos
- 1509 Views
- 4 replies
- 2 kudos
Resolved! How do I improve the performance of my Random Forest model on Databricks?
How can I make these people smarter or faster so the final answer is better?
- 1509 Views
- 4 replies
- 2 kudos
- 2 kudos
Improving the performance of a Random Forest model on Databricks is usually about data quality, feature engineering, and hyperparameter tuning. Some tips:Feature Engineering:Create meaningful features and remove irrelevant ones.Encode categorical var...
- 2 kudos
- 336 Views
- 1 replies
- 1 kudos
How do I implement and train a custom PyTorch model on Databricks using distributed training?
How can I build my own PyTorch machine-learning model and train it faster on Databricks by using multiple machines/GPUs instead of just one?
- 336 Views
- 1 replies
- 1 kudos
- 1 kudos
@Suheb , You may look at the torch distributor. It provides multiple distributed training options, including single-node with multiple-GPU training and multi-node training. Below are the references for you. https://docs.databricks.com/aws/en/machine-...
- 1 kudos
- 486 Views
- 2 replies
- 0 kudos
Vector search index very slow
Hello,I have created a vector search index for a delta table with 1,400 rows. Using this vector index to find matching records on a table with 52M records with the query below ran for 20hrs and failed with: 'HTTP request failed with status: {"error_c...
- 486 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @RodrigoE ,Your LATERAL subquery calls the Vector Search function once for every row of the 52M-row table, which results in tens of millions of remote calls to the Vector Search endpoint—this is not a nice pattern and will be extremely slow leadin...
- 0 kudos
