- 197 Views
- 4 replies
- 1 kudos
MLFlow Detailed Trace view doesn't work in some workspaces
I've created a Databricks Model Serving Endpoint which serves an MLFlow Pyfunc model. The model uses langchain and I'm using mlflow.langchain.autolog().At my company we have some production(-like) workspaces where users cannot e.g. run Notebooks and ...
- 197 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Jahnavi,Thanks for your reply. I think the issues you mentioned are not the cause of the discrepancy though. I have attached a screenshot of the same trace ID when displayed in the Experiments UI (where I cannot get a detailed trace view) and in t...
- 1 kudos
- 169 Views
- 2 replies
- 3 kudos
Is Delta Lake deeply tested in Professional Data Engineer Exam?
I wanted to ask people who have already taken the Databricks Certified Professional Data Engineer exam whether Delta Lake is tested in depth or not. While preparing, I’m currently using the Databricks Certified Professional Data Engineer sample quest...
- 169 Views
- 2 replies
- 3 kudos
- 3 kudos
Yes, Delta Lake concepts are an important part of the Databricks Professional Data Engineer exam, but they aren’t tested in extreme depth compared to core Spark transformations and data pipeline design. The exam mainly focuses on practical understand...
- 3 kudos
- 206 Views
- 1 replies
- 1 kudos
Resolved! Full list of serving endpoint metrics returned by api/2.0/serving-endpoints/[ENDPOINT_NAME]/metrics
Hello! Looking at the documentation for this metric endpoint: https://docs.databricks.com/aws/en/machine-learning/model-serving/metrics-export-serving-endpointIt does not include a sample API response, and the code examples given don't have the full ...
- 206 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @KyraHinnegan , I did some digging and here is what I found: Based on the Databricks documentation, GPU metrics exposed by the Serving Endpoint Metrics API follow a clear and consistent naming convention. Once you know the pattern, the response i...
- 1 kudos
- 284 Views
- 1 replies
- 1 kudos
Resolved! Options sporadic (and cost-efficient) Model Serving on Databricks?
Hi all,I'm new to Databricks so would appreciate some advice.I have a ML model deployed using Databricks Model Serving. My use case is very sporadic: I only need to make 5–15 prediction requests per day (industrial application), and there can be long...
- 284 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @cbossi , You are right! A 30-minute idle period precedes the endpoint's scaling down. You are billed for the compute resources used during this period, plus the actual serving time when requests are made. This is the current expected behaviour. Y...
- 1 kudos
- 403 Views
- 1 replies
- 1 kudos
Resolved! How does Databricks AutoML handle null imputation for categorical features by default?
Hi everyone I’m using Databricks AutoML (classification workflow) on Databricks Runtime 10.4 LTS ML+, and I’d like to clarify how missing (null) values are handled for categorical (string) columns by default.From the AutoML documentation, I see that:...
- 403 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @spearitchmeta , I looked internally to see if I could help with this and I found some information that will shed light on your question. Here’s how missing (null) values in categorical (string) columns are handled in Databricks AutoML on Dat...
- 1 kudos
- 736 Views
- 1 replies
- 1 kudos
Best Practices for Collaborative Notebook Development in Databricks
Hi everyone! I’m looking to learn more about effective strategies for collaborative development in Databricks notebooks. Since notebooks are often used by multiple data scientists, analysts, and engineers, managing collaboration efficiently is critic...
- 736 Views
- 1 replies
- 1 kudos
- 1 kudos
For version control, use this approach.Git Integration with Databricks ReposCore Features:Databricks Git Folders (Repos) provides native Git integration with visual UI and REST API access Supports all major providers: GitHub, GitLab, Azure DevOps, Bi...
- 1 kudos
- 1728 Views
- 3 replies
- 1 kudos
Distributed SparkXGBRanker training: failed barrier ResultStage
I'm following a variation of the tutorial [here](https://assets.docs.databricks.com/_extras/notebooks/source/xgboost-pyspark-new.html) to train an `SparkXGBRanker` in distributed mode. However, the line:pipeline_model = pipeline.fit(data) Is throwing...
- 1728 Views
- 3 replies
- 1 kudos
- 1 kudos
You have already mentioned you did turn off autoscaling, please try the num_workers too Step 1: Disable Dynamic Resource Allocation: Use spark.dynamicAllocation.enabled = false Step 2: Configure num_workers to Match Fixed Resources After disabling dy...
- 1 kudos
- 992 Views
- 1 replies
- 0 kudos
Lakehouse monitoring generates broken queries
Hi everyone,I’m setting up Databricks Lakehouse Monitoring to track my model’s performance using an inference-regression monitor. I’ve completed all the required configuration and successfully launched my first monitoring run.The quality tables are g...
- 992 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @the_p_l ,I want to confirm that I understand your situation correctly. You mentioned that you are not adding any custom code to the deployed Lakehouse Monitoring setup, and you believe the issue is related to the inline comments generated during ...
- 0 kudos
- 990 Views
- 3 replies
- 2 kudos
Unable to register Scikit-learn or XGBoost model to unity catalog
Hello, I'm following the tutorial provided here https://docs.databricks.com/aws/en/notebooks/source/mlflow/mlflow-classic-ml-e2e-mlflow-3.html for ML model management process using ML FLow, in a unity-catalog enabled workspace, however I'm facing an ...
- 990 Views
- 3 replies
- 2 kudos
- 2 kudos
Maybe add missing: mlflow.set_tracking_uri("databricks")mlflow.set_registry_uri("databricks")
- 2 kudos
- 1255 Views
- 3 replies
- 1 kudos
Endpoint deployment is very slow
HI team I am testing some changes on UAT / DEV environment and noticed that the model endpoint are very slow to deploy. Since the environment is just testing and not serving any production traffic, I was wondering if there was a way to expedite this ...
- 1255 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @WiliamRosa Thanks for your response on this. I have been using the setting you described aboved, with the exception of `scale_to_zero`. PFA screenshot of the endpoint settings. My deployment is a simple Pytorch Deep Learning model wrapped in a `s...
- 1 kudos
- 1862 Views
- 4 replies
- 4 kudos
Resolved! Distributed Optuna and MLflow
Hello All, I just tried running the following notebook (https://docs.databricks.com/aws/en/notebooks/source/machine-learning/optuna-mlflow.html) on the Databricks Free Edition platform , through Microsoft Account Authentication. It takes 15 minutes ...
- 1862 Views
- 4 replies
- 4 kudos
- 4 kudos
Great. Thank you. That worked. I still need more compute and networking resources to make it justifiable, but this confirms that it works !!!
- 4 kudos
- 976 Views
- 1 replies
- 1 kudos
[ERROR] Worker (pid:11) was sent code 132 When deploying a Custom Model in serving
Hi, I've been developing a custom model with mlflow.pyfunc.PythonModel. Among other libs, I use ANNOY. While trying to serve the model as an endpoint in "serving", After a few fixes my model worked fine as well the endpoin call.Altough, I tried updat...
- 976 Views
- 1 replies
- 1 kudos
- 1 kudos
Great observation! The difference between Using worker: sync and Using worker: gevent typically refers to the worker class used by Gunicorn, the web server behind many MLflow model deployments (like in Databricks model serving or other MLflow-compati...
- 1 kudos
- 1844 Views
- 2 replies
- 3 kudos
Resolved! Serving Endpoint: Container image creation
Hi TeamWhenever I try to create an endpoint from a model in Databricks, the process often gets stuck at the 'Container Image Creation' step. I've tried to understand what happens during this step, but couldn't find any detailed or helpful information...
- 1844 Views
- 2 replies
- 3 kudos
- 3533 Views
- 5 replies
- 3 kudos
Resolved! This API is disabled for users without the databricks-sql-access
Running a deply on github: Run databricks bundle deploydatabricks bundle deployshell: /usr/bin/bash -e {0}env:DATABRICKS_HOST: {{HOST}}DATABRICKS_CLIENT_ID: {{ID}}DATABRICKS_CLIENT_SECRET: ***DATABRICKS_BUNDLE_ENV: prodError: This API is disabled for...
- 3533 Views
- 5 replies
- 3 kudos
- 3 kudos
Got it working, yes I see it was a little confusing at first, the workspace displayed at the top right is the account information whereas the profile icon is where you can access the workspace settings. For anyone that got as confused as I did. Thank...
- 3 kudos
- 1092 Views
- 1 replies
- 1 kudos
Resolved! Model Inferencing
Any links, pointers to host a model in real time (similar to sagemaker endpoint in aws) - how can we host a model in DBX in real time - any documentation please?
- 1092 Views
- 1 replies
- 1 kudos
- 1 kudos
@Sachin_Amin you can find an example in our docs here: https://docs.databricks.com/aws/en/machine-learning/model-serving/model-serving-intro We also have free training courses on realtime model deployment for both classical ML (https://www.databricks...
- 1 kudos