- 214 Views
- 6 replies
- 11 kudos
ML course affected by phasing out of community edition
Hi there - I'm a professor at CMU teaching a course on scalable ML, and the last few semesters we have run assignments using the DataBricks Community Edition, which is no longer open for new accounts. The free accounts aren't a great substitute for ...
- 214 Views
- 6 replies
- 11 kudos
- 11 kudos
@wcohen have you looked into Databricks support for universities: https://www.databricks.com/university There's also this section which could be good to point out to your students:All the best,BS
- 11 kudos
- 194 Views
- 4 replies
- 11 kudos
Resolved! [Databricks User Research] AI Assistance for Data Science Work
Hi all! I'm running a user research series on the topic of AI Assistance for data science work (e.g. exploratory data analysis, feature engineering, model eval, etc.). If you are a Databricks Notebooks user, I would love to pick your brain on your pr...
- 194 Views
- 4 replies
- 11 kudos
- 11 kudos
@DataBrickatorPerfect timing! I'm a heavy Databricks Notebooks user working on performance optimization and ML pipelines.My daily workflows involve:EDA on 100M+ record Iceberg tablesReal-time feature engineering with Spark StreamingCustom indexing st...
- 11 kudos
- 191 Views
- 2 replies
- 4 kudos
Resolved! Does Databricks AutoML support multi-target/multi-output classification?
Hi Databricks community,I'm working on a classification problem where I need to predict 50 different target columns simultaneously using the same input features (X), commonly known as multi-target or multi-output classification.My question: Does Data...
- 191 Views
- 2 replies
- 4 kudos
- 4 kudos
Long story short after lots of research: No.. Sadly this is not possible. Yet the two mentioned options could be considered as workaround
- 4 kudos
- 300 Views
- 3 replies
- 3 kudos
Resolved! Error in automl.regress
Hi,I'm running example notebook from https://docs.databricks.com/aws/en/machine-learning/automl/regression-train-api on a node with ML cluster 17.0 (includes Apache Spark 4.0.0, Scala 2.13) and getting error at from databricks import automlsummary = ...
- 300 Views
- 3 replies
- 3 kudos
- 3 kudos
Ilir, greetings!Thank you for a prompt response. Unfortunately, none of the suggested solutions works. I checked with Genie:"The error occurs because databricks-automl is not available for Databricks Runtime 17.0.x. Databricks AutoML is not supported...
- 3 kudos
- 206 Views
- 2 replies
- 1 kudos
Data Drift & Model Comparison in Production MLOps: Handling Scale Changes with AutoML
BackgroundI'm implementing a production MLOps pipeline for part classification using Databricks AutoML. My pipeline automatically retrains models when new data arrives and compares performance with existing production models.The ChallengeI've encount...
- 206 Views
- 2 replies
- 1 kudos
- 1 kudos
Have you explored Lakehouse Monitoring? It provides a comprehensive solution for drift detection. You can read more here: https://docs.databricks.com/aws/en/lakehouse-monitoring/ Hope this helps, Louis.
- 1 kudos
- 81 Views
- 0 replies
- 0 kudos
Unable to register Scikit-learn or XGBoost model to unity catalog
Hello, I'm following the tutorial provided here https://docs.databricks.com/aws/en/notebooks/source/mlflow/mlflow-classic-ml-e2e-mlflow-3.html for ML model management process using ML FLow, in a unity-catalog enabled workspace, however I'm facing an ...
- 81 Views
- 0 replies
- 0 kudos
- 11691 Views
- 3 replies
- 3 kudos
Resolved! How to PREVENT mlflow's autologging from logging ALL runs?
I am logging runs from jupyter notebook. the cells which has `mlflow.sklearn.autlog()` behaves as expected. but, the cells which has .fit() method being called on sklearn's estimators are also being logged as runs without explicitly mentioning `mlflo...
- 11691 Views
- 3 replies
- 3 kudos
- 3 kudos
It looks like MLflow auto-logging is kicking in by default whenever you call .fit(), which is why you’re seeing runs even without explicitly using mlflow.sklearn.autolog(). To fix this, you can disable the global autologging and only trigger it when ...
- 3 kudos
- 305 Views
- 3 replies
- 0 kudos
Having a different performance while use GPU and CPU
I'm building a model that is mostly sklearn libraries, but I'm also using TF-IDF and RandomForest. In theory, they only need a CPU to work properly, but in fact, when I use a physical computer with about 32 GB of RAM, it runs very fast. There are som...
- 305 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @dangkhai : Have the below link and see parallelism enabled during transformation. ParallelTextProcessing/parallelizing_text_processing.ipynb at master · rafaelvalero/ParallelTextProcessing
- 0 kudos
- 2321 Views
- 1 replies
- 0 kudos
When does everyone utilize the model register?
Hi, I'm Yuki,I'm considering when I should use register_model.In my case, I'm running the training batch once a week and if the model is good, I want to update the champion.I have created the code to register the model if the score is the best.# star...
- 2321 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Yuki, Thank you for contacting the Databricks community. If you run register_model with the same run twice, you’ll create multiple versions pointing to the same source. To avoid that, you can check if the run is already registered before creati...
- 0 kudos
- 2336 Views
- 1 replies
- 0 kudos
Custom Multi-agent deployment error
Hi. I am deploying a custom multi-agent system comprising of a genie agent and a RAG solution. While deploying, I am getting the following error:I am using 16.1 ML (Node: Standard_D4ads_v5 16GB,4 core) cluster and I am using the following code for lo...
- 2336 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Nishat, Thank you for contacting the Databricks community. Which LLM model are you currently utilising?
- 0 kudos
- 2279 Views
- 1 replies
- 0 kudos
Infer_signature for a dictionary datasets during mlflow registration
Hello community,Can you please guide me here. I am trying to build custom Ensemble model where I will be passing a dictionary of datasets to the fit() and predict() with the keys being the model_names and value being the respective datasets for each ...
- 2279 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @skosaraju, Thank you for contacting Databricks support! Because infer_signature() cannot handle a dict of DataFrames directly, you will need to convert this structure into a dictionary of row dicts, or manually build a ModelSignature. Option A: F...
- 0 kudos
- 1144 Views
- 1 replies
- 0 kudos
How to Reduce Log Latency for AI Gateway-Enabled Inference Tables in Model Serving?
Hi everyone,I've recently deployed a custom model using Databricks Model Serving with AI Gateway-enabled inference tables. The model is built with:Python 3.11.11LightGBM 4.5.0MLflow 2.13.1I’ve noticed that the inference logs can take up to 1 hour to ...
- 1144 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ecram, Thank you for contacting Databricks community. As per the doc below, you'll see the latency for 1 hour for log delivery in the inference table. https://docs.databricks.com/aws/en/ai-gateway/inference-tables#:~:text=You%20can%20expect%20log...
- 0 kudos
- 370 Views
- 5 replies
- 2 kudos
Resolved! Databricks Machine Learning Practitioner Plan - DBC section unavailability
Hi Everyone,I am not able to locate any DBC folders for each course present in the machine learning practitioner plan.Earlier, we used to have DBC sections where we can access the course and lab materials.Do we have any solution to this??? Or can som...
- 370 Views
- 5 replies
- 2 kudos
- 426 Views
- 2 replies
- 3 kudos
Resolved! Unexpected ProtoBuf Version Changes on DBR 15.4 LTS Causing databricks-feature-client Failures
Hi all,I noticed an issue starting from Friday, Aug 15 through Monday, Aug 18. Our clusters running DBR 15.4 LTS experienced unexpected upgrades of the protobuf library:Friday: protobuf was at version 4.24.1 (as documented here: DBR 15.4 LTS release ...
- 426 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @guilhermeneves! Could you confirm if pip was used to install any Python packages in your workload? The version shifts you’re seeing are likely related to unpinned pip installs: when package versions aren’t pinned, dependency changes from third...
- 3 kudos
- 204 Views
- 1 replies
- 0 kudos
mlflow.store.artifact.cloud_artifact_repo. issue in Databricks APPs backend
INFO mlflow.store.artifact.cloud_artifact_repo: Failed to complete request, possibly due to credential expiration. Refreshing credentials and trying again... (Error: API request to https://dbc-d4dd6237-edea.cloud.databricks.com/api/2.0/fs/create-down...
- 204 Views
- 1 replies
- 0 kudos
- 0 kudos
Why MLflow and Databricks library versions?Did you try increasing the MLflow artifact upload/download timeout, for example:import osos.environ["MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT"] = "120"
- 0 kudos
