- 580 Views
- 4 replies
- 2 kudos
Data Drift & Model Comparison in Production MLOps: Handling Scale Changes with AutoML
BackgroundI'm implementing a production MLOps pipeline for part classification using Databricks AutoML. My pipeline automatically retrains models when new data arrives and compares performance with existing production models.The ChallengeI've encount...
- 580 Views
- 4 replies
- 2 kudos
- 2 kudos
Here are my thoughts to the questions you pose. However, it is important that you dig into the documentation to fully understand the capabilites of Lakehouse Monitoring. I will also be helpful if you deploy it to understand the mechanics of how it wo...
- 2 kudos
- 699 Views
- 3 replies
- 3 kudos
Resolved! Error in automl.regress
Hi,I'm running example notebook from https://docs.databricks.com/aws/en/machine-learning/automl/regression-train-api on a node with ML cluster 17.0 (includes Apache Spark 4.0.0, Scala 2.13) and getting error at from databricks import automlsummary = ...
- 699 Views
- 3 replies
- 3 kudos
- 3 kudos
Ilir, greetings!Thank you for a prompt response. Unfortunately, none of the suggested solutions works. I checked with Genie:"The error occurs because databricks-automl is not available for Databricks Runtime 17.0.x. Databricks AutoML is not supported...
- 3 kudos
- 648 Views
- 5 replies
- 2 kudos
Resolved! Databricks Machine Learning Practitioner Plan - DBC section unavailability
Hi Everyone,I am not able to locate any DBC folders for each course present in the machine learning practitioner plan.Earlier, we used to have DBC sections where we can access the course and lab materials.Do we have any solution to this??? Or can som...
- 648 Views
- 5 replies
- 2 kudos
- 661 Views
- 2 replies
- 1 kudos
Resolved! Installing opencv-python on DBX
Hi everyone,I was wondering how I can install such a basic Python package on Databricks without running into conflict issues or downgrading to a runtime version lower than 15.Specs:The worker type is g4dn.xlarge [T4].The runtime is 16.4 LTS (includes...
- 661 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @drii_cavalcanti ,You encountered this issue because opencv-python depends on packages that still require numpy in version lower than 2. You need to reinstall numpy to supported version and then try once again installing library. You can do it usi...
- 1 kudos
- 1769 Views
- 14 replies
- 12 kudos
Resolved! ML experiment giving error - RESOURCE_DOES_NOT_EXIST
Followed the below documentation to create a ML experiment - https://docs.databricks.com/aws/en/mlflow/experimentsI created an experiment using the databricks console, then tried running the below code but getting error - getting error - RESOURCE_DOE...
- 1769 Views
- 14 replies
- 12 kudos
- 12 kudos
can you mark your own post as a solution as well @dbuser24? (would be useful for the additional steps)Appreciate you feeding back your findings.Congrats on getting it working.All the best,BS
- 12 kudos
- 315 Views
- 1 replies
- 0 kudos
Forecasting serverless can write predicitons, compute cluster cannot ???
Hi! I have something I don't understand.... I used automl forecasting (serverless) to train a model and marked my schema edw_forecasting as output database where it saved the predictions of my best model. Awesome.However, when I try to do automl fore...
- 315 Views
- 1 replies
- 0 kudos
- 0 kudos
Did you contact your account team? @elisabethfalck Also as per the error: can you make 5 max worker nodes?
- 0 kudos
- 849 Views
- 1 replies
- 0 kudos
Not able to run end to end ML project on Databricks Trial
I started using Databricks trial version from today. I want to explore full end to end ML lifecycle on the databricks. I observed for the compute only 'serverless' option is available. I was trying to execute the notebook posted on https://docs.datab...
- 849 Views
- 1 replies
- 0 kudos
- 0 kudos
I can take up to 15 minutes for the serving endpoint to be created. Once you initiate the "create endpoint" chunk of code go and grab a cup of coffee and wait 15 minutes. Then, before you use it verify it is running (bottom left menu "Serving") by g...
- 0 kudos
- 1033 Views
- 1 replies
- 0 kudos
Interactive EDA task in a Job Workflow
I am trying to configure an interactive EDA task as part of a job workflow. I'd like to be able to trigger a workflow, perform some basic analysis then proceed to a subsequent task. I haven't had any success freezing execution. Also, the job workflow...
- 1033 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @cmd0160, Freezing job execution to perform interactive tasks directly within a job workflow is not natively supported in Databricks. The job workflow UI and the notebook UI serve different purposes, and the interactive capabilities you find in...
- 0 kudos
- 1434 Views
- 2 replies
- 1 kudos
deploy, train and monitor AI/ML model in databricks in automated way.
Hi Team, I have my databricks environment where I want to deploy, train and monitor ML model in automated way(github action). How I can do that?
- 1434 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi there @ncparab13,- https://docs.databricks.com/aws/en/dev-tools/bundles/mlops-stacks ,- https://docs.databricks.com/aws/en/machine-learning/mlops/ci-cd-for-ml , - https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-reposHere are some li...
- 1 kudos
- 3086 Views
- 0 replies
- 0 kudos
AutoML master notebook failing
I have recently been able to run AutoML successfully on a certain dataset. But it has just failed on a second dataset of similar construction, before being able to produce any machine learning training runs or output. The Experiments page says```Mo...
- 3086 Views
- 0 replies
- 0 kudos
- 2867 Views
- 0 replies
- 0 kudos
Patient Risk Score based on health history: Unable to create data folder for artifacts in S3 bucket
Hi All,we're using the below git project to build PoC on the concept of "Patient-Level Risk Scoring Based on Condition History": https://github.com/databricks-industry-solutions/hls-patient-riskI was able to import the solution into Databricks and ru...
- 2867 Views
- 0 replies
- 0 kudos
- 2866 Views
- 2 replies
- 6 kudos
Spark context not implemented Error when using Databricks connect
I am developing an application using databricks connect and when I try to use VectorAssembler I get the Error sc is not none Assertion Error. is there a workaround for this ?
- 2866 Views
- 2 replies
- 6 kudos
- 6 kudos
I have exactly the same problem.The error is in the line 84 of the file pyspark/ml/wrapper.py.assert sc is not NoneI create spark session with databricks connect as the following:from databricks.connect import DatabricksSessionspark = DatabricksSessi...
- 6 kudos
- 1752 Views
- 2 replies
- 1 kudos
Resolved! Save model from AutoML to MLflow in LightGBM flavor
I want to get the LightGBM built-in variable importance values from a model that was generated by AutoML. That's not logged in the metrics by default - can I change a setting so that it will be logged?More fundamentally: what I'd really like is to ...
- 1752 Views
- 2 replies
- 1 kudos
- 1 kudos
Additional Considerations The pyfunc.add_to_model() function you mentioned is used to add the Python Function flavor to the model, which is different from changing the primary flavor of the logged model. That's why changing its parameter didn't solve...
- 1 kudos
- 827 Views
- 0 replies
- 0 kudos
Learn Databricks AI medium article series for fellow learners.
When it comes to machine learning, the platform plays a pivotal role in successful implementation. Databricks offers a best-in-class machine learning platform with cutting-edge features such as MLflow, Model Registry, Feature Store, and MLOps, which ...
- 827 Views
- 0 replies
- 0 kudos
- 2531 Views
- 2 replies
- 2 kudos
Resolved! XGBoost Feature Weighting
We are trying to train a predictive ML model using the XGBoost Classifier. Part of the requirements we have gotten from our business team is to implement feature weighting as they have defined certain features mattering more than others. We have 69 f...
- 2531 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @sjohnston2 here is some information i found internally: Possible Causes Memory Access Issue: The segmentation fault suggests that the program is trying to access memory that it's not allowed to, which could be caused by an internal bug in XGBo...
- 2 kudos