cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

staskh
by New Contributor III
  • 344 Views
  • 3 replies
  • 3 kudos

Resolved! Error in automl.regress

Hi,I'm running example notebook from https://docs.databricks.com/aws/en/machine-learning/automl/regression-train-api on a node with ML cluster 17.0 (includes Apache Spark 4.0.0, Scala 2.13) and getting error at from databricks import automlsummary = ...

staskh_0-1756215619847.png
  • 344 Views
  • 3 replies
  • 3 kudos
Latest Reply
staskh
New Contributor III
  • 3 kudos

Ilir, greetings!Thank you for a prompt response. Unfortunately, none of the suggested solutions works. I checked with Genie:"The error occurs because databricks-automl is not available for Databricks Runtime 17.0.x. Databricks AutoML is not supported...

  • 3 kudos
2 More Replies
spearitchmeta
by New Contributor III
  • 232 Views
  • 2 replies
  • 1 kudos

Data Drift & Model Comparison in Production MLOps: Handling Scale Changes with AutoML

BackgroundI'm implementing a production MLOps pipeline for part classification using Databricks AutoML. My pipeline automatically retrains models when new data arrives and compares performance with existing production models.The ChallengeI've encount...

  • 232 Views
  • 2 replies
  • 1 kudos
Latest Reply
BigRoux
Databricks Employee
  • 1 kudos

Have you explored Lakehouse Monitoring?  It provides a comprehensive solution for drift detection.  You can read more here: https://docs.databricks.com/aws/en/lakehouse-monitoring/   Hope this helps, Louis.

  • 1 kudos
1 More Replies
Aravinda
by New Contributor III
  • 397 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Machine Learning Practitioner Plan - DBC section unavailability

Hi Everyone,I am not able to locate any DBC folders for each course present in the machine learning practitioner plan.Earlier, we used to have DBC sections where we can access the course and lab materials.Do we have any solution to this??? Or can som...

  • 397 Views
  • 5 replies
  • 2 kudos
Latest Reply
Aravinda
New Contributor III
  • 2 kudos

Thanks @szymon_dybczak !!

  • 2 kudos
4 More Replies
drii_cavalcanti
by New Contributor III
  • 355 Views
  • 2 replies
  • 1 kudos

Resolved! Installing opencv-python on DBX

Hi everyone,I was wondering how I can install such a basic Python package on Databricks without running into conflict issues or downgrading to a runtime version lower than 15.Specs:The worker type is g4dn.xlarge [T4].The runtime is 16.4 LTS (includes...

  • 355 Views
  • 2 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @drii_cavalcanti ,You encountered this issue because opencv-python depends on packages that still require numpy in version lower than 2. You need to reinstall numpy to supported version and then try once again installing library. You can do it usi...

  • 1 kudos
1 More Replies
dbuser24
by New Contributor III
  • 964 Views
  • 14 replies
  • 12 kudos

Resolved! ML experiment giving error - RESOURCE_DOES_NOT_EXIST

Followed the below documentation to create a ML experiment - https://docs.databricks.com/aws/en/mlflow/experimentsI created an experiment using the databricks console, then tried running the below code but getting error - getting error - RESOURCE_DOE...

  • 964 Views
  • 14 replies
  • 12 kudos
Latest Reply
BS_THE_ANALYST
Esteemed Contributor
  • 12 kudos

can you mark your own post as a solution as well @dbuser24? (would be useful for the additional steps)Appreciate you feeding back your findings.Congrats on getting it working.All the best,BS

  • 12 kudos
13 More Replies
elisabethfalck
by New Contributor
  • 229 Views
  • 1 replies
  • 0 kudos

Forecasting serverless can write predicitons, compute cluster cannot ???

Hi! I have something I don't understand.... I used automl forecasting (serverless) to train a model and marked my schema edw_forecasting as output database where it saved the predictions of my best model. Awesome.However, when I try to do automl fore...

elisabethfalck_0-1754142475527.png elisabethfalck_1-1754142546110.png
  • 229 Views
  • 1 replies
  • 0 kudos
Latest Reply
Khaja_Zaffer
Contributor
  • 0 kudos

Did you contact your account team? @elisabethfalck  Also as per the error: can you make 5 max worker nodes?  

  • 0 kudos
Sri2025
by New Contributor
  • 815 Views
  • 1 replies
  • 0 kudos

Not able to run end to end ML project on Databricks Trial

I started using Databricks trial version from today. I want to explore full end to end ML lifecycle on the databricks. I observed for the compute only 'serverless' option is available. I was trying to execute the notebook posted on https://docs.datab...

  • 815 Views
  • 1 replies
  • 0 kudos
Latest Reply
BigRoux
Databricks Employee
  • 0 kudos

I can take up to 15 minutes for the serving endpoint to be created. Once you initiate the "create endpoint" chunk of code go and grab a cup of coffee and wait 15 minutes.  Then, before you use it verify it is running (bottom left menu "Serving") by g...

  • 0 kudos
cmd0160
by New Contributor
  • 1000 Views
  • 1 replies
  • 0 kudos

Interactive EDA task in a Job Workflow

I am trying to configure an interactive EDA task as part of a job workflow. I'd like to be able to trigger a workflow, perform some basic analysis then proceed to a subsequent task. I haven't had any success freezing execution. Also, the job workflow...

  • 1000 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @cmd0160, Freezing job execution to perform interactive tasks directly within a job workflow is not natively supported in Databricks. The job workflow UI and the notebook UI serve different purposes, and the interactive capabilities you find in...

  • 0 kudos
ncparab13
by New Contributor II
  • 1379 Views
  • 2 replies
  • 1 kudos

deploy, train and monitor AI/ML model in databricks in automated way.

Hi Team, I have my databricks environment where I want to deploy, train and monitor ML model in automated way(github action). How I can do that?

  • 1379 Views
  • 2 replies
  • 1 kudos
Latest Reply
ashraf1395
Honored Contributor
  • 1 kudos

Hi there @ncparab13,- https://docs.databricks.com/aws/en/dev-tools/bundles/mlops-stacks ,- https://docs.databricks.com/aws/en/machine-learning/mlops/ci-cd-for-ml , - https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-reposHere are some li...

  • 1 kudos
1 More Replies
dkxxx-rc
by Contributor
  • 2900 Views
  • 0 replies
  • 0 kudos

AutoML master notebook failing

I have recently been able to run AutoML successfully on a certain dataset.  But it has just failed on a second dataset of similar construction, before being able to produce any machine learning training runs or output.  The Experiments page says```Mo...

dkxxxrc_0-1740403690249.png
  • 2900 Views
  • 0 replies
  • 0 kudos
SreeRam
by New Contributor
  • 2687 Views
  • 0 replies
  • 0 kudos

Patient Risk Score based on health history: Unable to create data folder for artifacts in S3 bucket

Hi All,we're using the below git project to build PoC on the concept of "Patient-Level Risk Scoring Based on Condition History": https://github.com/databricks-industry-solutions/hls-patient-riskI was able to import the solution into Databricks and ru...

  • 2687 Views
  • 0 replies
  • 0 kudos
MightyMasdo
by New Contributor III
  • 2763 Views
  • 2 replies
  • 6 kudos

Spark context not implemented Error when using Databricks connect

I am developing an application using databricks connect and when I try to use VectorAssembler I get the Error sc is not none Assertion Error. is there a workaround for this ?

  • 2763 Views
  • 2 replies
  • 6 kudos
Latest Reply
Łukasz1
New Contributor III
  • 6 kudos

I have exactly the same problem.The error is in the line 84 of the file pyspark/ml/wrapper.py.assert sc is not NoneI create spark session with databricks connect as the following:from databricks.connect import DatabricksSessionspark = DatabricksSessi...

  • 6 kudos
1 More Replies
dkxxx-rc
by Contributor
  • 1646 Views
  • 2 replies
  • 1 kudos

Resolved! Save model from AutoML to MLflow in LightGBM flavor

I want to get the LightGBM built-in variable importance values from a model that was generated by AutoML.  That's not logged in the metrics by default - can I change a setting so that it will be logged?More fundamentally:  what I'd really like is to ...

  • 1646 Views
  • 2 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Additional Considerations The pyfunc.add_to_model() function you mentioned is used to add the Python Function flavor to the model, which is different from changing the primary flavor of the logged model. That's why changing its parameter didn't solve...

  • 1 kudos
1 More Replies
Akash_Wadhankar
by New Contributor III
  • 795 Views
  • 0 replies
  • 0 kudos

Learn Databricks AI medium article series for fellow learners.

When it comes to machine learning, the platform plays a pivotal role in successful implementation. Databricks offers a best-in-class machine learning platform with cutting-edge features such as MLflow, Model Registry, Feature Store, and MLOps, which ...

Machine Learning
DatabricksML MachineLearning AI FeatureStore DecisionScience
  • 795 Views
  • 0 replies
  • 0 kudos
sjohnston2
by New Contributor II
  • 2430 Views
  • 2 replies
  • 2 kudos

Resolved! XGBoost Feature Weighting

We are trying to train a predictive ML model using the XGBoost Classifier. Part of the requirements we have gotten from our business team is to implement feature weighting as they have defined certain features mattering more than others. We have 69 f...

  • 2430 Views
  • 2 replies
  • 2 kudos
Latest Reply
Walter_C
Databricks Employee
  • 2 kudos

Hello @sjohnston2 here is some information i found internally: Possible Causes Memory Access Issue: The segmentation fault suggests that the program is trying to access memory that it's not allowed to, which could be caused by an internal bug in XGBo...

  • 2 kudos
1 More Replies
Labels