cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

SamGreene
by Contributor
  • 423 Views
  • 2 replies
  • 1 kudos

Issues with experiment errors over the last two weeks

Hi,I use Azure Databricks in the North Central US region and have had some issues over the last two weeks.  Three weeks ago, I was able to run a forecast experiment.  Last week I got this error on 7/24:[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, va...

  • 423 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @SamGreene,  This error indicates that the SQL engine cannot find the column ds. This could be due to changes in your data schema or the way the query is constructed. Ensure that the column ds exists in your dataset and is correctly referenced in ...

  • 1 kudos
1 More Replies
TSchmidt
by New Contributor
  • 349 Views
  • 1 replies
  • 0 kudos

large scale yolo inference

I have 50 Million Images sitting on s3 I have a Yolov8 model trained with ultralytics and want to run inference on those images. I suspect I should be running inference using ML flow, but I am confused on how. I don't need to track experiments/traini...

  • 349 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @TSchmidt, To efficiently run inference on your 50 million images stored in S3 using a trained YOLOv8 model from Ultralytics, start by downloading your model from S3 and loading it locally. Use the `boto3` library to list images in your S3 bucket ...

  • 0 kudos
MightyMasdo
by New Contributor
  • 433 Views
  • 1 replies
  • 0 kudos

Spark context not implemented Error when using Databricks connect

I am developing an application using databricks connect and when I try to use VectorAssembler I get the Error sc is not none Assertion Error. is there a workaround for this ?

  • 433 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

@MightyMasdo could you please share the screenshot of the error along with the command?

  • 0 kudos
117074
by New Contributor III
  • 455 Views
  • 0 replies
  • 0 kudos

AutoML models not completing

Hello, Whilst using a cluster set-up running 14.3 LTS ML, 2-10 workers, worker and driver type of r5d.xlarge I am having issues creating a regression model on 700k rows and 80 factors (no high cardinality in any factor shown).The first phase of the e...

  • 455 Views
  • 0 replies
  • 0 kudos
bothma2
by New Contributor II
  • 714 Views
  • 3 replies
  • 0 kudos

How to I select an 80/10/10 split when doing AutoML

Headline says it all. I am doing a regression and want to select a testvaltrain split that is not 60/20/20. Anyone know how to do this?

  • 714 Views
  • 3 replies
  • 0 kudos
Latest Reply
mhiltner
Contributor III
  • 0 kudos

You'd need to put 80% of your data with the earliest timestamp, then 10% with another one and 10% with another. 

  • 0 kudos
2 More Replies
amal15
by New Contributor II
  • 1231 Views
  • 2 replies
  • 0 kudos

error: not found: type XGBoostEstimator

error: not found: type XGBoostEstimator Spark & Scala  

  • 1231 Views
  • 2 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@amal15 - can you please include the below to the import statement and see if it works. ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator 

  • 0 kudos
1 More Replies
tanjil
by New Contributor III
  • 1266 Views
  • 3 replies
  • 0 kudos

Import mlflow Error

Hello, I am trying to replicate this motebook in my environment: mlflow-end-to-end-example - Databricks However, I am getting the following error when I run "import mlflow": "TypeError: bases must be types"How can I solve this issue? Thank you, Tanji...

  • 1266 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kumaran
Valued Contributor
  • 0 kudos

Hello @tanjil    Thank you for contacting databricks community support. Could you check what version of protobuf you have? If you are using 10.4 ML cluster, the MLflow 1.x is not compatible with protobuf 4.x. The default version of protobuf in MLR 10...

  • 0 kudos
2 More Replies
Amoozegar
by New Contributor II
  • 1053 Views
  • 1 replies
  • 0 kudos

Error in Tensorflow training job

I upgraded Tensorflow on Databricks notebook using %pip command. Now when running the training job, I get this error: "DNN library initialization failed."

Machine Learning
GPU enabled clusters
Tensorflow
  • 1053 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Amoozegar,  Check TensorFlow Version: Ensure that the TensorFlow version you upgraded to is compatible with your existing code and dependencies. Sometimes, upgrading TensorFlow can lead to compatibility issues. You might want to verify if the sp...

  • 0 kudos
User100024
by New Contributor II
  • 1196 Views
  • 2 replies
  • 1 kudos

Using AutoML to predict completion dates of a project management dataset

Hello! I am fairly new to Databricks. I'm trying to do a proof of concept with AutoML in Databricks at my organization, and the dataset I am using is a project management dataset. Here's a sample: project_idmarketgeneral_contractorproject_typepermit_...

  • 1196 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @User100024, Let’s break down your requirements and tackle them step by step. Reducing Completion Date Period: To understand how different factors impact the completion date, you can use regression analysis. Specifically, you want to predict t...

  • 1 kudos
1 More Replies
DanMaycock
by New Contributor III
  • 6959 Views
  • 14 replies
  • 6 kudos

Resolved! Can't Run an AutoML Experiment Because Button is Greyed Out

I am trying to run an AutoML experiment but the button stays greyed out no matter what I do. I've tried different cluster configurations, different datasets, even blew away the instance in Azure and re-created it across two different Azure accounts s...

Machine Learning
AutoML
Databricks
machine learning
  • 6959 Views
  • 14 replies
  • 6 kudos
Latest Reply
oleclercq
New Contributor II
  • 6 kudos

Thanks. AutoML is back on

  • 6 kudos
13 More Replies
miahopman
by New Contributor II
  • 2163 Views
  • 2 replies
  • 0 kudos

AutoML Runs Failing

After the Data Exploration notebook runs successfully, all AutoML trials fail without providing a source notebook. I have ensured that the training data labels have no null values or any labels with 16 or less occurrences associated with them. I cann...

  • 2163 Views
  • 2 replies
  • 0 kudos
Latest Reply
Annapurna_Hiriy
New Contributor III
  • 0 kudos

@miahopman We understand that you are looking for a better way of troubleshooting in AutoML. We have an internal feature request raised to address precisely the issues you have discussed here.

  • 0 kudos
1 More Replies
miahopman
by New Contributor II
  • 2598 Views
  • 2 replies
  • 1 kudos

AutoML Trials Failing

Sometimes an AutoML experiment will have all trials fail and I cannot figure out what is causing it. Each individual run reports a validation f1 value but the source notebook is not available so I cannot track down the error. This seems to happen at ...

  • 2598 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @miahopman, Did you check the dataset for errors, missing values or other anomalies affecting the AutoML performance?

  • 1 kudos
1 More Replies
Om1992
by New Contributor
  • 681 Views
  • 0 replies
  • 0 kudos

Automl

How to efficiently use automl

  • 681 Views
  • 0 replies
  • 0 kudos
aranyics
by New Contributor
  • 764 Views
  • 1 replies
  • 1 kudos

Is it possible to start Databricks AutoML experiment remotely? (Azure Databricks)

Currently I am using Azure Machine Learning Studio for my work, and would like to compare performance of Azure and Databricks automl algorithms. Is it possible to write a notebook in Azure to start the automl algorithm in Databricks? My data is found...

  • 764 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Csaba Aranyi​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
jaredaw
by New Contributor II
  • 7928 Views
  • 2 replies
  • 2 kudos

Resolved! AutoML with Stratified Sampling

Is it possible to use a stratified sampling strategy for the train/test/validate splits that the automl library does? We are working in a context where we need to segregate certain groups from the training and test sets to see how our models general...

  • 7928 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

HI @Jared Webb​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 2 kudos
1 More Replies
Labels