cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

oteng
by New Contributor III
  • 1084 Views
  • 3 replies
  • 3 kudos

DLT with unity catalog and ML

We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog. When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User se...

  • 1084 Views
  • 3 replies
  • 3 kudos
Latest Reply
pg5
New Contributor II
  • 3 kudos

I recently hit the same issue.Seems like this is a limitation of DLT with Unity Catalog.Did you find a workaround @oteng? Otherwise I will try copying the materialized views to a table before doing the ML work.

  • 3 kudos
2 More Replies
Saurabh707344
by New Contributor III
  • 1409 Views
  • 3 replies
  • 2 kudos

Resolved! ML usecase feasibility for Databricks ML Vs AWS Sagemaker/Azure ML

What complexity of ML models are feasible to be created in Databricks ML and further that we have to rely on AWS Sagamaker or Azure ML ?Do we have clear segragation around it by ML usecases ?

  • 1409 Views
  • 3 replies
  • 2 kudos
Latest Reply
corvo
New Contributor II
  • 2 kudos

In Databricks, your usecase can be solved by the notebooks provided here in databricks. There is no dependency on AWS sagemaker directly. All the model traiing and deployement that can be done in sagemaker, is supported via databricks as well. 

  • 2 kudos
2 More Replies
Cristianmarja
by New Contributor
  • 487 Views
  • 1 replies
  • 0 kudos

2.0 Train and Validate ML Model - Exercise / Double Type is not defined

Hi everyone,Please note that I stuck with exercise 2.0 Train and Validate ML Model because when I run code appear a NameError with the following label: name 'DoubleType' is not defined.I would like any help about this subject.

  • 487 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Cristian Martinez​ :In Databricks, you need to import the necessary classes from the pyspark.sql.types module in order to use them in your code. To fix the NameError you're encountering with the label "name 'DoubleType' is not defined" in Exercise 2...

  • 0 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 575 Views
  • 1 replies
  • 5 kudos

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outs...

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outside of Databricks. While serving the model via REST API is ideal for external use cases, it is recom...

1677238735536
  • 575 Views
  • 1 replies
  • 5 kudos
Latest Reply
jose_gonzalez
Moderator
  • 5 kudos

Thank you for sharing this!!!

  • 5 kudos
anvil
by New Contributor II
  • 1559 Views
  • 3 replies
  • 4 kudos

Are UDFs necessary for applying models from ML libraries at scale ?

Hello,I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method. Spark MLlib models don't need this k...

  • 1559 Views
  • 3 replies
  • 4 kudos
Latest Reply
Manoj12421
Valued Contributor II
  • 4 kudos

MlLib is in the maintenance model and udf is not used by creating model in most cases

  • 4 kudos
2 More Replies
Slalom_Tobias
by New Contributor III
  • 1760 Views
  • 3 replies
  • 3 kudos

Resolved! ML Practioner | ML 11 - XGBoost notebook | cannot import keras.applications.resnet50

the following code...from sparkdl.xgboost import XgboostRegressorfrom pyspark.ml import Pipelineparams = {"n_estimators": 100, "learning_rate": 0.1, "max_depth": 4, "random_state": 42, "missing": 0}xgboost = XgboostRegressor(**params)pipeline = Pipel...

  • 1760 Views
  • 3 replies
  • 3 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 3 kudos

You need to choose the runtime for ML instead of the standard.

  • 3 kudos
2 More Replies
TomasP
by New Contributor III
  • 898 Views
  • 3 replies
  • 0 kudos

Two or more different ml model on one cluster.

Hi, have you already dealt with the situation that you would like to have two different ml models in one cluster? i.e: I have a project which contains two or more different models with more different pursposes. The goals is to have three differ...

  • 898 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Tomas Peterek​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 0 kudos
2 More Replies
isaac_gritz
by Valued Contributor II
  • 1698 Views
  • 1 replies
  • 1 kudos

Responsible AI on Databricks

Looking to learn how you can use responsible AI toolkits on Databricks? Interested in learning how you can incorporate open source tools like SHAP and Fairlearn with Databricks?I would recommend checking out this blog: Mitigating Bias in Machine Lear...

  • 1698 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Awesome!

  • 1 kudos
Benji
by New Contributor II
  • 2595 Views
  • 5 replies
  • 0 kudos

Error when running job in databricks

Hello, I am very new with databricks and MLflow. I faced with the problem about running job. When the job is run, it usually failed and retried itself, so it incasesed running time, i.e., from normally 6 hrs to 12-18 hrs. From the error log, it shows...

Fail image
  • 2595 Views
  • 5 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @Tanawat Benchasirirot​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hea...

  • 0 kudos
4 More Replies
confusedIntern
by New Contributor III
  • 1827 Views
  • 4 replies
  • 2 kudos

Uploaded Docker image into cluster. Used cluster for MLFlow experiment, but no experiment is logged/there are no experiment runs. Why is this?

Hi! So I used this MLFlow experiment I found from the databricks website: https://docs.databricks.com/_static/notebooks/machine-learning-with-unity-catalog.htmlAnd I created this cluster using a custom Docker image I created myself:  Usually when I c...

Screen Shot 2022-08-02 at 4.13.14 PM Screen Shot 2022-08-02 at 4.17.10 PM Screen Shot 2022-08-02 at 4.17.47 PM
  • 1827 Views
  • 4 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Have you tried the steps mentioned in the below URL:https://docs.databricks.com/clusters/custom-containers.html#step-3-launch-your-cluster

  • 2 kudos
3 More Replies
Slalom_Tobias
by New Contributor III
  • 1438 Views
  • 4 replies
  • 1 kudos

Resolved! ML Practioner | ml 09 - automl notebook | error on importing databricks.automl

executing the following code...from databricks import automlsummary = automl.regress(train_df, target_col="price", primary_metric="rmse", timeout_minutes=5, max_trials=10)generates the error...ImportError: cannot import name 'automl' from 'databricks...

  • 1438 Views
  • 4 replies
  • 1 kudos
Latest Reply
Krueger156
New Contributor II
  • 1 kudos

I'm happy to see a particularly subject.

  • 1 kudos
3 More Replies
Slalom_Tobias
by New Contributor III
  • 1639 Views
  • 2 replies
  • 3 kudos

Resolved! ML Practioner | ML 10 - Feature Store notebook | feature_store import error

the following code...from pyspark.sql.functions import monotonically_increasing_id, lit, expr, randimport uuidfrom databricks import feature_storefrom pyspark.sql.types import StringType, DoubleTypefrom databricks.feature_store import feature_table, ...

  • 1639 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hope that was an easy fix - @Tobias Cortese​ ! Thanks for marking the "best answer"!

  • 3 kudos
1 More Replies
Joseph_B
by New Contributor III
  • 640 Views
  • 0 replies
  • 1 kudos

mlflow.org

2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (Post 2 of 2)Thank you to everyone who joined! You can access the on-demand recording here and the code in this Github repo.We're sharing a subset of the questions asked an...

  • 640 Views
  • 0 replies
  • 1 kudos
Joseph_B
by New Contributor III
  • 472 Views
  • 0 replies
  • 1 kudos

docs.databricks.com

2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (post 1 of 2)Thank you to everyone who joined the Automating the ML Lifecycle With Databricks Machine Learning webinar! You can access the on-demand recording here and the ...

  • 472 Views
  • 0 replies
  • 1 kudos
MoJaMa
by Valued Contributor II
  • 607 Views
  • 1 replies
  • 0 kudos
  • 607 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

Data is stored in the control plane. Metadata (eg feature table descriptions, column types, etc) is stored in the control plane. The location where the Delta table is stored is determined by the database location. The customer could call  CREATE DATA...

  • 0 kudos
Labels