cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Science & Machine Learning

Forum Posts

girl_chan
by New Contributor II
  • 691 Views
  • 1 replies
  • 0 kudos

Azure databricks API and DLT databricks

how can i pass parameter from Azure data factory rest web API to delta live Databricks?I get this error: "Py4JJavaError: An error occurred while calling o382.getArgument.: com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named *** def...

  • 691 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @milena chambe​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
VVM
by New Contributor III
  • 982 Views
  • 2 replies
  • 2 kudos

Bug with display function with structs?

It appears to me that there's a deceptive bug when using the databricks display function and viewing struct data. For whatever reason, multiple spaces are cut down to only one:from pyspark.sql.functions import struct, col   df = spark.createDataFrame...

  • 982 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Patrick Mascari​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
1 More Replies
ChrisS
by New Contributor III
  • 7740 Views
  • 14 replies
  • 5 kudos

Resolved! How do you get data from Azure Data Lake Gen 2 Mounted or Imported and Exported from Databricks?

The example that data bricks gives is not helpful and does not tell me exactly what I need to do. I am new to this and not sure what I need to do in azure to get this done. I just need to be able to pull data and write data to the data containers. Be...

  • 7740 Views
  • 14 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Chris Sarrico​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 5 kudos
13 More Replies
Priyag1
by Honored Contributor II
  • 1508 Views
  • 1 replies
  • 5 kudos

docs.databricks.com

New unified Databricks navigationDatabricks plans to enable the new navigation experience (Public Preview) by default for all users. You’ll be able to opt out by clicking Disable new UI in the sidebar.The goal of the new experience is to reduce click...

  • 1508 Views
  • 1 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Priyadarshini G​ Thank you for providing accurate and valuable information.Best Regards

  • 5 kudos
tariq
by New Contributor III
  • 1649 Views
  • 4 replies
  • 3 kudos

dbx execute cluster issue

I am facing issue with cluster not being able to run the updated code using dbx execute command. Any changes I make to the code is not reflected in the execution until I restart the cluster. I am using a photon enable cluster with Standard_D4s_v5 as ...

  • 1649 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Tarique Anwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 3 kudos
3 More Replies
Haekyung
by New Contributor II
  • 991 Views
  • 3 replies
  • 3 kudos

Can someone help my academy account switching from partner to customer?

Hi, Today I accidentally created my account in partner-academy.I created my academy account in partner-academy with my company email, then I realized I can't create account with same email in customer-academy which is correct place I should have crea...

  • 991 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Haekyung Won​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
aranyics
by New Contributor
  • 627 Views
  • 1 replies
  • 1 kudos

Is it possible to start Databricks AutoML experiment remotely? (Azure Databricks)

Currently I am using Azure Machine Learning Studio for my work, and would like to compare performance of Azure and Databricks automl algorithms. Is it possible to write a notebook in Azure to start the automl algorithm in Databricks? My data is found...

  • 627 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Csaba Aranyi​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
StephanieRivera
by Valued Contributor II
  • 657 Views
  • 1 replies
  • 2 kudos

How do I move the template files into my own repo when cloning the MLflow recipes templates into Databricks?

Here https://mlflow.org/docs/latest/recipes.html#model-development-workflow, there are directions to add the repo. Is this best practice in Databricks? I tried exporting the repo code (inside of a Databricks notebook).. My DBC export was successful. ...

  • 657 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Stephanie Rivera​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
Saurabh707344
by New Contributor III
  • 556 Views
  • 1 replies
  • 2 kudos

AWS Databricks - Distributed ML Models in Sagameker and Databricks

While using Databricks on AWS, What will be impact if few ML models are build using Sagemaker pipelines, whereas other models build on databricks ML itself ?Any other impact apart from infra maintainance cost ?Are there any prefered tool that can eas...

  • 556 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Saurabh Singh​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
rgbuckley
by New Contributor III
  • 6499 Views
  • 5 replies
  • 6 kudos

Resolved! Fix Hanging Task in Databricks

I am applying a pandas UDF to a grouped dataframe in databricks. When I do this, a couple tasks hang forever, while the rest complete quickly.I start by repartitioning my dataset so that each group is in one partition:group_factors = ['a','b','c'] #m...

Spark UI for compute cluster stderr for hanging task stdout for hanging task
  • 6499 Views
  • 5 replies
  • 6 kudos
Latest Reply
rgbuckley
New Contributor III
  • 6 kudos

Thank you Suteja. I had watched the resources and had never reached capacity for any. The data was evenly distributed across partitions and groups as well. I did end up taking your advice in (1). I set a timer and killed the process if the group took...

  • 6 kudos
4 More Replies
smedegaard
by New Contributor III
  • 3088 Views
  • 3 replies
  • 5 kudos

Resolved! Difference between MLFlow recipes and projects?

MLFlow projects are described asAn MLflow Project is a format for packaging data science code in a reusable and reproducible way, based primarily on conventions. In addition, the Projects component includes an API and command-line tools for running p...

  • 3088 Views
  • 3 replies
  • 5 kudos
Latest Reply
smedegaard
New Contributor III
  • 5 kudos

Thanks for the answer @Priyadarshini G​ . Although a project has a pre-defined folder structure and standard files, it also "... includes an API and command-line tools for running projects, making it possible to chain together projects into workflows...

  • 5 kudos
2 More Replies
js54123875
by New Contributor III
  • 2631 Views
  • 4 replies
  • 3 kudos

Resolved! How to enforce schema with Autoloader?

I have a number of csv files that I am working to ingest using autoloader. There is an ID field that I want to require to be a STRING, but using SchemaHints is not working and is instead setting as an INT.The first few csv files have just integer va...

  • 2631 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jennette Shepard​ We haven't heard from you since the last response from @Suteja Kanuri​  . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards

  • 3 kudos
3 More Replies
reachbharathan
by New Contributor III
  • 2170 Views
  • 4 replies
  • 5 kudos

Resolved! Authenticating gitlab with databricks via username & password?

Currently we have azure databricks and gitlab in our project, for integrating with code repository we have only gitlab, integrating with personal access token is possible,But it flagged out as potential risk of personal access token exposure, wanted...

  • 2170 Views
  • 4 replies
  • 5 kudos
Latest Reply
reachbharathan
New Contributor III
  • 5 kudos

Thank you folks,currently only way to integrate with gitlab is only with Personal Access Token,There is not way to intergrate gitlab via password, as per our security recommendation, we need to have additional mechanism to integrate as exposure of Pe...

  • 5 kudos
3 More Replies
Jaeseon
by New Contributor II
  • 2049 Views
  • 3 replies
  • 3 kudos

Resolved! Distributed training on building object detection model on PyTorch and PySpark.

I'm currently immersed in a project where I'm leveraging PyTorch to develop an object detection model using satellite imagery. My immediate objective is to perform distributed training on this model using PySpark. While I have found several tutorials...

  • 2049 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jaeseon Song​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
fsimoes
by New Contributor II
  • 1490 Views
  • 2 replies
  • 1 kudos

Resolved! Docker image with libraries + MLFlow Experiments

Hi everybody,I have a scenario where we have multiple teams working with Python and R, and this teams uses a lot of different libraries. Because of this dozen of libraries, the cluster start took much time. Then I created a Docker image, where I can ...

  • 1490 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Fabio Simoes​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 1 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels