cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Science & Machine Learning

Forum Posts

Northp
by New Contributor II
  • 4133 Views
  • 2 replies
  • 2 kudos

Resolved! Differences between Feature Store and Unity Catalog

Our small team has just finished the data preparation phase of our project and started data analysis in Databricks. As we go deeper into this field, we're trying to understand the distinctions and appropriate uses for a Feature Store versus a Unity C...

  • 4133 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vinay_M_R
Databricks Employee
  • 2 kudos

Hi @Northp  Good day!1.)  A Feature Store is a centralized repository that enables data scientists to find and share features, ensuring that the same code used to compute the feature values is used for model training and inference. It is particularly...

  • 2 kudos
1 More Replies
db_noob
by New Contributor II
  • 3504 Views
  • 4 replies
  • 8 kudos

Azure - Databricks - account storage gen 2

Hello Every one, i am really new to databricks, just passed my apache developer certification on it.i also have a certification on data engineering with Azure.some fancy words here but i only started doing real deep work on them as i started a person...

  • 3504 Views
  • 4 replies
  • 8 kudos
Latest Reply
Debayan
Databricks Employee
  • 8 kudos

Hi,If we go by the error , Invalid configuration value detected for fs.azure.account.keyStorage account access key to access data using the abfssprotocol cannot be used. Please refer this https://learn.microsoft.com/en-us/azure/databricks/storage/azu...

  • 8 kudos
3 More Replies
gary7135
by New Contributor II
  • 3865 Views
  • 6 replies
  • 1 kudos

CUDA out of memory

I am trying out the new Meta LLama2 model.Following the databricks provided notebook example: https://github.com/databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-13b/01_load_inference.py I keep getting CUDA out of memory. My G...

  • 3865 Views
  • 6 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Kumaran  Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!

  • 1 kudos
5 More Replies
vayunandan_tupa
by New Contributor
  • 2514 Views
  • 3 replies
  • 0 kudos
  • 2514 Views
  • 3 replies
  • 0 kudos
Latest Reply
dplante
Contributor II
  • 0 kudos

You can take all the Databricks exams as many times as you want, but you have to pay a fee each time you take the exam.

  • 0 kudos
2 More Replies
m_koch_unify
by New Contributor II
  • 8752 Views
  • 2 replies
  • 1 kudos

Running test inference on Llama-2-70B-chat-GPTQ… are C++ libraries installed correctly?

Hi all,I was following the hugging face model https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ, which points to use Exllama (https://github.com/turboderp/exllama/), which has 4 bit quantization.Running on a A10-Single-GPU-64GB,I've cloned the Ex...

  • 8752 Views
  • 2 replies
  • 1 kudos
Latest Reply
m_koch_unify
New Contributor II
  • 1 kudos

Hi @Kumaran,Thanks so much for the quick reply. When I run the script with !bash install_cusparse.shIt runs for a bit, but ultimately encounters an error. When I run !ls -l, i dont even see a data-mle directory in dbfshere is the full output from run...

  • 1 kudos
1 More Replies
akc
by New Contributor III
  • 1194 Views
  • 2 replies
  • 1 kudos

E-mail notification on failure run with DBX deployment

I am deploying workflow to Databricks using DBX. Here I want to add that when the workflow runs and if it fails I will get an e-mail on my_email@email.com. I have included an example workflow. deployments: - name: my_workflow ... # Other wo...

  • 1194 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kumaran
Databricks Employee
  • 1 kudos

Hi @akc,Thank you for posting your question in the Databricks community.Please refer to this documentation for the email notification.

  • 1 kudos
1 More Replies
phdykd
by New Contributor
  • 5115 Views
  • 1 replies
  • 0 kudos

Cannot re-initialize CUDA in forked subprocess.

This is the error I am getting :"RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method". I am using 13.0nc12s_v3 Cluster.I used this one :"import torch.multiprocessing as...

  • 5115 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kumaran
Databricks Employee
  • 0 kudos

Hi @phdykd,Thank you for posting your question in the Databricks community.One approach is to include the start_method="fork" parameter in the spawn function call as follows: mp.spawn(*prev_args, start_method="fork"). Although this will work, it migh...

  • 0 kudos
416412
by New Contributor
  • 1099 Views
  • 0 replies
  • 0 kudos

Unable to Infer Spark ML Pipeline model when built using Custom Preprocessing Stages

We are trying to build an internal use case based on PySpark. The data we have requires a lot of pre-processing. Hence, to cater to that we have used custom Spark ML pipeline stages as some of the transformations that need to be done on our data aren...

code_snipped.png Screenshot 2023-07-14 165821.png
Machine Learning
estimator
machine-learning
mlflow
spark-ml
transformer
  • 1099 Views
  • 0 replies
  • 0 kudos
YanhDong_68817
by New Contributor II
  • 2319 Views
  • 4 replies
  • 4 kudos

Resolved! Pyspark streaming optimization we need to focus on

What optimization we should focus on?

  • 2319 Views
  • 4 replies
  • 4 kudos
Latest Reply
Tharun-Kumar
Databricks Employee
  • 4 kudos

@YanhDong_68817 This document is one of the good places to start evaluating our streaming pipeline - https://docs.databricks.com/structured-streaming/production.html

  • 4 kudos
3 More Replies
raghagra
by New Contributor III
  • 1882 Views
  • 2 replies
  • 2 kudos

Resolved! sparkxgbregressor and RandomForestRegressor not able to deploy for inferencing

I have been trying to deploy spark ML Models from the experiement page via UI, the deployment gets aborted after a long run, any particular reason for why this might be happening? I have also taken care of dependencies still it is failing.Dependency ...

  • 1882 Views
  • 2 replies
  • 2 kudos
Latest Reply
raghagra
New Contributor III
  • 2 kudos

@Kumaran Thanks for the reply kumaram The deployment was finally successful for Random Forest algorithm, failing for sparkxgbregressor.Sharing code snippet:from xgboost.spark import SparkXGBRegressor vec_assembler = VectorAssembler(inputCols=train_df...

  • 2 kudos
1 More Replies
F5
by New Contributor
  • 1007 Views
  • 2 replies
  • 0 kudos

Databricks machine learning associate certificate test was suspended

Hello, I was doing this test on July 5th at 1:45pm (Los Angelas) time, and the test screen complained about my eyes not looking at screen for 5 seconds (when my room got really hot and I picked up a fan on the side), then the test was suspended. Can ...

  • 1007 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kumaran
Databricks Employee
  • 0 kudos

Hi @F5 ,Thank you for posting question in Databricks community.I can see that support request #00342886 has been already handled, Please let us know if you need more help on this.

  • 0 kudos
1 More Replies
KevinKnights
by New Contributor II
  • 2326 Views
  • 5 replies
  • 0 kudos

Fine Tuning Dolly

Hi Community, could you guys share your experience fine tuning Dolly? Thanks, Kevin K.

  • 2326 Views
  • 5 replies
  • 0 kudos
Latest Reply
Kumaran
Databricks Employee
  • 0 kudos

Hi @KevinKnights,Thank you for your question in the Databricks community.I understand that you are looking for documentation on how to fine-tune the Dolly model. Please refer to this documentation on how to fine-tune the model, and let us know if it ...

  • 0 kudos
4 More Replies
Nico3
by New Contributor
  • 947 Views
  • 2 replies
  • 1 kudos

Keeping track of ML models and data assets with Unity Catalogue

Hi all,Our company has trouble keeping track of existing ML models developed by several different teams.On top of this it is hard to keep track of all the data assets collected by all our different divisions.Does Unity Catalogue keep track of ML mode...

  • 947 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Nico3  Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!

  • 1 kudos
1 More Replies
Dawid
by New Contributor II
  • 1311 Views
  • 2 replies
  • 2 kudos

Resolved! How to use ml flow with pytorch?

Excited to learn how can I efficiently and quickly log metrics in ml flow from my pytorch model? #summit23

  • 1311 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Dawid  Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!

  • 2 kudos
1 More Replies
bb_s
by New Contributor
  • 958 Views
  • 2 replies
  • 0 kudos
  • 958 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @bb_s  Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels