I am deploying workflow to Databricks using DBX. Here I want to add that when the workflow runs and if it fails I will get an e-mail on my_email@email.com. I have included an example workflow. deployments:
- name: my_workflow
...
# Other wo...
This is the error I am getting :"RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method". I am using 13.0nc12s_v3 Cluster.I used this one :"import torch.multiprocessing as...
Hi @phdykd,Thank you for posting your question in the Databricks community.One approach is to include the start_method="fork" parameter in the spawn function call as follows: mp.spawn(*prev_args, start_method="fork"). Although this will work, it migh...
We are trying to build an internal use case based on PySpark. The data we have requires a lot of pre-processing. Hence, to cater to that we have used custom Spark ML pipeline stages as some of the transformations that need to be done on our data aren...
@YanhDong_68817 This document is one of the good places to start evaluating our streaming pipeline - https://docs.databricks.com/structured-streaming/production.html
I have been trying to deploy spark ML Models from the experiement page via UI, the deployment gets aborted after a long run, any particular reason for why this might be happening? I have also taken care of dependencies still it is failing.Dependency ...
@Kumaran Thanks for the reply kumaram The deployment was finally successful for Random Forest algorithm, failing for sparkxgbregressor.Sharing code snippet:from xgboost.spark import SparkXGBRegressor
vec_assembler = VectorAssembler(inputCols=train_df...
Hello, I was doing this test on July 5th at 1:45pm (Los Angelas) time, and the test screen complained about my eyes not looking at screen for 5 seconds (when my room got really hot and I picked up a fan on the side), then the test was suspended. Can ...
Hi @F5 ,Thank you for posting question in Databricks community.I can see that support request #00342886 has been already handled, Please let us know if you need more help on this.
Hi @KevinKnights,Thank you for your question in the Databricks community.I understand that you are looking for documentation on how to fine-tune the Dolly model. Please refer to this documentation on how to fine-tune the model, and let us know if it ...
Hi all,Our company has trouble keeping track of existing ML models developed by several different teams.On top of this it is hard to keep track of all the data assets collected by all our different divisions.Does Unity Catalogue keep track of ML mode...
Hi @Nico3
Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.
Cheers!
Hi @Dawid
Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.
Cheers!
Hi @bb_s
Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.
Cheers!
I'm trying to get a visualisation made using R's networkD3 package to show up in my notebook, but it simply refuses to work. I simply get an empty cell result. Apparently this is a common problem for HTML/JS packages (cfr. https://github.com/Microsof...
Hi @gkrishnan82
We haven't heard from you since the last response from @Kumaran , and I was checking back to see if her suggestions helped you.
Or else, If you have any solution, please share it with the community, as it can be helpful to others.
...
We have been facing issues serving even simple models in production:https://community.databricks.com/t5/machine-learning/errors-using-dolly-deployed-as-a-rest-api/m-p/37405#M1939
hi all,I have been working on the concept on of centralized workspace where I have been trying to create a centralized workspace repo.I am running my model on Workspace A, while I am logging my model on to a remote Databricks workspace B, I have conn...
I am running a databricks job on a cluster and I keep running into the following issue (pasted below in bold) The job trains a machine learning model on a modestly sized dataset (~ half GB). Note that I use pandas dataframes for the data, sklearn for...
