- 3747 Views
- 4 replies
- 1 kudos
Unable to access python variables in-between shells in same notebook
Unable to access Python variables in-between shells in the same notebook even if the entire code is written in Python. Getting error that unable to identify variable in the new cell
- 3747 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi, thanks for your message. Can you sent a screenshot of your notebook maybe?This makes it hard to debug what could be the cause of it?Also which runtime are you using, this could might be the issue. With my experience loading the pandas library sho...
- 1 kudos
- 7794 Views
- 4 replies
- 0 kudos
Not able to log xgboost model to mlflow
I have been trying to log mlflow model but seems to be not working. It logs only the last(which is also the worst run).#-------------------------------------------------------13.0 ML XGBOost------------------------------------------------------------...
- 7794 Views
- 4 replies
- 0 kudos
- 0 kudos
@Kumaran Ran this code, but any specific log that I should be looking for?
- 0 kudos
- 1899 Views
- 1 replies
- 0 kudos
MlflowException: Unsupported Databricks profile key prefix: ''. Key prefixes cannot be empty.
I am trying to fetch data from mlflow model registry in Databricks and to use it in my local notebook. But I don't find any resource in internet to do so. I want to configure my mlflow in such a way i can fetch model registry values from databricks w...
- 1899 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AnnamalaiVR,Thank you for posting the question in Databricks Community.In your Python code, import the MLflow library and create a client object to access your Model Registry. %pythonimport mlflow# Set the Databricks tracking URIdatabricks_host =...
- 0 kudos
- 10612 Views
- 2 replies
- 2 kudos
Resolved! Differences between Feature Store and Unity Catalog
Our small team has just finished the data preparation phase of our project and started data analysis in Databricks. As we go deeper into this field, we're trying to understand the distinctions and appropriate uses for a Feature Store versus a Unity C...
- 10612 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Northp Good day!1.) A Feature Store is a centralized repository that enables data scientists to find and share features, ensuring that the same code used to compute the feature values is used for model training and inference. It is particularly...
- 2 kudos
- 6133 Views
- 4 replies
- 8 kudos
Azure - Databricks - account storage gen 2
Hello Every one, i am really new to databricks, just passed my apache developer certification on it.i also have a certification on data engineering with Azure.some fancy words here but i only started doing real deep work on them as i started a person...
- 6133 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi,If we go by the error , Invalid configuration value detected for fs.azure.account.keyStorage account access key to access data using the abfssprotocol cannot be used. Please refer this https://learn.microsoft.com/en-us/azure/databricks/storage/azu...
- 8 kudos
- 6605 Views
- 6 replies
- 1 kudos
CUDA out of memory
I am trying out the new Meta LLama2 model.Following the databricks provided notebook example: https://github.com/databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-13b/01_load_inference.py I keep getting CUDA out of memory. My G...
- 6605 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Kumaran Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!
- 1 kudos
- 4674 Views
- 3 replies
- 0 kudos
- 4674 Views
- 3 replies
- 0 kudos
- 0 kudos
You can take all the Databricks exams as many times as you want, but you have to pay a fee each time you take the exam.
- 0 kudos
- 10951 Views
- 2 replies
- 1 kudos
Running test inference on Llama-2-70B-chat-GPTQ… are C++ libraries installed correctly?
Hi all,I was following the hugging face model https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ, which points to use Exllama (https://github.com/turboderp/exllama/), which has 4 bit quantization.Running on a A10-Single-GPU-64GB,I've cloned the Ex...
- 10951 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kumaran,Thanks so much for the quick reply. When I run the script with !bash install_cusparse.shIt runs for a bit, but ultimately encounters an error. When I run !ls -l, i dont even see a data-mle directory in dbfshere is the full output from run...
- 1 kudos
- 2352 Views
- 2 replies
- 1 kudos
E-mail notification on failure run with DBX deployment
I am deploying workflow to Databricks using DBX. Here I want to add that when the workflow runs and if it fails I will get an e-mail on my_email@email.com. I have included an example workflow. deployments: - name: my_workflow ... # Other wo...
- 2352 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @akc,Thank you for posting your question in the Databricks community.Please refer to this documentation for the email notification.
- 1 kudos
- 8684 Views
- 1 replies
- 0 kudos
Cannot re-initialize CUDA in forked subprocess.
This is the error I am getting :"RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method". I am using 13.0nc12s_v3 Cluster.I used this one :"import torch.multiprocessing as...
- 8684 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @phdykd,Thank you for posting your question in the Databricks community.One approach is to include the start_method="fork" parameter in the spawn function call as follows: mp.spawn(*prev_args, start_method="fork"). Although this will work, it migh...
- 0 kudos
- 1843 Views
- 0 replies
- 0 kudos
Unable to Infer Spark ML Pipeline model when built using Custom Preprocessing Stages
We are trying to build an internal use case based on PySpark. The data we have requires a lot of pre-processing. Hence, to cater to that we have used custom Spark ML pipeline stages as some of the transformations that need to be done on our data aren...
- 1843 Views
- 0 replies
- 0 kudos
- 3889 Views
- 4 replies
- 4 kudos
Resolved! Pyspark streaming optimization we need to focus on
What optimization we should focus on?
- 3889 Views
- 4 replies
- 4 kudos
- 4 kudos
@YanhDong_68817 This document is one of the good places to start evaluating our streaming pipeline - https://docs.databricks.com/structured-streaming/production.html
- 4 kudos
- 3356 Views
- 2 replies
- 2 kudos
Resolved! sparkxgbregressor and RandomForestRegressor not able to deploy for inferencing
I have been trying to deploy spark ML Models from the experiement page via UI, the deployment gets aborted after a long run, any particular reason for why this might be happening? I have also taken care of dependencies still it is failing.Dependency ...
- 3356 Views
- 2 replies
- 2 kudos
- 2 kudos
@Kumaran Thanks for the reply kumaram The deployment was finally successful for Random Forest algorithm, failing for sparkxgbregressor.Sharing code snippet:from xgboost.spark import SparkXGBRegressor vec_assembler = VectorAssembler(inputCols=train_df...
- 2 kudos
- 1590 Views
- 2 replies
- 0 kudos
Databricks machine learning associate certificate test was suspended
Hello, I was doing this test on July 5th at 1:45pm (Los Angelas) time, and the test screen complained about my eyes not looking at screen for 5 seconds (when my room got really hot and I picked up a fan on the side), then the test was suspended. Can ...
- 1590 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @F5 ,Thank you for posting question in Databricks community.I can see that support request #00342886 has been already handled, Please let us know if you need more help on this.
- 0 kudos
- 3956 Views
- 5 replies
- 0 kudos
Fine Tuning Dolly
Hi Community, could you guys share your experience fine tuning Dolly? Thanks, Kevin K.
- 3956 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @KevinKnights,Thank you for your question in the Databricks community.I understand that you are looking for documentation on how to fine-tune the Dolly model. Please refer to this documentation on how to fine-tune the model, and let us know if it ...
- 0 kudos
