- 697 Views
- 2 replies
- 0 kudos
Unable to convert R dataframe to spark dataframe
Hi All, Does anyone knows how to convert R dataframe to spark dataframe to Pandas dataframe? I wanted to get a Pandas dataframe ultimately but I guess I need to convert to spark first. I've been using this sparklyr library but my code did not work. T...
- 697 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Paddy_chu, Here's an updated version of the R code: %r library(sparklyr) library(SparkR) sc <- spark_connect(method = "databricks") matched_rdf <- psm_tbl %>% select(c(code_treat, code_control)) %>% data.frame() # Write the R dataframe t...
- 0 kudos
- 10850 Views
- 4 replies
- 2 kudos
MetadataChangedException Exception in databricks
Reading around 20 text files from ADLS, doing some transformations, and after that these files are written back to ADLS as a single delta file (all operations are in parallel through the thread pool). Here from 20 threads, it is writing to a single f...
- 10850 Views
- 4 replies
- 2 kudos
- 2 kudos
How can we import the exception "MetadataChangedException"?Or does Databricks recommend to catch / except Exception and parse the string?
- 2 kudos
- 1050 Views
- 0 replies
- 0 kudos
Vector Index Creation for external embedding model takes a lot of time
I have embedding model endpoint created and served. It is huggingface model which databricks doesnt provide. I am using this model to create vector search index however this takes a lot of time to get created. I observed that when I use databricks of...
- 1050 Views
- 0 replies
- 0 kudos
- 8979 Views
- 4 replies
- 1 kudos
Permission denied: Lightning Logs
I'm doing parameter tuning for a NeuralProphet model (you can see in the image the parameters and code for training)When I try to parallelize the training, it gives me Permission Error.Why can't I access the folder '/databricks/spark/work/*'? Do I ne...
- 8979 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Ruben!I am facing exactly the same error running a similar approach when using runtime 16.2 ML. I didn't have this issue when using runtime 12.2 LTS ML or 13.3 ML. Did you find a solution?Many thanks!
- 1 kudos
- 2386 Views
- 2 replies
- 0 kudos
Can we retrieve experiment results via MLflow API or is this only possible using UI?
Yes, you can use the API https://www.mlflow.org/docs/latest/python_api/index.html
- 2386 Views
- 2 replies
- 0 kudos
- 0 kudos
And how about tracing data? Do you know how to read likespark.read.format("mlflow-experiment").load() ?
- 0 kudos
- 4164 Views
- 6 replies
- 1 kudos
Hosting R apps and models
Hello!Has anyone tried to host an R Shiny app using Databricks apps? It is very useful for Python apps, but it would be nice to know if anyone have any workaround to make it work with R Shiny Apps.We also wonder about Databricks model hosting for R m...
- 4164 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @parthSundarka!We tried, but had some problems making it work. Do you have a more complete code example? That would be very helpfull!BR, Anton
- 1 kudos
- 1482 Views
- 5 replies
- 0 kudos
Community Edition workspace not found
Suddenly got logout from my account in the Community Edition. When I tried to login again, I received this error message: "We were not able to find a Community Edition workspace with this email. Please login to accounts.cloud.databricks.com to find t...
- 1482 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @all,I solved the issue by signing up for a new Community Edition account using the same email.The only downside is I lost all my notebooks :(.
- 0 kudos
- 1116 Views
- 0 replies
- 0 kudos
Working with pyspark dataframe with machine learning libraries / statistical model libraries
Hi Team, I am working with huge volume of data (50GB) and i decompose the time series data using the statsmodel.Having said that the major challenge i am facing is the compatibility of the pyspark dataframe with the machine learning algorithms. altho...
- 1116 Views
- 0 replies
- 0 kudos
- 2010 Views
- 4 replies
- 2 kudos
Resolved! Facing issues with passing memory checkpointer in lanngraph agents
Hi,I am trying to create a simple langgraph agent in Databricks, the agent also uses lanngraph memory checkpoint which enables to store the state of the graph. This is working fine when I am trying it in Databricks notebook, but when I tried to log t...
- 2010 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi all. I have the same issue, could you deploy the graph with the MemorySaver?
- 2 kudos
- 1101 Views
- 0 replies
- 1 kudos
Proper mlflow run logging with SparkTrials and Hyperopt
Hello!I'm attempting to run a hyperparameter search using hyperopt and SparkTrials(), and log the resulting runs to an existing experiment (experiment A). I can see on this page that databricks suggests wrapping the `fmin()` call within a `mlflow.sta...
- 1101 Views
- 0 replies
- 1 kudos
- 1159 Views
- 1 replies
- 0 kudos
Error when creating model env using 'virtualenv' with DBR 14.3
We were trying to inference from a logged model but had the following errorPreviously, we had been using `conda` as the environment manager, but that is no longer supported. I tried to update pyenv as some suggested but didn't get anywhere. Any insig...
- 1159 Views
- 1 replies
- 0 kudos
- 0 kudos
Happened to me too, and pyenv update command is missing , any workarounds here?
- 0 kudos
- 3746 Views
- 2 replies
- 2 kudos
Resolved! Understanding compute requirements for Deploying Deepseek-R1-Distilled-Llama Models on databricks
Hi I have read the blog Deploying Deepseek-R1-Distilled-Llama Models on Databricks at https://www.databricks.com/blog/deepseek-r1-databricksI am new to using custom models that are not available as part of foundation models.According to the blog, I n...
- 3746 Views
- 2 replies
- 2 kudos
- 1582 Views
- 2 replies
- 0 kudos
Foundation Model APIs HIPAA compliance
I saw that Foundation Model API is not HIPAA compliant. Is there a timeline in which we could expect it to be HIPAA compliant? I work for a healthcare company with a BAA with Databricks.
- 1582 Views
- 2 replies
- 0 kudos
- 0 kudos
It is now HIPAA compliant: Provisioned throughput endpoints are available with compliance certifications like HIPAA, Model Serving is HIPAA compliant in all regions
- 0 kudos
- 1206 Views
- 2 replies
- 1 kudos
Resolved! Feature store and medallion data location
Hello Folks,If we have 3 environments (dev/preprod/prod) and would like to have medallion data shared among them - I guess delta share is a good way to go. Now if we want to use "Feature Store (FS)" then I am a bit confused and seeking some clarity. ...
- 1206 Views
- 2 replies
- 1 kudos
- 1073 Views
- 0 replies
- 0 kudos
TorchDistributor: installation of custom python package via wheel across all nodes in cluster
I am trying to set up a training pipeline of a distributed PyTorch model using TorchDistributor. I have defined a train_object (in my case it is a Callable) that runs my training code. However, this method requires custom code from modules that I hav...
- 1073 Views
- 0 replies
- 0 kudos
