- 32991 Views
- 9 replies
- 6 kudos
Spark with LSTM
I am still lost on the Spark and Deep Learning model.If I have a (2D) time series that I want to use for e.g. an LSTM model. Then I first convert it to a 3D array and then pass it to the model. This is normally done in memory with numpy. But what hap...
- 32991 Views
- 9 replies
- 6 kudos
- 6 kudos
Same problem as @imgaboy here, is the solution was to save into table our inputs after formating them ready to feed the lstm and just turn 2d to 3d via datagenerator??
- 6 kudos
- 6299 Views
- 6 replies
- 3 kudos
Resolved! Nested runs don't group correctly in MLflow
How do I get MLflow child runs to appear as children of their parent run in the MLflow GUI, if I'm choosing my own experiment location instead of letting everything be written to the default experiment location?If I run the standard tutorial (https:/...
- 6299 Views
- 6 replies
- 3 kudos
- 3 kudos
OK, here's more info about what's wrong, and a solution.I used additional parameter logging to determine that no matter how I adjust the parameters of the inner call to ```mlflow.start_run()```the `experiment_id` parameter of the child runs differs f...
- 3 kudos
- 1529 Views
- 2 replies
- 0 kudos
GitHub Actions workflow cannot find the Databricks Unity Catalog and its tables
Context: Running the train_model_py.py file stored in Databricks through GitHub Actions. The notebook reads the Unity Catalog tables for pre-processing and works fine when run through the Databricks UI. However, it gives an error when run through Git...
- 1529 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sagarb, It sounds like a permission issue or setup issue... what is the error you are hitting?
- 0 kudos
- 3312 Views
- 1 replies
- 0 kudos
Resolved! Custom model serving using Databricks Asset Bundles
I am using MLFlow to register custom model (python model) in Unity Catalog, and Databricks Asset Bundle to create a serving endpoint for that custom model. I was able to create the serving endpoint using DABs, but I want to deploy the model by using ...
- 3312 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MLOperator Since model_serving_endpoints only accepts a version number of a served entity, I think that is not possible. However, the get-by-alias version API can be used to retrieve a version number from a model alias name. Then the model name...
- 0 kudos
- 6413 Views
- 1 replies
- 0 kudos
Convert the tensorflow datatset to numpy tuples
Hello everyone ,Here are the sequence of steps i have followed:1. I have used petastorm to convert the spark dataframe to tf.datasetimport numpy as np# Read the Petastorm dataset and convert it to TensorFlow Datasetwith converter.make_tf_dataset() as...
- 6413 Views
- 1 replies
- 0 kudos
- 0 kudos
The error occurs because make_tf_dataset() returns an inferred_schema_view object, which is a Petastorm wrapper representing the dataset schema. This object does not have a .numpy() attribute, so calling batch.numpy() will throw the AttributeError. ...
- 0 kudos
- 1202 Views
- 1 replies
- 0 kudos
Interactive EDA task in a Job Workflow
I am trying to configure an interactive EDA task as part of a job workflow. I'd like to be able to trigger a workflow, perform some basic analysis then proceed to a subsequent task. I haven't had any success freezing execution. Also, the job workflow...
- 1202 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @cmd0160, Freezing job execution to perform interactive tasks directly within a job workflow is not natively supported in Databricks. The job workflow UI and the notebook UI serve different purposes, and the interactive capabilities you find in...
- 0 kudos
- 4820 Views
- 5 replies
- 1 kudos
DatabricksApiException Error in Microsoft Azure Databricks
I am doing a course on Machine Learning Associate course , at the starting itseld i am getting error while running in Azure Databricks.Can somebody help me in solving this error.
- 4820 Views
- 5 replies
- 1 kudos
- 1 kudos
The error message indicates that Workspace Feature Store has been deprecated in your Azure Databricks workspace. The error occurs because the Feature Store API is no longer supported in your environment.How to Fix It:Check If Your Databricks Workspac...
- 1 kudos
- 1838 Views
- 2 replies
- 1 kudos
deploy, train and monitor AI/ML model in databricks in automated way.
Hi Team, I have my databricks environment where I want to deploy, train and monitor ML model in automated way(github action). How I can do that?
- 1838 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi there @ncparab13,- https://docs.databricks.com/aws/en/dev-tools/bundles/mlops-stacks ,- https://docs.databricks.com/aws/en/machine-learning/mlops/ci-cd-for-ml , - https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-reposHere are some li...
- 1 kudos
- 4475 Views
- 3 replies
- 1 kudos
Gemini though Mosaic Gateway
I am trying to configure the Gemini Vertex API in Databricks. In simple Python code, everything works fine, which indicates that I have correctly set up the API and credentials. Error message: {"error_code":"INVALID_PARAMETER_VALUE","message":"INVALI...
- 4475 Views
- 3 replies
- 1 kudos
- 1 kudos
With support from a helpful Databricks employee, we found out that the problem was that the `private_key` / `private_key_plaintext` field needs to be the entire JSON object that GCP creates for the service account not just the private key string from...
- 1 kudos
- 879 Views
- 1 replies
- 0 kudos
unable to Publish Notebook
Hi,I am unable to publish Notebook from my workspace in community editionIt just give me blank error message
- 879 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Saty1 Publishing a notebook in Databricks Community Edition can sometimes encounter issues due to various reasons, such as browser compatibility, network issues, or limitations within the Community Edition itself. Here are some steps you can take...
- 0 kudos
- 1596 Views
- 2 replies
- 0 kudos
Unable to convert R dataframe to spark dataframe
Hi All, Does anyone knows how to convert R dataframe to spark dataframe to Pandas dataframe? I wanted to get a Pandas dataframe ultimately but I guess I need to convert to spark first. I've been using this sparklyr library but my code did not work. T...
- 1596 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Paddy_chu, Here's an updated version of the R code: %r library(sparklyr) library(SparkR) sc <- spark_connect(method = "databricks") matched_rdf <- psm_tbl %>% select(c(code_treat, code_control)) %>% data.frame() # Write the R dataframe t...
- 0 kudos
- 12362 Views
- 4 replies
- 2 kudos
MetadataChangedException Exception in databricks
Reading around 20 text files from ADLS, doing some transformations, and after that these files are written back to ADLS as a single delta file (all operations are in parallel through the thread pool). Here from 20 threads, it is writing to a single f...
- 12362 Views
- 4 replies
- 2 kudos
- 2 kudos
How can we import the exception "MetadataChangedException"?Or does Databricks recommend to catch / except Exception and parse the string?
- 2 kudos
- 11390 Views
- 4 replies
- 1 kudos
Permission denied: Lightning Logs
I'm doing parameter tuning for a NeuralProphet model (you can see in the image the parameters and code for training)When I try to parallelize the training, it gives me Permission Error.Why can't I access the folder '/databricks/spark/work/*'? Do I ne...
- 11390 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Ruben!I am facing exactly the same error running a similar approach when using runtime 16.2 ML. I didn't have this issue when using runtime 12.2 LTS ML or 13.3 ML. Did you find a solution?Many thanks!
- 1 kudos
- 3354 Views
- 2 replies
- 0 kudos
Can we retrieve experiment results via MLflow API or is this only possible using UI?
Yes, you can use the API https://www.mlflow.org/docs/latest/python_api/index.html
- 3354 Views
- 2 replies
- 0 kudos
- 0 kudos
And how about tracing data? Do you know how to read likespark.read.format("mlflow-experiment").load() ?
- 0 kudos
- 6948 Views
- 6 replies
- 1 kudos
Hosting R apps and models
Hello!Has anyone tried to host an R Shiny app using Databricks apps? It is very useful for Python apps, but it would be nice to know if anyone have any workaround to make it work with R Shiny Apps.We also wonder about Databricks model hosting for R m...
- 6948 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @parthSundarka!We tried, but had some problems making it work. Do you have a more complete code example? That would be very helpfull!BR, Anton
- 1 kudos
