- 1070 Views
- 1 replies
- 0 kudos
Not able to run end to end ML project on Databricks Trial
I started using Databricks trial version from today. I want to explore full end to end ML lifecycle on the databricks. I observed for the compute only 'serverless' option is available. I was trying to execute the notebook posted on https://docs.datab...
- 1070 Views
- 1 replies
- 0 kudos
- 0 kudos
I can take up to 15 minutes for the serving endpoint to be created. Once you initiate the "create endpoint" chunk of code go and grab a cup of coffee and wait 15 minutes. Then, before you use it verify it is running (bottom left menu "Serving") by g...
- 0 kudos
- 2115 Views
- 1 replies
- 0 kudos
Resolved! Exploring Serverless Features in Databricks for ML Use Cases
Hello, I need to develop some ML use case. I would like to understand if the serverless functionality unlocks any additional features or if it is mandatory for certain capabilities.Thank you!
- 2115 Views
- 1 replies
- 0 kudos
- 0 kudos
Serverless functionality in Databricks is not mandatory for utilizing machine learning (ML) capabilities. However, it does unlock specific benefits and features that can enhance certain workflows. Here’s how serverless compute can add value, based on...
- 0 kudos
- 7940 Views
- 1 replies
- 4 kudos
Error when reading Excel file: "org.apache.poi.ooxml.POIXMLException: Strict OOXML isn't currently supported, please see bug #57699"
Hi,I want to read an Excel "xlsx" file. The excel file has several sheets and multi-row header. The original file format was "xlsm" and I changed the extension to "xlsx". I try the following code:filepath_xlsx = "dbfs:/FileStore/Sample_Excel/data.xl...
- 7940 Views
- 1 replies
- 4 kudos
- 4 kudos
copying the data onto a newer file solved my issue. Likely issue related to files metadata!
- 4 kudos
- 32818 Views
- 9 replies
- 6 kudos
Spark with LSTM
I am still lost on the Spark and Deep Learning model.If I have a (2D) time series that I want to use for e.g. an LSTM model. Then I first convert it to a 3D array and then pass it to the model. This is normally done in memory with numpy. But what hap...
- 32818 Views
- 9 replies
- 6 kudos
- 6 kudos
Same problem as @imgaboy here, is the solution was to save into table our inputs after formating them ready to feed the lstm and just turn 2d to 3d via datagenerator??
- 6 kudos
- 5899 Views
- 6 replies
- 3 kudos
Resolved! Nested runs don't group correctly in MLflow
How do I get MLflow child runs to appear as children of their parent run in the MLflow GUI, if I'm choosing my own experiment location instead of letting everything be written to the default experiment location?If I run the standard tutorial (https:/...
- 5899 Views
- 6 replies
- 3 kudos
- 3 kudos
OK, here's more info about what's wrong, and a solution.I used additional parameter logging to determine that no matter how I adjust the parameters of the inner call to ```mlflow.start_run()```the `experiment_id` parameter of the child runs differs f...
- 3 kudos
- 1486 Views
- 2 replies
- 0 kudos
GitHub Actions workflow cannot find the Databricks Unity Catalog and its tables
Context: Running the train_model_py.py file stored in Databricks through GitHub Actions. The notebook reads the Unity Catalog tables for pre-processing and works fine when run through the Databricks UI. However, it gives an error when run through Git...
- 1486 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sagarb, It sounds like a permission issue or setup issue... what is the error you are hitting?
- 0 kudos
- 3149 Views
- 1 replies
- 0 kudos
Resolved! Custom model serving using Databricks Asset Bundles
I am using MLFlow to register custom model (python model) in Unity Catalog, and Databricks Asset Bundle to create a serving endpoint for that custom model. I was able to create the serving endpoint using DABs, but I want to deploy the model by using ...
- 3149 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MLOperator Since model_serving_endpoints only accepts a version number of a served entity, I think that is not possible. However, the get-by-alias version API can be used to retrieve a version number from a model alias name. Then the model name...
- 0 kudos
- 6379 Views
- 1 replies
- 0 kudos
Convert the tensorflow datatset to numpy tuples
Hello everyone ,Here are the sequence of steps i have followed:1. I have used petastorm to convert the spark dataframe to tf.datasetimport numpy as np# Read the Petastorm dataset and convert it to TensorFlow Datasetwith converter.make_tf_dataset() as...
- 6379 Views
- 1 replies
- 0 kudos
- 0 kudos
The error occurs because make_tf_dataset() returns an inferred_schema_view object, which is a Petastorm wrapper representing the dataset schema. This object does not have a .numpy() attribute, so calling batch.numpy() will throw the AttributeError. ...
- 0 kudos
- 1186 Views
- 1 replies
- 0 kudos
Interactive EDA task in a Job Workflow
I am trying to configure an interactive EDA task as part of a job workflow. I'd like to be able to trigger a workflow, perform some basic analysis then proceed to a subsequent task. I haven't had any success freezing execution. Also, the job workflow...
- 1186 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @cmd0160, Freezing job execution to perform interactive tasks directly within a job workflow is not natively supported in Databricks. The job workflow UI and the notebook UI serve different purposes, and the interactive capabilities you find in...
- 0 kudos
- 4726 Views
- 5 replies
- 1 kudos
DatabricksApiException Error in Microsoft Azure Databricks
I am doing a course on Machine Learning Associate course , at the starting itseld i am getting error while running in Azure Databricks.Can somebody help me in solving this error.
- 4726 Views
- 5 replies
- 1 kudos
- 1 kudos
The error message indicates that Workspace Feature Store has been deprecated in your Azure Databricks workspace. The error occurs because the Feature Store API is no longer supported in your environment.How to Fix It:Check If Your Databricks Workspac...
- 1 kudos
- 1785 Views
- 2 replies
- 1 kudos
deploy, train and monitor AI/ML model in databricks in automated way.
Hi Team, I have my databricks environment where I want to deploy, train and monitor ML model in automated way(github action). How I can do that?
- 1785 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi there @ncparab13,- https://docs.databricks.com/aws/en/dev-tools/bundles/mlops-stacks ,- https://docs.databricks.com/aws/en/machine-learning/mlops/ci-cd-for-ml , - https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-reposHere are some li...
- 1 kudos
- 4389 Views
- 3 replies
- 1 kudos
Gemini though Mosaic Gateway
I am trying to configure the Gemini Vertex API in Databricks. In simple Python code, everything works fine, which indicates that I have correctly set up the API and credentials. Error message: {"error_code":"INVALID_PARAMETER_VALUE","message":"INVALI...
- 4389 Views
- 3 replies
- 1 kudos
- 1 kudos
With support from a helpful Databricks employee, we found out that the problem was that the `private_key` / `private_key_plaintext` field needs to be the entire JSON object that GCP creates for the service account not just the private key string from...
- 1 kudos
- 838 Views
- 1 replies
- 0 kudos
unable to Publish Notebook
Hi,I am unable to publish Notebook from my workspace in community editionIt just give me blank error message
- 838 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Saty1 Publishing a notebook in Databricks Community Edition can sometimes encounter issues due to various reasons, such as browser compatibility, network issues, or limitations within the Community Edition itself. Here are some steps you can take...
- 0 kudos
- 1517 Views
- 2 replies
- 0 kudos
Unable to convert R dataframe to spark dataframe
Hi All, Does anyone knows how to convert R dataframe to spark dataframe to Pandas dataframe? I wanted to get a Pandas dataframe ultimately but I guess I need to convert to spark first. I've been using this sparklyr library but my code did not work. T...
- 1517 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Paddy_chu, Here's an updated version of the R code: %r library(sparklyr) library(SparkR) sc <- spark_connect(method = "databricks") matched_rdf <- psm_tbl %>% select(c(code_treat, code_control)) %>% data.frame() # Write the R dataframe t...
- 0 kudos
- 12251 Views
- 4 replies
- 2 kudos
MetadataChangedException Exception in databricks
Reading around 20 text files from ADLS, doing some transformations, and after that these files are written back to ADLS as a single delta file (all operations are in parallel through the thread pool). Here from 20 threads, it is writing to a single f...
- 12251 Views
- 4 replies
- 2 kudos
- 2 kudos
How can we import the exception "MetadataChangedException"?Or does Databricks recommend to catch / except Exception and parse the string?
- 2 kudos
