- 1055 Views
- 2 replies
- 0 kudos
GitHub Actions workflow cannot find the Databricks Unity Catalog and its tables
Context: Running the train_model_py.py file stored in Databricks through GitHub Actions. The notebook reads the Unity Catalog tables for pre-processing and works fine when run through the Databricks UI. However, it gives an error when run through Git...
- 1055 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sagarb, It sounds like a permission issue or setup issue... what is the error you are hitting?
- 0 kudos
- 2018 Views
- 0 replies
- 0 kudos
databricks-vectorsearch 0.53 unable to use similarity_search()
I have an issue with databricks-vectorsearch package. Version 0.51 suddenly stopped working this week because:It now expected me to provide azure_tenant_id in addition to service principal's client ID and secret.After supplying tenant ID, it showed s...
- 2018 Views
- 0 replies
- 0 kudos
- 2266 Views
- 1 replies
- 0 kudos
Resolved! Custom model serving using Databricks Asset Bundles
I am using MLFlow to register custom model (python model) in Unity Catalog, and Databricks Asset Bundle to create a serving endpoint for that custom model. I was able to create the serving endpoint using DABs, but I want to deploy the model by using ...
- 2266 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MLOperator Since model_serving_endpoints only accepts a version number of a served entity, I think that is not possible. However, the get-by-alias version API can be used to retrieve a version number from a model alias name. Then the model name...
- 0 kudos
- 2991 Views
- 1 replies
- 0 kudos
Convert the tensorflow datatset to numpy tuples
Hello everyone ,Here are the sequence of steps i have followed:1. I have used petastorm to convert the spark dataframe to tf.datasetimport numpy as np# Read the Petastorm dataset and convert it to TensorFlow Datasetwith converter.make_tf_dataset() as...
- 2991 Views
- 1 replies
- 0 kudos
- 0 kudos
The error occurs because make_tf_dataset() returns an inferred_schema_view object, which is a Petastorm wrapper representing the dataset schema. This object does not have a .numpy() attribute, so calling batch.numpy() will throw the AttributeError. ...
- 0 kudos
- 938 Views
- 1 replies
- 0 kudos
Interactive EDA task in a Job Workflow
I am trying to configure an interactive EDA task as part of a job workflow. I'd like to be able to trigger a workflow, perform some basic analysis then proceed to a subsequent task. I haven't had any success freezing execution. Also, the job workflow...
- 938 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @cmd0160, Freezing job execution to perform interactive tasks directly within a job workflow is not natively supported in Databricks. The job workflow UI and the notebook UI serve different purposes, and the interactive capabilities you find in...
- 0 kudos
- 3779 Views
- 5 replies
- 1 kudos
DatabricksApiException Error in Microsoft Azure Databricks
I am doing a course on Machine Learning Associate course , at the starting itseld i am getting error while running in Azure Databricks.Can somebody help me in solving this error.
- 3779 Views
- 5 replies
- 1 kudos
- 1 kudos
The error message indicates that Workspace Feature Store has been deprecated in your Azure Databricks workspace. The error occurs because the Feature Store API is no longer supported in your environment.How to Fix It:Check If Your Databricks Workspac...
- 1 kudos
- 1256 Views
- 2 replies
- 1 kudos
deploy, train and monitor AI/ML model in databricks in automated way.
Hi Team, I have my databricks environment where I want to deploy, train and monitor ML model in automated way(github action). How I can do that?
- 1256 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi there @ncparab13,- https://docs.databricks.com/aws/en/dev-tools/bundles/mlops-stacks ,- https://docs.databricks.com/aws/en/machine-learning/mlops/ci-cd-for-ml , - https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-reposHere are some li...
- 1 kudos
- 3638 Views
- 3 replies
- 1 kudos
Gemini though Mosaic Gateway
I am trying to configure the Gemini Vertex API in Databricks. In simple Python code, everything works fine, which indicates that I have correctly set up the API and credentials. Error message: {"error_code":"INVALID_PARAMETER_VALUE","message":"INVALI...
- 3638 Views
- 3 replies
- 1 kudos
- 1 kudos
With support from a helpful Databricks employee, we found out that the problem was that the `private_key` / `private_key_plaintext` field needs to be the entire JSON object that GCP creates for the service account not just the private key string from...
- 1 kudos
- 4130 Views
- 6 replies
- 2 kudos
Unable to Use VectorAssembler in PySpark 3.5.0 Due to Whitelisting
Hi,I am currently using PySpark version 3.5.0 on my Databricks cluster. Despite setting the required configuration using the command: spark.conf.set("spark.databricks.ml.whitelist", "true"), I am still encountering an issue while trying to use the Ve...
- 4130 Views
- 6 replies
- 2 kudos
- 2 kudos
Glad to hear it works for you now! The ML runtime has variety of preinstalled integrations such as MLflow, which provides ML lifecycle management, MLOps ... etc. Please explore them if you haven't done it already, to establish benefits of the extra
- 2 kudos
- 547 Views
- 1 replies
- 0 kudos
unable to Publish Notebook
Hi,I am unable to publish Notebook from my workspace in community editionIt just give me blank error message
- 547 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Saty1 Publishing a notebook in Databricks Community Edition can sometimes encounter issues due to various reasons, such as browser compatibility, network issues, or limitations within the Community Edition itself. Here are some steps you can take...
- 0 kudos
- 1980 Views
- 0 replies
- 0 kudos
How to transpose spark dataframe using R API?
Hello,I recently discovered the sparklyr package and found it quite useful. After setting up the Spark connection, I can apply dplyr functions to manipulate large tables. However, it seems that any functions outside of dplyr cannot be used on Spark v...
- 1980 Views
- 0 replies
- 0 kudos
- 2492 Views
- 0 replies
- 0 kudos
AutoGluon MLflow integration
I am working on a personalized price package recommendation and implemented an AutoGluon code integrating it with MLflow.The code has been created in a modular fashion to be used by other team members. They just need to pass the data, target column a...
- 2492 Views
- 0 replies
- 0 kudos
- 2091 Views
- 0 replies
- 0 kudos
Pickle/joblib.dump a pre-processing function defined in a notebook
I've built a custom MLFlow model class which I know functions. As part of a given run the model class uses `joblib.dump` to store necessary parameters on the databricks DBFS before logging them as artifacts in the MLFlow run. This works fine when usi...
- 2091 Views
- 0 replies
- 0 kudos
- 2125 Views
- 0 replies
- 0 kudos
AutoML master notebook failing
I have recently been able to run AutoML successfully on a certain dataset. But it has just failed on a second dataset of similar construction, before being able to produce any machine learning training runs or output. The Experiments page says```Mo...
- 2125 Views
- 0 replies
- 0 kudos
- 927 Views
- 2 replies
- 0 kudos
Unable to convert R dataframe to spark dataframe
Hi All, Does anyone knows how to convert R dataframe to spark dataframe to Pandas dataframe? I wanted to get a Pandas dataframe ultimately but I guess I need to convert to spark first. I've been using this sparklyr library but my code did not work. T...
- 927 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Paddy_chu, Here's an updated version of the R code: %r library(sparklyr) library(SparkR) sc <- spark_connect(method = "databricks") matched_rdf <- psm_tbl %>% select(c(code_treat, code_control)) %>% data.frame() # Write the R dataframe t...
- 0 kudos
