Hi,I have a use case of developing an entire ML solution within Databricks starting from ingestion to inference and monitoring, but the problem is that we have unstructured data containing Images and Video for training the model using frameworks such...
Context: Running the train_model_py.py file stored in Databricks through GitHub Actions. The notebook reads the Unity Catalog tables for pre-processing and works fine when run through the Databricks UI. However, it gives an error when run through Git...
Hi @sagarb, It sounds like a permission issue or setup issue... what is the error you are hitting?
I have an issue with databricks-vectorsearch package. Version 0.51 suddenly stopped working this week because:It now expected me to provide azure_tenant_id in addition to service principal's client ID and secret.After supplying tenant ID, it showed s...
I am using MLFlow to register custom model (python model) in Unity Catalog, and Databricks Asset Bundle to create a serving endpoint for that custom model. I was able to create the serving endpoint using DABs, but I want to deploy the model by using ...
Hi @MLOperator Since model_serving_endpoints only accepts a version number of a served entity, I think that is not possible. However, the get-by-alias version API can be used to retrieve a version number from a model alias name. Then the model name...
Hello everyone ,Here are the sequence of steps i have followed:1. I have used petastorm to convert the spark dataframe to tf.datasetimport numpy as np# Read the Petastorm dataset and convert it to TensorFlow Datasetwith converter.make_tf_dataset() as...
The error occurs because make_tf_dataset() returns an inferred_schema_view object, which is a Petastorm wrapper representing the dataset schema. This object does not have a .numpy() attribute, so calling batch.numpy() will throw the AttributeError. ...
I am trying to configure an interactive EDA task as part of a job workflow. I'd like to be able to trigger a workflow, perform some basic analysis then proceed to a subsequent task. I haven't had any success freezing execution. Also, the job workflow...
Hello @cmd0160, Freezing job execution to perform interactive tasks directly within a job workflow is not natively supported in Databricks. The job workflow UI and the notebook UI serve different purposes, and the interactive capabilities you find in...
I am doing a course on Machine Learning Associate course , at the starting itseld i am getting error while running in Azure Databricks.Can somebody help me in solving this error.
The error message indicates that Workspace Feature Store has been deprecated in your Azure Databricks workspace. The error occurs because the Feature Store API is no longer supported in your environment.How to Fix It:Check If Your Databricks Workspac...
Hi Team, I have my databricks environment where I want to deploy, train and monitor ML model in automated way(github action). How I can do that?
Hi there @ncparab13,- https://docs.databricks.com/aws/en/dev-tools/bundles/mlops-stacks ,- https://docs.databricks.com/aws/en/machine-learning/mlops/ci-cd-for-ml , - https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-reposHere are some li...
I am trying to configure the Gemini Vertex API in Databricks. In simple Python code, everything works fine, which indicates that I have correctly set up the API and credentials. Error message: {"error_code":"INVALID_PARAMETER_VALUE","message":"INVALI...
With support from a helpful Databricks employee, we found out that the problem was that the `private_key` / `private_key_plaintext` field needs to be the entire JSON object that GCP creates for the service account not just the private key string from...
Hi,I am currently using PySpark version 3.5.0 on my Databricks cluster. Despite setting the required configuration using the command: spark.conf.set("spark.databricks.ml.whitelist", "true"), I am still encountering an issue while trying to use the Ve...
Glad to hear it works for you now! The ML runtime has variety of preinstalled integrations such as MLflow, which provides ML lifecycle management, MLOps ... etc. Please explore them if you haven't done it already, to establish benefits of the extra
Hi,I am unable to publish Notebook from my workspace in community editionIt just give me blank error message
Hi @Saty1 Publishing a notebook in Databricks Community Edition can sometimes encounter issues due to various reasons, such as browser compatibility, network issues, or limitations within the Community Edition itself. Here are some steps you can take...
Hello,I recently discovered the sparklyr package and found it quite useful. After setting up the Spark connection, I can apply dplyr functions to manipulate large tables. However, it seems that any functions outside of dplyr cannot be used on Spark v...
I am working on a personalized price package recommendation and implemented an AutoGluon code integrating it with MLflow.The code has been created in a modular fashion to be used by other team members. They just need to pass the data, target column a...
I've built a custom MLFlow model class which I know functions. As part of a given run the model class uses `joblib.dump` to store necessary parameters on the databricks DBFS before logging them as artifacts in the MLFlow run. This works fine when usi...
I have recently been able to run AutoML successfully on a certain dataset. But it has just failed on a second dataset of similar construction, before being able to produce any machine learning training runs or output. The Experiments page says```Mo...
