I am training Spark ML model (concretely a SynapseML LightGBM ) in Databricks using mlflow and autologWhen I try to register my model in Unity catalog I get the following error: MlflowException: Model passed for registration contained a signature th...
How to get the content of a returned variable in zenml without having this error:TypeError: float() argument must be a string or a number, not 'StepArtifact'?
Hi everybody, I have a question re: the deployment as code pattern on databricks. I found and watched a great demo here: https://www.youtube.com/watch?v=JApPzAnbfPIMy question is, in the case where I can get read access to prod data in dev env, the d...
HelloCurrently we have multiple ML models running in Production which are logging metrics and other meta-data on mlflow. I wanted to ask is it possible somehow to build Databricks dashboards on top of this data and also can this data be somehow avail...
Hello @Retired_mod Thanks for responding. I think you are talking about using the Python API. But we don't want that is it possible since MLFlow also uses an sql table to store metrics. To expose those tables as a part of our meta-store and build da...
Hi everybody. I am relatively new to Databricks. I am working on an ML model promotion process between different Databricks workspaces. I am aware that best practice should be deployment as code (e.g. export the whole training pipeline and model regi...
I am aware that models registered in Databricks Unity Catalog (UC) in the prod workspace can be loaded from dev workspace for model comparison/debugging. But to comply with best practices, we restrict access to assets in UC in the dev workspace fro...
I read the link about Databricks ARC - https://github.com/databricks-industry-solutions/auto-data-linkageand run on DBR 12.2 LTS ML runtime environment on DB cloud communityBut I got the error below: 2024/07/08 04:25:33 INFO mlflow.tracking.fluent: E...
Hi, I was trying to update a config for an endpoint, by adding a new version of an entity (version 7). The new model entered "DEPLOYMENT_READY" state, but the deployment failed with timed out exception. I didn't get any other exception in Build or Se...
Hi @adrianna2942842, Thank you for contacting the Databricks community. May I know how you are loading the model?
Does Pyspark tree based models have iterative or augmented training capabilities ? Similar to sklearn package can be used to train models using model artifact and use that model to train using additional data? #ML_Models_Pyspark
Hi @ChanduBhujang, Thank you for contacting Databricks community. PySpark tree-based models do not have built-in iterative or augmented training capabilities like Scikit-learn's partial_fit method. While there are workarounds to update the model wit...
Hello,I'm following courses on the Databricks academy and using for that purpose the Databricks Community edition using a runtime 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) and I believe it can't be changedI'm following the Data engineering c...
I was facing the same error. This could be resolved by adding the version that you are currently working with in the config function present in '_common' notebook in the "Includes' folder. (This was the case of my folder structure that I downloaded f...
Hi All,I am unable to execute "Classroom-Setup-02.2" setup in Data Engineering Course. There is the following error: FileNotFoundError: [errno 2] no such file or directory: '/dbfs/mnt/dbacademy-datasets/data-engineer-learning-path/v01/ecommerce/raw/u...
Inspired by https://stackoverflow.com/questions/58984925/pandas-missing-read-parquet-function-in-azure-databricks-notebookI changed df = pd.read_parquet(path = datasource_path.replace("dbfs:/", '/dbfs/')) # original, error!intodf = spark.read.format(...
Trying to implement an Early Stopping mechanism in SparkXGBRegressor model with Pipeline: from pyspark.ml.feature import VectorAssembler, StringIndexer from pyspark.ml import Pipeline, PipelineModel from xgboost.spark import SparkXGBRegressor from x...
Ok, I finally solved it - added a column to the dataset validation_indicator_col='validation_0', and did not pass it the the VectorAssembler:xgboost_regressor = SparkXGBRegressor() xgboost_regressor.setParams( gamma=0.2, max_depth=6, obje...
I am trying to create a custom transformer as a stage in my pipeline. A few of the transformations I am doing via SparkNLP and the next few using MLlib. To pass the result of SparkNLP transformation at a stage to the next MLlib transformation, I need...
The following assignment:from langchain.sql_database import SQLDatabasedbase = SQLDatabase.from_databricks(catalog=catalog, schema=db,host=host, api_token=token,)fails with ValueError: invalid literal for int() with base 10: ''because ofcls._assert_p...
I am also facing the same issue. not able to connect even after using sqlalchemy
Generally speaking what are the tips to make cicd process better with having different versions and models?
Hi @Betul, I think that there are different ways but it really depends on what do you mean by different models and versions.One simple option is to use Databricks Asset Bundles to create multiple workflows (one for each model) and use the champion-ch...
Hi there I have used the Databricks Model Serving Endpoints to serve a model which depends on some config files and a custom library. The library has been included by logging the model with the `code_path` argument in `mlflow.pyfunc.log_model` and it...
Hi @rasgaard, one way to achieve that without inspecting the container is to use MLflow artifacts. Artifacts allow you to log files together with your models and reference them inside the endpoint.For example, let's assume that you need to include a ...
