I am trying to create a custom transformer as a stage in my pipeline. A few of the transformations I am doing via SparkNLP and the next few using MLlib. To pass the result of SparkNLP transformation at a stage to the next MLlib transformation, I need...
The following assignment:from langchain.sql_database import SQLDatabasedbase = SQLDatabase.from_databricks(catalog=catalog, schema=db,host=host, api_token=token,)fails with ValueError: invalid literal for int() with base 10: ''because ofcls._assert_p...
I am also facing the same issue. not able to connect even after using sqlalchemy
Generally speaking what are the tips to make cicd process better with having different versions and models?
Hi @Betul, I think that there are different ways but it really depends on what do you mean by different models and versions.One simple option is to use Databricks Asset Bundles to create multiple workflows (one for each model) and use the champion-ch...
Hi there I have used the Databricks Model Serving Endpoints to serve a model which depends on some config files and a custom library. The library has been included by logging the model with the `code_path` argument in `mlflow.pyfunc.log_model` and it...
Hi @rasgaard, one way to achieve that without inspecting the container is to use MLflow artifacts. Artifacts allow you to log files together with your models and reference them inside the endpoint.For example, let's assume that you need to include a ...
Hi guys,We're facing a weird behavior or we're missing some configuration in our code. I've tried to find some information unsuccessfully. Let me try to explain our case, we have implemented a custom Evaluator in python using PySpark API, something l...
Hi, I was wondering whether model serving endpoints support authentication with Azure Managed Identities.
@larsr Databricks itself supports authentication through Managed Identity and Model Serving Endpoint requires bearer token, so yeah - i suppose it's doable.
I am trying to deploy phi-3 model in databricks but getting below error while creating serving endpoint. Help us on this as soon as possible.
Hello, I'm facing the same issue. No matter what I am trying, I end up with dependencies issues...
I am having a similar issue thislog signature and input data for Spark LinearRegression using mlflow v2.13.0 and using mlflow.pyfunc.log_model to log my model. Starting a new post here since there doesn't seem to be any follow up from the community o...
@Miki can you please share you code for logging the signature with array types
how does data bricks handle. computer vision related use cases? (eg defects detection for a manufacturing industry) is there a reference architecture
My first Data + AI summit and it's been a great experience
Hello EveryoneWe are trying to create an ML pipeline on Databricks using the famous Databricks workflows. Currently our pipeline includes having 3 major components: Data Ingestion, Model Training and Model Testing. My question is whether it is possib...
Hello!I'm having an issue registering a model saved in a mounted S3 bucket using mlflow.Let me give a little bit more context:1. First I mounted my S3 with all the corresponding IAM permissions:s3_bucket_name = f"s3a://{s3_bucket}"dbutils.fs.mount(so...
Are LLMs really ready for production deployment?
You should be careful while putting them to production without guardrails, perhaps using Mosaic AI gateway announced today that would aggregate these functionalities, it should be something to start. These are not the only things you should worry abo...
I'm testing the Databricks Jobs feature with a dbt task and wanted to know if you had any advice for me for managing dbt documentation.I can use "dbt run" commands to run my models then "dbt docs generate" to generate the documentation. But is it pos...
How can I access these target files from the task itself ? I am trying to use dbt's state modifiers for detecting models that changed and only running models when the source freshness changed. Is there an easy way to store and use these state files i...
What's the best option to store your trained ML models
Depending on how many you have, different solutions may be appropriate - and conveniently, you can use MLflow as a front end for most of these if you're working in Python. If you're working on personal projects, a local MLflow instance might be the r...
