- 4864 Views
- 1 replies
- 0 kudos
How to implement early stop in SparkXGBRegressor with Pipeline?
Trying to implement an Early Stopping mechanism in SparkXGBRegressor model with Pipeline: from pyspark.ml.feature import VectorAssembler, StringIndexer from pyspark.ml import Pipeline, PipelineModel from xgboost.spark import SparkXGBRegressor from x...
- 4864 Views
- 1 replies
- 0 kudos
- 0 kudos
Ok, I finally solved it - added a column to the dataset validation_indicator_col='validation_0', and did not pass it the the VectorAssembler:xgboost_regressor = SparkXGBRegressor() xgboost_regressor.setParams( gamma=0.2, max_depth=6, obje...
- 0 kudos
- 2389 Views
- 0 replies
- 1 kudos
Pyspark custom Transformer class -AttributeError: 'DummyMod' object has no attribute 'MyTransformer'
I am trying to create a custom transformer as a stage in my pipeline. A few of the transformations I am doing via SparkNLP and the next few using MLlib. To pass the result of SparkNLP transformation at a stage to the next MLlib transformation, I need...
- 2389 Views
- 0 replies
- 1 kudos
- 4906 Views
- 3 replies
- 1 kudos
port undefined error in SQLDatabase.from_databricks (langchain.sql_database)
The following assignment:from langchain.sql_database import SQLDatabasedbase = SQLDatabase.from_databricks(catalog=catalog, schema=db,host=host, api_token=token,)fails with ValueError: invalid literal for int() with base 10: ''because ofcls._assert_p...
- 4906 Views
- 3 replies
- 1 kudos
- 1 kudos
I am also facing the same issue. not able to connect even after using sqlalchemy
- 1 kudos
- 1300 Views
- 1 replies
- 0 kudos
How to do cicd with different models/versions using databricks resources?
Generally speaking what are the tips to make cicd process better with having different versions and models?
- 1300 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Betul, I think that there are different ways but it really depends on what do you mean by different models and versions.One simple option is to use Databricks Asset Bundles to create multiple workflows (one for each model) and use the champion-ch...
- 0 kudos
- 4004 Views
- 1 replies
- 0 kudos
Model Serving Endpoints - Build configuration and Interactive access
Hi there I have used the Databricks Model Serving Endpoints to serve a model which depends on some config files and a custom library. The library has been included by logging the model with the `code_path` argument in `mlflow.pyfunc.log_model` and it...
- 4004 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rasgaard, one way to achieve that without inspecting the container is to use MLflow artifacts. Artifacts allow you to log files together with your models and reference them inside the endpoint.For example, let's assume that you need to include a ...
- 0 kudos
- 2167 Views
- 1 replies
- 0 kudos
Serializing custom SparkMLlib Evaluator
Hi guys,We're facing a weird behavior or we're missing some configuration in our code. I've tried to find some information unsuccessfully. Let me try to explain our case, we have implemented a custom Evaluator in python using PySpark API, something l...
- 2167 Views
- 1 replies
- 0 kudos
- 1829 Views
- 1 replies
- 0 kudos
Authentication model serving endpoint
Hi, I was wondering whether model serving endpoints support authentication with Azure Managed Identities.
- 1829 Views
- 1 replies
- 0 kudos
- 0 kudos
@larsr Databricks itself supports authentication through Managed Identity and Model Serving Endpoint requires bearer token, so yeah - i suppose it's doable.
- 0 kudos
- 4818 Views
- 3 replies
- 1 kudos
Unable to deploy phi-3 model due to packaging library
I am trying to deploy phi-3 model in databricks but getting below error while creating serving endpoint. Help us on this as soon as possible.
- 4818 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello, I'm facing the same issue. No matter what I am trying, I end up with dependencies issues...
- 1 kudos
- 2537 Views
- 2 replies
- 0 kudos
Logging signature slows down inference to a crawl
I am having a similar issue thislog signature and input data for Spark LinearRegression using mlflow v2.13.0 and using mlflow.pyfunc.log_model to log my model. Starting a new post here since there doesn't seem to be any follow up from the community o...
- 2537 Views
- 2 replies
- 0 kudos
- 0 kudos
@Miki can you please share you code for logging the signature with array types
- 0 kudos
- 5518 Views
- 0 replies
- 0 kudos
computer vision
how does data bricks handle. computer vision related use cases? (eg defects detection for a manufacturing industry) is there a reference architecture
- 5518 Views
- 0 replies
- 0 kudos
- 2463 Views
- 4 replies
- 1 kudos
Data AI Summit 2024
My first Data + AI summit and it's been a great experience
- 2463 Views
- 4 replies
- 1 kudos
- 3839 Views
- 4 replies
- 0 kudos
Sharing Output between different tasks for MLOps pipeline as a Databricks Jobs
Hello EveryoneWe are trying to create an ML pipeline on Databricks using the famous Databricks workflows. Currently our pipeline includes having 3 major components: Data Ingestion, Model Training and Model Testing. My question is whether it is possib...
- 3839 Views
- 4 replies
- 0 kudos
- 2536 Views
- 0 replies
- 0 kudos
Register Model mounted in S3
Hello!I'm having an issue registering a model saved in a mounted S3 bucket using mlflow.Let me give a little bit more context:1. First I mounted my S3 with all the corresponding IAM permissions:s3_bucket_name = f"s3a://{s3_bucket}"dbutils.fs.mount(so...
- 2536 Views
- 0 replies
- 0 kudos
- 4682 Views
- 3 replies
- 0 kudos
Resolved! Llm
Are LLMs really ready for production deployment?
- 4682 Views
- 3 replies
- 0 kudos
- 0 kudos
You should be careful while putting them to production without guardrails, perhaps using Mosaic AI gateway announced today that would aggregate these functionalities, it should be something to start. These are not the only things you should worry abo...
- 0 kudos
- 7014 Views
- 4 replies
- 4 kudos
Generate and export dbt documentation from the Workflow dbt task to S3
I'm testing the Databricks Jobs feature with a dbt task and wanted to know if you had any advice for me for managing dbt documentation.I can use "dbt run" commands to run my models then "dbt docs generate" to generate the documentation. But is it pos...
- 7014 Views
- 4 replies
- 4 kudos
- 4 kudos
How can I access these target files from the task itself ? I am trying to use dbt's state modifiers for detecting models that changed and only running models when the source freshness changed. Is there an easy way to store and use these state files i...
- 4 kudos
