cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

damselfly20
by New Contributor III
  • 532 Views
  • 2 replies
  • 1 kudos

Endpoint creation without scale-to-zero

Hi, I've got a question about deploying an endpoint for Llama 3.1 8b. The following code should create the endpoint without scale-to-zero. The endpoint is being created, but with scale-to-zero, although scale_to_zero_enabled is set to False. Instead ...

  • 532 Views
  • 2 replies
  • 1 kudos
Latest Reply
damselfly20
New Contributor III
  • 1 kudos

Thanks for the reply @Walter_C. This didn't quite work, since it used a CPU and didn't consider the max_provisioned_throughput, but I finally got it to work like this: from mlflow.deployments import get_deploy_client client = get_deploy_client("data...

  • 1 kudos
1 More Replies
NielsMH
by New Contributor III
  • 493 Views
  • 1 replies
  • 0 kudos

spark_session invocation from executor side error, when using sparkXGBregressor and fe client

Hi I have created a model and pipeline using xgboost.spark's sparkXGBregressor and pyspark.ml's Pipeline instance. However, i run into a "RuntimeError: _get_spark_session should not be invoked from executor side." when i try to save the predictions i...

  • 493 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

The error you're encountering is due to attempting to access the Spark session on the executor side, which is not allowed in Spark's distributed computing model. This typically happens when trying to use Spark-specific functionality within a UDF or d...

  • 0 kudos
cmilligan
by Contributor II
  • 6270 Views
  • 5 replies
  • 2 kudos

Issue with Multi-column In predicates are not supported in the DELETE condition.

I'm trying to delete rows from a table with the same date or id as records in another table. I'm using the below query and get the error 'Multi-column In predicates are not supported in the DELETE condition'. delete from cost_model.cm_dispatch_consol...

  • 6270 Views
  • 5 replies
  • 2 kudos
Latest Reply
thisisthemurph
New Contributor II
  • 2 kudos

I seem to get this error on some DeltaTables and not others:df.createOrReplaceTempView("channels_to_delete") spark.sql(""" delete from lake.something.earnings where TenantId = :tenantId and ChannelId = in ( select ChannelId ...

  • 2 kudos
4 More Replies
amirA
by New Contributor II
  • 1538 Views
  • 3 replies
  • 1 kudos

Resolved! Extracting Topics From Text Data Using PySpark

Hi EveryoneI tried to follow the same steps in Topic from Text on similar data as example. However, when I tri to fit the model with data I get this error.IllegalArgumentException: requirement failed: Column features must be of type equal to one of t...

  • 1538 Views
  • 3 replies
  • 1 kudos
Latest Reply
filipniziol
Contributor III
  • 1 kudos

Hi @amirA ,The LDA model expects the features column to be of type Vector from the pyspark.ml.linalg module, specifically either a SparseVector or DenseVector, whereas you have provided Row type.You need to convert your Row object to SparseVector.Che...

  • 1 kudos
2 More Replies
ukaplan
by New Contributor III
  • 3423 Views
  • 11 replies
  • 2 kudos

Serving Endpoint Container Image Creation Fails

Hello, I trained a model using MLFlow, and saved the model as an artifact. I can load the model from a notebook and it works as expected (i.e. I can load the model using its URI).However, when I want to deploy it using Databricks endpoints, container...

  • 3423 Views
  • 11 replies
  • 2 kudos
Latest Reply
damselfly20
New Contributor III
  • 2 kudos

@ivan_calvo The problem still exists. Surely there has to be some other option than downgrading the ML cluster to DBR 14.3 LTS ML?

  • 2 kudos
10 More Replies
Swappatil2506
by New Contributor II
  • 542 Views
  • 2 replies
  • 0 kudos

I want to develop an automated lead allocation system to prospect sales representatives.

I want to develop an automated lead allocation system to prospect sales representatives. Please suggest a suitable solution also any links if available.

  • 542 Views
  • 2 replies
  • 0 kudos
Latest Reply
Swappatil2506
New Contributor II
  • 0 kudos

Hi jamesl,My use case is related to match the prospect sales agent for the customer entering retail store, when a customer enters a store based on the inputs provided and checking on if the customer is existing or new customer, I want to create a rea...

  • 0 kudos
1 More Replies
hawa
by New Contributor II
  • 558 Views
  • 1 replies
  • 0 kudos

Problem serving a langchain model on Databricks

Hi, I've encountered a problem of serving a langchain model I just created successfully on Databricks.I was using the following code to set up a model in unity catalog:from mlflow.models import infer_signatureimport mlflowimport langchainmlflow.set_r...

  • 558 Views
  • 1 replies
  • 0 kudos
Latest Reply
hawa
New Contributor II
  • 0 kudos

I suspected the issue is coming from this small error I got: Got error: Must specify a chain Type in config. I used the chain_type="stuff" when building the langchain but I'm not sure how to fix it.

  • 0 kudos
javeed
by New Contributor
  • 349 Views
  • 0 replies
  • 0 kudos

Convert the tensorflow datatset to numpy tuples

Hello everyone ,Here are the sequence of steps i have followed:1. I have used petastorm to convert the spark dataframe to tf.datasetimport numpy as np# Read the Petastorm dataset and convert it to TensorFlow Datasetwith converter.make_tf_dataset() as...

  • 349 Views
  • 0 replies
  • 0 kudos
avishkarborkar
by New Contributor III
  • 2570 Views
  • 6 replies
  • 4 kudos
  • 2570 Views
  • 6 replies
  • 4 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 4 kudos

There could be multiple reasone why you're getting this error @avishkarborkar . If the course you're following requires Unity Catalog, first you need to check if you have a premium workspace. Next you need to make sure that your workspace is enabled ...

  • 4 kudos
5 More Replies
Mikkel
by New Contributor
  • 419 Views
  • 1 replies
  • 0 kudos

Unable to Check Experiment Existence with path starting with /Workspace/ Directory in Databricks Pla

https://github.com/mlflow/mlflow/issues/11077  In Databricks, when attempting to set an experiment with an experiment_name specified as an absolute path from /Workspace/Shared/mlflow_experiment/<experiment_name>, the mlflow.set_experiment() function ...

  • 419 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Before setting the experiment, use mlflow.get_experiment_by_name() to check if the experiment already exists. If it does, you can set the experiment without attempting to create it again.

  • 0 kudos
sergiopolimante
by New Contributor II
  • 368 Views
  • 1 replies
  • 0 kudos

What is the best to way to not deploy/run a workflow in production?

I am building and MLOps architecture.I do not want to deploy the training workflow to prod. My first approach was to selectively not deploy the workflow to prod, but this does not seem to be possible as in this thread:https://community.databricks.com...

  • 368 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

  Target Override Feature: You can use the target override feature to specify different configurations for different environments. However, this does not provide a direct way to exclude specific job resources.Environment-Specific Folders: Another app...

  • 0 kudos
DaPo
by New Contributor II
  • 1079 Views
  • 1 replies
  • 1 kudos

Resolved! Using Datbricks Connect with serverless compute and MLflow

Hi all,I have been using databricks-connect with serverless compute to develop and debug my databricks related code. It worked great so far. Now I started integrating ML-Flow in my workflow, and I am encountering an issue. When I run the following co...

  • 1079 Views
  • 1 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

The error you are encountering, pyspark.errors.exceptions.connect.AnalysisException: [CONFIG_NOT_AVAILABLE] Configuration spark.mlflow.modelRegistryUri is not available. SQLSTATE: 42K0I, is a known issue when using MLflow with serverless clusters in ...

  • 1 kudos
rtreves
by Contributor
  • 3982 Views
  • 8 replies
  • 0 kudos

One-hot encoding of strong cardinality features failing, causes downstream issues

Hi Databricks support,I'm training an ML model using mlflow on DBR 13.3 LTS ML, Spark 3.4.1 using databricks.automl_runtime 0.2.17 and databricks.automl 1.20.3, with shap 0.45.1. My training data has two float-type columns with three or fewer unique ...

  • 3982 Views
  • 8 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi @rtreves , sorry I was not able to investigate on the above. Not sure if you would be able to create a support ticket with Databricks as it may be an involved effort to review the code.  I do have a suggestion, instead of relying on the automatic ...

  • 0 kudos
7 More Replies
M_B
by New Contributor II
  • 412 Views
  • 1 replies
  • 1 kudos

Resolved! Serving model with custom scoring script to a real-time endpoint

Hi, new to databricks here and wasn't able to find relevant info in the documentation.Is it not possible to serve a model with a custom scoring script to an online endpoint on databricks to customise inference ? the customisation is related to incomi...

  • 412 Views
  • 1 replies
  • 1 kudos
Latest Reply
HaggMan
New Contributor III
  • 1 kudos

If I'm understanding, all you really want to do is have a pre/post - process function running with your model, is that correct? If so, you can do this by using the MLflow pyfunc model. Something  like they do here:https://docs.databricks.com/en/machi...

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels