- 2066 Views
- 3 replies
- 1 kudos
Expose low latency APIs from Deltalake for mobile apps and microservices
My company is using Deltalake to extract customer insights and run batch scoring with ML models. I need to expose this data to some microservices thru gRPC and REST APIs. How to do this? I'm thinking to build Spark pipelines to extract teh data, stor...
- 2066 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @John Capplefield​ Gentle follow-up, please let us know if you need further help on this.
- 1 kudos
- 1529 Views
- 1 replies
- 2 kudos
CountVectorizer no longer works through Azure ML
Hello. I am trying to use the CountVectorizer module as part of our feature engineering. It works on a Databricks notebook directly, but when I try to run the code through Azure with the databricks connection, it throws an error. This isn't the first...
- 1529 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Danny Siu​ Please check that you are using the latest dbconnect version corresponding to the DBR version that you are using in the databricks cluster.You can check the latest dbr version here: https://pypi.org/project/databricks-connect/#history
- 2 kudos
- 2331 Views
- 2 replies
- 2 kudos
Resolved! Failure in mlflow.spark.load_model : Random Forrest pretrained model
model = mlflow.spark.load_model(model_uri=f"models:/{model_name}/{model_version}")Log:An error occurred while calling o2861.load.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 4599.0 failed 4 times, most recent f...
- 2331 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Ashraf Khan​ Did you get a chance to look into Sean's response. Please let us know if you need more help on this.
- 2 kudos
- 4010 Views
- 1 replies
- 3 kudos
Resolved! Installing pyspark.pandas
Hello guys,I'm trying to migrate a python project from Pandas to Pandas API on Spark, on Azure Databricks using MLFlow on a conda env.The thing is I'm getting the next error:Traceback (most recent call last): File "/databricks/mlflow/projects/x/data_...
- 4010 Views
- 1 replies
- 3 kudos
- 3 kudos
it should be yes.can you elaborate on how you create your notebook (and the conda env you talk about)?
- 3 kudos
- 1332 Views
- 0 replies
- 0 kudos
Is Model Serving REST API available?
This is mentioned in:https://learn.microsoft.com/en-us/azure/databricks/mlflow/create-manage-serverless-model-endpointswith api call example, while in:https://learn.microsoft.com/en-us/answers/questions/892678/how-to-enable-databricks-model-serving-w...
- 1332 Views
- 0 replies
- 0 kudos
- 8755 Views
- 5 replies
- 4 kudos
Unity Catalog - existing dbfs mounts and feature store
Hi All, We're currently considering turning on Unity Catalog but before we flick the switch I'm hoping I can get a bit more confidence of what will happen with our existing dbfs mounts and feature store. The bit that makes me nervous is the crede...
- 8755 Views
- 5 replies
- 4 kudos
- 4 kudos
@Ashley Betts​ can you please check below article, as far as i know we can use external mount points by configuring storage credentials in unity catalog . default method is managed tables, but we can point external tables also. 1. you can upgrade exi...
- 4 kudos
- 4048 Views
- 2 replies
- 0 kudos
Cannot serialize this model error when attempting MLFlow for SparkNLP
I'm attempting to use MLFlow to register models in Databricks and am following the recipe at...https://nlp.johnsnowlabs.com/docs/en/licensed_serving_spark_nlp_via_api_databricks_mlflowwhen i execute...mlflow.spark.log_model(pipeline, "lemmatizer", co...
- 4048 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Tobias Cortese​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 0 kudos
- 9780 Views
- 4 replies
- 4 kudos
How to use Parallel processing using Concurrent Jobs in Databricks ?
QuestionIt would be great if you could recommend how I go about solving the below problem. I haven't been able to find much help online. A. Background:A1. I have to text manipulation using python (like concatenation , convert to spacy doc , get verbs...
- 9780 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 4 kudos
- 2208 Views
- 2 replies
- 0 kudos
API limit on mlflow.tracking.client.MlflowClient.list_run_infos method?
I'm trying out managed MLflow on Databricks Community edition, with tracking data saved on Databricks and artifacts saved on my own AWS S3 bucket. I created one experiment and logged 768 runs in the experiment. When I try to get the list of the runs ...
- 2208 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @jae baak​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 3242 Views
- 0 replies
- 1 kudos
Responsible AI on Databricks
Looking to learn how you can use responsible AI toolkits on Databricks? Interested in learning how you can incorporate open source tools like SHAP and Fairlearn with Databricks?I would recommend checking out this blog: Mitigating Bias in Machine Lear...
- 3242 Views
- 0 replies
- 1 kudos
- 9815 Views
- 3 replies
- 1 kudos
Resolved! Shap Values for predictions from registered model
I have saved a model in the model registry using MLFlow. How can I find the shap values for this model once I have generated predictions in batch mode? Shap tree explainer does not support the mlflow pyfunc model type. When I use mlflow.shap.log_exp...
- 9815 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, Could you please check if this section helps in the below documentation:https://www.databricks.com/blog/2019/06/17/detecting-bias-with-shap.html
- 1 kudos
- 8281 Views
- 6 replies
- 1 kudos
MLflow Project run always comes back as status failed.
Hi! This is kind of an urgent question so any help would be greatly appreciated! Thanks so much! So I'm following this tutorial to try to create an MLflow project: https://docs.databricks.com/applications/mlflow/projects.htmlI tried with the example ...
- 8281 Views
- 6 replies
- 1 kudos
- 1 kudos
This is generally not how you use MLflow in Databricks. You are already in Databricks so do not need to send code to Databricks to execute. Instead just run your code in a notebook; there is no need to package as an MLflow Project. Projects are prima...
- 1 kudos
- 3242 Views
- 4 replies
- 0 kudos
What are the parameters For MLflow Project file
Hi! I was just wondering what are the parameters For MLflow Project file? I'm following this tutorial to create my own MLflow Project: https://docs.databricks.com/applications/mlflow/projects.htmland within this tutorial, the MLproject file looks lik...
- 3242 Views
- 4 replies
- 0 kudos
- 0 kudos
These parameters are parameters that you will specify when you run the MLflow Project with the mlflow CLI. It lets you parameterize your code, and then pass different parameters to it. How you use them is up to your code. These are not model hyperpar...
- 0 kudos
- 2873 Views
- 3 replies
- 3 kudos
Resolved! How to drop single feature from feature store table
I have a feature store table and I would like to change one of the features from IntegerType to FloatType, I can't merge this change as it violates the schema. Is it possible to drop a single feature from the table and add the revised feature?Current...
- 2873 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi there @_ _​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 1422 Views
- 0 replies
- 1 kudos
Substantial performance issues/degradation on Databricks when migrating job over to EMR
Versions of Code:Databricks: 7.3 LTS ML (includes Apache Spark 3.0.1, Scala 2.12)AWS EMR: 6.1.0 (Spark 3.0.0, Scala 2.12)https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-610-release.html The problem:Errors in Databricks when replicating job th...
- 1422 Views
- 0 replies
- 1 kudos
