cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
BeardyMan
by • New Contributor III
  • 6053 Views
  • 9 replies
  • 3 kudos

Resolved! MLFlow Serve Logging

When using Azure Databricks and serving a model, we have received requests to capture additional logging. In some instances, they would like to capture input and output or even some of the steps from a pipeline. Is there any way we can extend the lo...

Data Engineering
Reply
  • 6053 Views
  • 9 replies
  • 3 kudos
View Replies
Latest Reply
Dan_Z
Databricks Employee
  • 3 kudos

Another word from a Databricks employee:"""You can use the custom model approach but configuring it is painful. Plus you have ended every loggable model in the custom model. Another less intrusive solution would be to have a proxy server do the loggi...

  • 3 kudos
8 More Replies
mrcity
by • New Contributor II
  • 2342 Views
  • 3 replies
  • 1 kudos

Exclude absent lookup keys from dataframes made by create_training_set()

I've got data stored in feature tables, plus in a data lake. The feature tables are expected to lag the data lake by at least a little bit. I want to filter data coming out of the feature store by querying the data lake for lookup keys out of my inde...

Data Engineering
Reply
  • 2342 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Quinten
New Contributor II
  • 1 kudos

I'm facing the same issue as described by @mrcity. There is no easy way to alter the dataframe, which is created inside the score_batch() function. Filtering out rows in the (sklearn) pipeline itself is also not convenient since these transformers ar...

  • 1 kudos
2 More Replies
Yoni
by • New Contributor
  • 14943 Views
  • 5 replies
  • 3 kudos

Resolved! MLFlow failed: You haven't configured the CLI yet

I'm getting an errorYou haven’t configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launcher.py configure`My cluster is running Databricks Runtime Version 10.1I've also installed mlflow to the cluster l...

Data Engineering
Reply
  • 14943 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
HemantKumar
New Contributor II
  • 3 kudos

dbutils.library.restartPython()Add that after you run the pip install mlflow, it worked for me in a non-ML cluster

  • 3 kudos
4 More Replies
Nikhil3107
by • New Contributor III
  • 2013 Views
  • 1 replies
  • 2 kudos

Deploy model to AWS Sagemaker: ModuleNotFoundError: No module named 'docker'

Greetings, When trying to run the following command: %sh mlflow sagemaker build-and-push-containerI get the following error:/databricks/python3/lib/python3.9/site-packages/click/core.py:2309: UserWarning: Virtualenv support is still experimental and ...

Data Engineering
Reply
  • 2013 Views
  • 1 replies
  • 2 kudos
View Replies
SagarK1
by • New Contributor
  • 4391 Views
  • 4 replies
  • 2 kudos

Managing the permissions using MLFlow APIs

Hello All,I am trying to manage the permissions on the experiments using the MLFLow API. Do we have any MLFlow API which helps to manage the permissions of Can Read ,Can Edit , Can Manage.Example :I create the model using MLFlow APIs and through my c...

Data Engineering
Reply
  • 4391 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
jsan
New Contributor II
  • 2 kudos

Hey folks, did we get any workaround for this or what @Sean Owen​ said is true ?

  • 2 kudos
3 More Replies
Taha_Hussain
by • Databricks Employee
  • 2242 Views
  • 2 replies
  • 6 kudos

Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMT Databricks Office Hours connects you directly with exper...

Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTDatabricks Office Hours connects you directly with experts to answer your Databricks questions.Join us to:• Troubleshoot your technical questions• Learn the ...

Data Engineering
Reply
  • 2242 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
Taha_Hussain
Databricks Employee
  • 6 kudos

Cont...Q: Do generated columns in Delta Live Tables include IDENTITY columns?A: My understanding is that generated columns in Delta Live Tables do not contain IDENTITY columns. Here is more on generated columns in DLT.Q: We store raw data for each cu...

  • 6 kudos
1 More Replies
Christine
by • Contributor II
  • 8754 Views
  • 7 replies
  • 5 kudos

Resolved! autoML' is not found when using databricks.automl with runtime 112.ML (and runtime 10.4 LTS ML).

I have tried to set up a autoML experiment with runtime 11.2ML and data from a delta table. However I receive the error "ModuleNotFoundError: No module named 'databricks.automl'" and "AutoML not available: Use Databricks Runtime 8.3 ML or above." tho...

image
Data Engineering
Reply
  • 8754 Views
  • 7 replies
  • 5 kudos
View Replies
Latest Reply
Christine
Contributor II
  • 5 kudos

I deleted the cluster and created a new with runtime 9.1 LTS ML which solved the problem.

  • 5 kudos
6 More Replies
Taha_Hussain
by • Databricks Employee
  • 1574 Views
  • 1 replies
  • 3 kudos

Register for Databricks Office HoursSeptember 14: 8:00 - 9:00 AM PT | 3:00pm - 4:00pm GMTSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMT Dat...

Register for Databricks Office HoursSeptember 14: 8:00 - 9:00 AM PT | 3:00pm - 4:00pm GMTSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTDatabricks Office Hours connects you directly with experts to answer your Databricks questions.Join us t...

Data Engineering
Reply
  • 1574 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Taha_Hussain
Databricks Employee
  • 3 kudos

Check out some of the questions from fellow users during our last Office Hours. All these questions were answered live by a Databricks expert!Q: What's the best way of using a UDF in a class?A: You need to define your class and then register the func...

  • 3 kudos
Taha_Hussain
by • Databricks Employee
  • 1198 Views
  • 0 replies
  • 3 kudos

Register for Databricks Office HoursAugust 17 & August 31 from 8:00am - 9:00am PT | 3:00pm - 4:00pm GMT. Databricks Office Hours connects you dire...

Register for Databricks Office HoursAugust 17 & August 31 from 8:00am - 9:00am PT | 3:00pm - 4:00pm GMT.Databricks Office Hours connects you directly with experts to answer your Databricks questions.Join us to: • Troubleshoot your technical questions...

Data Engineering
Reply
  • 1198 Views
  • 0 replies
  • 3 kudos
Taha_Hussain
by • Databricks Employee
  • 1170 Views
  • 0 replies
  • 8 kudos

Databricks Office Hours Register for Office Hours to participate in a LIVE Q&A session and receive technical support directly from Databricks expe...

Databricks Office HoursRegister for Office Hours to participate in a LIVE Q&A session and receive technical support directly from Databricks experts! Our next event is scheduled for July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GMT.Whether you ...

Data Engineering
Reply
  • 1170 Views
  • 0 replies
  • 8 kudos
Taha_Hussain
by • Databricks Employee
  • 875 Views
  • 0 replies
  • 5 kudos

Databricks Office Hours Register for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks expe...

Databricks Office HoursRegister for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks experts! Our next events are scheduled for July 13th & July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GM...

Data Engineering
Reply
  • 875 Views
  • 0 replies
  • 5 kudos
harish_s
by • New Contributor II
  • 5368 Views
  • 3 replies
  • 4 kudos

Resolved! Hi, I get the following error when I enable model serving for spacy model via MLFLOW.

+ echo 'GUNICORN_CMD_ARGS=--timeout 63 --workers 4 'GUNICORN_CMD_ARGS=--timeout 63 --workers 4 + mlflow models serve --no-conda -m /tmp/tmp1a4ltdrk/spacymodelv1 -h unix:/tmp/3.sock -p12022/03/01 08:26:37 INFO mlflow.models.cli: Selected backend for f...

Data Engineering
Reply
  • 5368 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 4 kudos

Hi @Harish S​ this error could happen if the backend services are not updated. Are you doing this test in a PVC environment or a standard workspace?

  • 4 kudos
2 More Replies
Maverick1
by • Valued Contributor II
  • 3227 Views
  • 3 replies
  • 6 kudos

How to deploy mlflow models to sagemaker endpoints where sagemaker refers the private docker registry?

Is it possible to deploy the mlflow model to a sagemaker endpoint where the image URL is not referring to an image in ECR but the image is actually present in a private docker registry?

Data Engineering
Reply
  • 3227 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 6 kudos

@Saurabh Verma​ , this to create the endpoint.also, check this out - https://github.com/mlflow/mlflow/blob/0fa849ad75e5733bf76cc14a4455657c5c32f107/mlflow/sagemaker/__init__.py#L361

  • 6 kudos
2 More Replies
Krishscientist
by • New Contributor III
  • 1156 Views
  • 1 replies
  • 0 kudos

Resolved! AutoML : data set for problem type "Classification"

HI,I am working on AutoML Experiment. Could you plz help me with data set for problem type "Classification"Regards.

Data Engineering
Reply
  • 1156 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

There are a lot of datasets available in /databricks-datasets/ that you can look through. You'll have to turn them into a table so that you can access them in automl. There are datasets associated with the spark definitive guide and learning spark ...

  • 0 kudos
gibbona1
by • New Contributor II
  • 4227 Views
  • 2 replies
  • 1 kudos

Resolved! Correct setup and format for calling REST API for image classification

I trained a basic image classification model on MNIST using Tensorflow, logging the experiment run with MLflow.Model: "my_sequential" _________________________________________________________________ Layer (type) Output Shape ...

mnist_model_error
Data Engineering
Reply
  • 4227 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 1 kudos

@Anthony Gibbons​  may be this git should work with your use case - https://github.com/mlflow/mlflow/issues/1661

  • 1 kudos
1 More Replies
Labels