Databricks Community

2021-07-Webinar--Hassle-Free-Data-Ingestion-Social-1200x628

Thanks to everyone who joined the Hassle-Free Data Ingestion webinar. You can access the on-demand recording here. We're sharing a subset of the phenomenal questions asked and answered throughout the session. You'll find Ingestion Q&A listed first, f...

2021-08-Best-Practices-for-Your-Data-Architecture-v3-OG-1200x628

Thanks to everyone who joined the Best Practices for Your Data Architecture session on Optimizing Data Performance. You can access the on-demand session recording here and the pre-run performance benchmarks using the Spark UI Simulator. Proper cluste...

Hi, I am new in Databricks.

Hi, I am new in Databricks.

Model monitoring issues for Databricks Trainied Model and deployed in Sagemaker

We have trined the model in Databricks and Deployed in SageMaker. After deployment, We set the baseline for the model and enable model monitoring. After enabling the data capture for the SageMaker endpoint, we receive the following error when we do t...

I have created a key in Azure Key Vault to store my secrets in it. In order to use it securely in Azure DataBricks, have created the secret scope and ...

I have created a key in Azure Key Vault to store my secrets in it. In order to use it securely in Azure DataBricks, have created the secret scope and configured the Azure Key Vault properties. Out of curiosity, just wanted to check whether my key is ...

spark.apache.org

mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:https://spa...

Ready to get hands-on? Explore the collaborative notebook environment: This gallery showcases some of the possibilities through Notebooks focused on ...

Ready to get hands-on? Explore the collaborative notebook environment: This gallery showcases some of the possibilities through Notebooks focused on technologies and use cases which can easily be imported into your own Databricks environment or the f...

mlflow.org

2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (Post 2 of 2)Thank you to everyone who joined! You can access the on-demand recording here and the code in this Github repo.We're sharing a subset of the questions asked an...

docs.databricks.com

2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (post 1 of 2)Thank you to everyone who joined the Automating the ML Lifecycle With Databricks Machine Learning webinar! You can access the on-demand recording here and the ...

What does it mean if an MLflow run is "UNFINISHED?"

The run has a model logged to it already, but the "Status" field says "UNFINISHED."

Delta Lake as source of images to train a classification model on a local computer

Hi Folks, I'm evaluating Delta Lake to store image / data version control to be used to train models. I looked at a session explaining how to do this and also using MLflow to manage training (https://databricks.com/session_na21/image-processing-on-d...

Is there a way to add an instance profile to a Databricks group using REST API?

Please seehttps://docs.databricks.com/dev-tools/api/latest/scim/scim-groups.htmlAn examplecurl -X PATCH -n 'https://test-fw-us-1.cloud.databricks.com/api/2.0/preview/scim/v2/Groups/4062877510650442' -d \ "{ \"schemas\":[\"urn:ietf:params:scim:api:mes...

Where can I get documentation around managed MLFlow? What is the difference between open source and managed MLFlow?

Runtime 8.4 ML still showing TF 2.3

I updated my cluster runtime to use 8.4 ML, I did a !pip list and saw that tensorflow was 2.3. What should I do to resolve this? Not sure if I should just do a pip install Tensorflow==2.5.0

How would the schema evolution of a feature work in Databricks Feature Store?