Thanks to everyone who joined the Best Practices for Your Data Architecture session on Optimizing Data Performance. You can access the on-demand session recording here and the pre-run performance benchmarks using the Spark UI Simulator. Proper cluste...
Hi, I am new in Databricks.
We have trined the model in Databricks and Deployed in SageMaker. After deployment, We set the baseline for the model and enable model monitoring. After enabling the data capture for the SageMaker endpoint, we receive the following error when we do t...
I have created a key in Azure Key Vault to store my secrets in it. In order to use it securely in Azure DataBricks, have created the secret scope and configured the Azure Key Vault properties. Out of curiosity, just wanted to check whether my key is ...
mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:https://spa...
Ready to get hands-on? Explore the collaborative notebook environment: This gallery showcases some of the possibilities through Notebooks focused on technologies and use cases which can easily be imported into your own Databricks environment or the f...
2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (Post 2 of 2)Thank you to everyone who joined! You can access the on-demand recording here and the code in this Github repo.We're sharing a subset of the questions asked an...
2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (post 1 of 2)Thank you to everyone who joined the Automating the ML Lifecycle With Databricks Machine Learning webinar! You can access the on-demand recording here and the ...
The run has a model logged to it already, but the "Status" field says "UNFINISHED."
The MLflow run was probably created either (a) via notebook autologging or (b) via a call to `mlflow.start_run()`. With (a), when the notebook first logs something to MLflow, it starts a run. But if the notebook is still active and attached to a clu...
Hi Folks, I'm evaluating Delta Lake to store image / data version control to be used to train models. I looked at a session explaining how to do this and also using MLflow to manage training (https://databricks.com/session_na21/image-processing-on-d...
I can think of 3 ways for doing this:using the web UI (the create table option or upload data into DBFS)using databricks-connect, which bridges your local machine with the remote databricks clustersusing the databricks-cli to copy local files to dbfs...
Please seehttps://docs.databricks.com/dev-tools/api/latest/scim/scim-groups.htmlAn examplecurl -X PATCH -n 'https://test-fw-us-1.cloud.databricks.com/api/2.0/preview/scim/v2/Groups/4062877510650442' -d \ "{ \"schemas\":[\"urn:ietf:params:scim:api:mes...
Without knowing all that you are trying to do, the answer is yes, with the Instance Profile API. https://docs.databricks.com/dev-tools/api/latest/instance-profiles.html. You might also check out the SCIM APIs to associate the Instance Profile to a g...
Have a look at https://databricks.com/product/managed-mlflow
I updated my cluster runtime to use 8.4 ML, I did a !pip list and saw that tensorflow was 2.3. What should I do to resolve this? Not sure if I should just do a pip install Tensorflow==2.5.0
If the answer is the latter, do i need to install the supported versions of cudnn and cuda?
I think I need a little more context here on what you're trying to achieve. If you're generally interested in schema evolution, this post talks about feature store: https://databricks.com/blog/2021/05/27/databricks-announces-the-first-feature-store-i...
I'd like to use Deep Learning on Spark within AWS EMR.Is there a best practice or 'recommended' DL framework to run on Spark? It looks like Databricks' spark-deep-learning has been replaced by Horovod—should this the first option to consider? If th...
