Ready to get hands-on? Explore the collaborative notebook environment: This gallery showcases some of the possibilities through Notebooks focused on technologies and use cases which can easily be imported into your own Databricks environment or the f...
2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (Post 2 of 2)Thank you to everyone who joined! You can access the on-demand recording here and the code in this Github repo.We're sharing a subset of the questions asked an...
2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (post 1 of 2)Thank you to everyone who joined the Automating the ML Lifecycle With Databricks Machine Learning webinar! You can access the on-demand recording here and the ...
The run has a model logged to it already, but the "Status" field says "UNFINISHED."
The MLflow run was probably created either (a) via notebook autologging or (b) via a call to `mlflow.start_run()`. With (a), when the notebook first logs something to MLflow, it starts a run. But if the notebook is still active and attached to a clu...
Hi Folks, I'm evaluating Delta Lake to store image / data version control to be used to train models. I looked at a session explaining how to do this and also using MLflow to manage training (https://databricks.com/session_na21/image-processing-on-d...
I can think of 3 ways for doing this:using the web UI (the create table option or upload data into DBFS)using databricks-connect, which bridges your local machine with the remote databricks clustersusing the databricks-cli to copy local files to dbfs...
Please seehttps://docs.databricks.com/dev-tools/api/latest/scim/scim-groups.htmlAn examplecurl -X PATCH -n 'https://test-fw-us-1.cloud.databricks.com/api/2.0/preview/scim/v2/Groups/4062877510650442' -d \ "{ \"schemas\":[\"urn:ietf:params:scim:api:mes...
Without knowing all that you are trying to do, the answer is yes, with the Instance Profile API. https://docs.databricks.com/dev-tools/api/latest/instance-profiles.html. You might also check out the SCIM APIs to associate the Instance Profile to a g...
Have a look at https://databricks.com/product/managed-mlflow
I updated my cluster runtime to use 8.4 ML, I did a !pip list and saw that tensorflow was 2.3. What should I do to resolve this? Not sure if I should just do a pip install Tensorflow==2.5.0
If the answer is the latter, do i need to install the supported versions of cudnn and cuda?
I think I need a little more context here on what you're trying to achieve. If you're generally interested in schema evolution, this post talks about feature store: https://databricks.com/blog/2021/05/27/databricks-announces-the-first-feature-store-i...
I'd like to use Deep Learning on Spark within AWS EMR.Is there a best practice or 'recommended' DL framework to run on Spark? It looks like Databricks' spark-deep-learning has been replaced by Horovod—should this the first option to consider? If th...
I am not aware of any special requirement for this migration, my suggestion to you is to try it on a small scale (one notebook) and observe the results showing in the tracker server, if everything looks OK, then migrate the rest.
If you have configured your Structured Streaming query to use RocksDB as the state store, you can now get better visibility into the performance of RocksDB, with detailed metrics on get/put latencies, compaction latencies, cache hits, and so on. Thes...
Advantage of using Photon EngineThe following summarizes the advantages of Photon:Supports SQL and equivalent DataFrame operations against Delta and Parquet tables.Expected to accelerate queries that process a significant amount of data (100GB+) and ...
check if your workspace has the IP access list feature enabled, call the get feature status API (GET /workspace-conf). Pass keys=enableIpAccessLists as arguments to the request.In the response, the enableIpAccessListsthe field specifies either true o...
Wondering about best practices for how to handle collaboration between multiple ML practitioners working on a single experiment. Do we have to share the same notebook between people or is it possible to have individual notebooks going but still work ...
Yes, multiple users could work on individual notebooks and still use the same experiment via mlflow.set_experiment(). You could also assign different permission levels to experiments from a governance point of view
