Using Ganglia you can monitor how busy is the GPU(s). Increasing the batch size would increase that utilization. Bigger batches improve how well each batch updates the model (up to a point) with more accurate gradients. That in turn can allow traini...
It’s advantageous to stop running trials if progress has stopped. Hyperopt offers an early_stop_fn parameter, which specifies a function that decides when to stop trials before max_evals has been reached. Hyperopt provides a function no_progress_loss...
DBR minor version details are not exposed. However, in the documentation, it mentioned Databricks performs maintenance releases every 2 weeks. How can I determine if I am using the same minor version
The below code snippet can help to determine the DBR Hash string for the DBR version. DBR hash string is unique for the DBR minor version. val scalaVersion = scala.util.Properties.versionString val hadoopVersion = org.apache.hadoop.util.VersionInf...
Delete workspace doesn't delete the root bucket. You could choose to use the same root bucket for more than one workspace ( though not recommended ) It is recommended to automate the infrastructure creation via terraform or quickstart so that cleanup...
Are there any event streams that are or could be exposed in AWS (such as Cloudwatch Eventbridge events or SNS messages? In particular I'm interested in events that detail jobs being run. The use case here would be for monitoring jobs from our web app...
You could write code to call the PutLogEvents api at the beginning of each job to write out custom events to cloudwatch / or use aws sdk to send and SNS notification and route it to a desired consumer.
Are Repos HIPAA compliant, or is there a plan and timeline to support this? Customer is getting a warning when trying to enable the Repos feature in a HIPAA deployment on Azure Databricks.
There is a plan to support this. For timeline, please reach out to your Databricks account team.
Unfortunately this is not possible. The default user workspace name will be the user's email address.
I am working on a disaster recovery plan for my environment which includes Databricks. Where do I start with my planning? What all do I need to consider when building a DR plan?
Depending on your RPO/RTOs there are different recovery solution strategies that could be considered (active/passive, active/active) for Databricks deployments. A detailed explanation of these approaches are mentioned here
I would like it if I could make it so that the credentials of the user who initiates a job are used as the credentials for the job run. Is this possible?
Is this in Azure? If so, it is not supported currently. https://docs.microsoft.com/en-us/azure/databricks/security/credential-passthrough/adls-passthrough
Yes. We support this.Please see https://docs.databricks.com/administration-guide/workspace/storage.html#modify-the-storage-location-for-notebook-results and https://docs.databricks.com/administration-guide/workspace/storage.html#configure-the-storage...
All Databricks workspaces are deployed with this setting as of Q1 of 2021.
Yes. Users can have access to multiple workspaces. Workspaces are units of collaboration.
Databricks only charges for compute time while machines are being used. If a machine is on "IDLE" then Databricks does not charge you for those machines. The cloud providers will charge you for the machines that are running regardless if they are IDL...
Yes, you need an Azure Databricks account on the Premium plan to enable Databricks SQL
Yes. It is not self-service. We can "merge" accounts on AWS, such that you can manage all your Databricks workspaces from a single place at https://accounts.cloud.databricks.com/loginPlease contact your Databricks Representative.
