https://community.databricks.com/t5/get-started-discussions/databricks-setup-deployment-checklist-best-practices/m-p/54099#M1885 <P>The Databricks platform is on azure</P> Tue, 28 Nov 2023 12:37:01 GMT Phani1 2023-11-28T12:37:01Z https://community.databricks.com/t5/get-started-discussions/databricks-setup-deployment-checklist-best-practices/m-p/54098#M1884 <P><SPAN><SPAN class="">Hi Team, could you please share or guide us on any checklist/best practices for Databricks setup/deployment?</SPAN></SPAN></P> Tue, 28 Nov 2023 12:35:06 GMT https://community.databricks.com/t5/get-started-discussions/databricks-setup-deployment-checklist-best-practices/m-p/54098#M1884 Phani1 2023-11-28T12:35:06Z https://community.databricks.com/t5/get-started-discussions/databricks-setup-deployment-checklist-best-practices/m-p/54099#M1885 <P>The Databricks platform is on azure</P> Tue, 28 Nov 2023 12:37:01 GMT https://community.databricks.com/t5/get-started-discussions/databricks-setup-deployment-checklist-best-practices/m-p/54099#M1885 Phani1 2023-11-28T12:37:01Z https://community.databricks.com/t5/get-started-discussions/databricks-setup-deployment-checklist-best-practices/m-p/54121#M1886 <P>Hi <a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/36892">@Phani1</a>&nbsp;, here are some best practices <A href="https://github.com/Azure/AzureDatabricksBestPractices/tree/master" target="_blank">https://github.com/Azure/AzureDatabricksBestPractices/tree/master</A> and you could take these points as your "checklist".</P><OL><LI><P><STRONG>Choose the right Databricks Workspace:</STRONG></P><UL><LI>Decide on the appropriate Azure region for your Databricks workspace.</LI><LI>Consider using the Azure Portal to create the Databricks workspace or use infrastructure-as-code tools like ARM templates or Terraform.</LI></UL></LI><LI><P><STRONG>Authentication and Authorization:</STRONG></P><UL><LI>Integrate Databricks with Azure Active Directory (Azure AD) for authentication.</LI><LI>Implement role-based access control (RBAC) to manage authorization.</LI></UL></LI><LI><P><STRONG>Networking and Security:</STRONG></P><UL><LI>Configure Azure Virtual Network (VNet) settings for Databricks clusters.</LI><LI>Utilize Azure Network Security Groups (NSGs) for firewall rules.</LI><LI>Consider Private Link for enhanced security.</LI></UL></LI><LI><P><STRONG>Use Azure Key Vault for Secrets:</STRONG></P><UL><LI>Store sensitive information such as API keys, passwords, and tokens in Azure Key Vault.</LI><LI>Integrate Databricks with Azure Key Vault for secure access to secrets.</LI></UL></LI><LI><P><STRONG>Cluster Configuration:</STRONG></P><UL><LI>Leverage Azure Databricks Autoscaling for dynamic resource allocation.</LI><LI>Integrate Databricks clusters with Azure Virtual Network for enhanced security.</LI></UL></LI><LI><P><STRONG>Data Storage and Integration:</STRONG></P><UL><LI>Use Azure Data Lake Storage (ADLS) or Azure Blob Storage for data storage.</LI></UL></LI><LI><P><STRONG>Logging and Monitoring:</STRONG></P><UL><LI>Configure Azure Monitor for logging and monitoring.</LI><LI>Utilize Azure Log Analytics for centralized log storage and analysis.</LI></UL></LI><LI><P><STRONG>Azure Databricks Jobs:</STRONG></P><UL><LI>Schedule jobs using Azure Databricks Jobs for automated execution.</LI><LI>Use Azure Data Factory for orchestrating ETL workflows if needed.</LI></UL></LI><LI><P><STRONG>Azure Databricks Delta Lake:</STRONG></P><UL><LI>Consider using Delta Lake for efficient storage, management, and processing of big data.</LI><LI>Utilize Delta Lake for ACID transactions and schema evolution.</LI></UL></LI><LI><P><STRONG>Azure DevOps Integration:</STRONG></P><UL><LI>Integrate Databricks with Azure DevOps for continuous integration and deployment.</LI><LI>Automate deployments using Azure DevOps pipelines.</LI></UL></LI></OL> Tue, 28 Nov 2023 16:07:03 GMT https://community.databricks.com/t5/get-started-discussions/databricks-setup-deployment-checklist-best-practices/m-p/54121#M1886 icyflame92 2023-11-28T16:07:03Z