Security Controls to implement on Machine Learning Persona

[VJ3]

Hello,

Hope everyone are doing well. You may be aware that we are using Table ACL enabled cluster to ensure the adequate security controls on Databricks. You may be also aware that we can not use Table enabled ACL cluster on Machine Learning Persona. Hence, I am looking for Security best practice to follow to use Machine Learning Persona on Databricks. 

Please let me know.

[Kaniz]

Hi @VJ3, Databricks is a powerful platform that combines data engineering, machine learning, and business intelligence. When deploying Databricks in an enterprise environment, it’s crucial to establish robust security practices.

Let’s focus on best practices for using the Machine Learning Persona on Databricks:

1. Persona-Based Workspace Permissions:

   • Databricks provides workspace access control that allows you to assign different privileges to different job-role personas within a workspace. This segregation ensures compliance and minimizes risks.
   • The administrator should enable workspace access control, which is available in the Databricks Premium plan and above.
   • By mapping typical personas (such as data scientists, ML engineers, and analysts) to predefined grou...¹.

2. Network Access Control:

   • Control network access to your Databricks clusters by configuring network security groups. Limit inbound and outbound traffic to specific IP ranges or virtual networks.

3. Identity and Access Management (IAM):

   • Implement a robust IAM strategy to manage access to Databricks components.
   • Assign roles and permissions based on job roles (e.g., data scientist, ML engineer) to ensure segregation of duties.

4. Data Encryption:

   • Encrypt data at rest and in transit.
   • Utilize Azure Key Vault to securely manage encryption keys.

5. Auditing and Monitoring:

   • Enable audit log delivery to track user activities.
   • Use tools like Overwatch to monitor workspace usage and detect anomalies.

6. Secure Secrets and Passwords:

   • Avoid hardcoding secrets or passwords in notebooks or scripts.
   • Utilize Databricks’ secret management features to securely store and retrieve sensitive information.

Remember that these practices are essential for regulatory compliance, risk avoidance, and maintaining a secure environment. By following these guidelines, you can effectively use the Machine Learning Persona on Databricks wh...²³.

[VJ3]

Hello Kaniz,

Thank you for the update. If above controls are enough, why are we using Table ACLs as Security control on Non ML cluster?  The Security controls you mentioned above are generic controls. Do we have any specific security controls which can be implemented on ML clusters?