Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks

mh2587 — Tue, 19 Nov 2024 13:47:14 GMT

Hello Databricks Community
I am currently using Azure Databricks with PCI-DSS compliance enabled in our workspace, as maintaining stringent security standards is crucial for our organization. However, I've discovered that once PCI-DSS compliance is turned on, it cannot be disabled, and this has impacted our ability to access certain features, such as serverless compute.
My Questions:

Are there any ways to adjust compliance settings or policies to enable serverless compute while still adhering to PCI-DSS standards?
Would using a separate, non-compliant workspace for specific tasks be a recommended practice, and if so, how can this be managed effectively alongside our compliant workspace?

Re: Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks

mark_ott — Wed, 19 Nov 2025 11:59:45 GMT

Once PCI-DSS compliance is enabled in Azure Databricks, the workspace is locked into a set of restrictions to maintain those standards and safeguard sensitive data. These restrictions include disabling access to features like serverless compute, which do not currently meet the required PCI-DSS controls for data security and isolation. There is currently no supported way to adjust compliance settings within that workspace to selectively enable serverless compute while keeping compliance active—this is by design, to ensure strict adherence to PCI-DSS requirements.

Managing Feature Gaps with Separate Workspaces

Using a separate, non-compliant Azure Databricks workspace for tasks that require features not allowed under PCI-DSS (such as serverless compute) is a recommended practice from a security, compliance, and operational perspective. This approach is called "workload isolation," and it allows the organization to maintain PCI-DSS compliance for sensitive workflows while utilizing cost-effective or flexible features elsewhere for non-sensitive workloads.

Key strategies for managing multiple workspaces:

Clear Data Segregation: Ensure no PCI-sensitive data is moved into or processed in the non-compliant workspace.
Access Controls: Enforce strong identity and access management policies, limiting data movement between workspaces to only authorized users or automated, audited processes.
Governance and Monitoring: Use centralized tools and logging to retain oversight of workspace usage and data flows. Azure offers features such as Azure Policy, Azure Monitor, and logging integrations for compliance and management across environments.
Naming/Tagging Conventions: Consistently label workspaces based on their compliance posture for easy identification and management.

Best Practices and Considerations

Always keep strict boundaries between compliant and non-compliant workspaces, including separate compute resources and dedicated storage.
Document and regularly review data flows to ensure compliance is not accidentally compromised.
Schedule regular audits and reviews of workspace configurations, permissions, and monitoring rules.
For any tasks or workloads that require PCI scope or protected data, ensure they only occur in the compliant workspace.

Summary Table

Feature / Practice	PCI Workspace	Non-PCI Workspace	Notes
Serverless Compute	Not available	Available	Use non-PCI workspace for these tasks
Data Storage	PCI-compliant only	General storage	No PCI data in non-compliant workspace
Access Controls / Monitoring	Strict, central logging	Flexible, but monitored	May use different configuration sets

Implementing this separation allows organizations to balance compliance and productivity, maintaining PCI standards for sensitive workloads while using advanced Databricks features for general analytics or development.

If additional guidance or architectural support is required, Azure offers reference architectures and compliance documentation for managing these scenarios effectively.

topic Re: Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks in Data Engineering

Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks

Re: Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks

Managing Feature Gaps with Separate Workspaces

Best Practices and Considerations

Summary Table