topic Re: DQX usage outside Databricks in Data Engineering

DQX usage outside Databricks

wkgcls — Thu, 13 Nov 2025 09:36:30 GMT

Hello,
When evaluating data quality frameworks for PySpark pipelines, I came across DQX. I noticed it's available on PyPI (databricks-labs-dqx) and GitHub, which is great for accessibility.

However, I'm trying to understand the licensing requirements. The LICENSE information on PyPI mentions it as "Other/Proprietary", and the license file on GitHub states that the materials can only be used "in connection with your use of the Databricks Services."

I would like to understand,

Does this mean DQX requires a Databricks platform agreement even when used in other Spark environments (e.g., AWS EMR, Dataproc, or standalone Spark clusters)?
If a Databricks agreement is required, are there specific licensing options for using Labs projects outside the Databricks platform?
Are there any recommended alternatives for data quality validation in non-Databricks PySpark environments that you'd suggest?

I want to ensure we're compliant with all licensing requirements before adopting any framework. Any clarification would be greatly appreciated.

Thanks in advance!
Kiran

Re: DQX usage outside Databricks

ManojkMohan — Thu, 13 Nov 2025 10:00:21 GMT

@wkgcls DQX requires a Databricks platform license for production use, and its licensing restricts usage to environments connected with Databricks Services. This means you cannot use DQX freely in external Spark environments such as AWS EMR

To your questions

1. Yes, DQX requires a Databricks platform agreement for legitimate usage.

2.No, there are currently no publicly documented licensing options that allow Databricks Labs projects, including DQX, to be used outside the Databricks platform.

3.Yes, there are several open-source and permissively licensed alternatives example- Great Expectations: Python-based framework with Spark DataFrame support, configurable validation rules, and strong community adoption.

Re: DQX usage outside Databricks

wkgcls — Thu, 13 Nov 2025 10:37:24 GMT

Thanks a lot for the quick response, @ManojkMohan! This was very helpful.
I'll keep this in mind.