- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2024 03:53 PM
Hello! I have a delta table with a check constraint - it's one of many that a config-driven ETL pipeline of mine generates. When someone edits the config file and deploys the change, I'd like for the check constraint to be updated as well if it's different. The problem I'm running into is that the check constraint's condition is redacted in the table properties, so I can't compare what's in the config with what is in the table properties. I am an admin in this workspace. Is this redacted condition expected, or is it possible to get the actual SQL syntax to show through another setting?
Output from SHOW TBLPROPERTIES or DESCRIBE EXTENDED:
```'delta.minReaderVersion': '1', 'delta.minWriterVersion': '3', 'delta.constraints.sql_data_quality': '*********(redacted)'```
Thanks in advance if anyone knows what's going on.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2024 06:51 PM
Figured this out with the help of @SamDataWalk 's post https://community.databricks.com/t5/data-engineering/databricks-bug-with-show-tblproperties-redacted...
It happens because Databricks thinks certain keywords in the constrain query columns like "url" or "userid" are sensitive. You can disable this behavior with
spark.conf.set("spark.databricks.behaviorChange.SC102534CommandRedactProperties.enabled", False)