As part of our Azure Data Factory pipeline, we utilize Databricks to run some scripts that identify which files we need to load from a certain source. This list of files is then passed back into Azure Data Factory utilizing the Exit status from the notebook.
The problem that we’ve noticed is that portions of certain filenames are being [REDACTED] as there is a pattern in the filename that matches one of our secrets.
My questions for you:
- Is it possible to turn off the redaction of secrets?
- Currently doesn’t pose an issue for us as developers have access to key vaults in the environments they have access to Dataabricks in
- Is there another way to return values from a Notebook to Data Factory other than the dbutils.notebook.exit() function?