Is it possible to turn off the redaction of secrets? Is there a better way to solve this?

[StephanieAlba]

As part of our Azure Data Factory pipeline, we utilize Databricks to run some scripts that identify which files we need to load from a certain source. This list of files is then passed back into Azure Data Factory utilizing the Exit status from the notebook.

The problem that we’ve noticed is that portions of certain filenames are being [REDACTED] as there is a pattern in the filename that matches one of our secrets.

My questions for you:

• Is it possible to turn off the redaction of secrets?
  • Currently doesn’t pose an issue for us as developers have access to key vaults in the environments they have access to Dataabricks in
• Is there another way to return values from a Notebook to Data Factory other than the dbutils.notebook.exit() function?

[StephanieAlba]

No, it is not possible to turn off redaction. No, there is not another way to return values from a notebook.

1) Using a native Databricks feature such as Autoloader is suggested.

2) They could write the list of files to be processed to a delta table and then use a lookup activity - https://docs.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity