We are ingesting our data from ADLS into databricks as delta table. After raw layer we need to refer to a control\mapping layer which defines certain logic\measure definition. This would be incorporated in the subsequent silver or gold layer. This control layer wil be maintained by business users and regularly updated by them. My questions based on this scenario:
a. Should this control layer reside in databricks or as a csv file? I do not think business users can directly manipulate a control (delta table) on regular basis without a script.
b. If this is going to be a csv file maintained by business users then should it reside in a Sharepoint and we pull this file into ADLS using ADF pipeline and then databricks can pick from ADLS location?
c. Do you propose a better way for the correct location and maintainance of the control table for the databricks to consume for transformation logic while giving the business users rights to modify it easily?
Thanks!
