Mapping Control data - Maintained by Business User

[MattM]

We are ingesting our data from ADLS into databricks as delta table. After raw layer we need to refer to a control\mapping layer which defines certain logic\measure definition. This would be incorporated in the subsequent silver or gold layer. This control layer wil be maintained by business users and regularly updated by them. My questions based on this scenario:

a. Should this control layer reside in databricks or as a csv file? I do not think business users can directly manipulate a control (delta table) on regular basis without a script.

b. If this is going to be a csv file maintained by business users then should it reside in a Sharepoint and we pull this file into ADLS using ADF pipeline and then databricks can pick from ADLS location?

c. Do you propose a better way for the correct location and maintainance of the control table for the databricks to consume for transformation logic while giving the business users rights to modify it easily?

Thanks!

[Kaniz · Accepted solution]

Hi @Matt M​, Azure Databricks offers a variety of ways to help you ingest data into Delta Lake.

You can securely create tables from CSV files using the Create Table UI in Databricks SQL.

Databricks partner integrations enable you to load data into Azure Databricks quickly. These integrations allow low-code, easy-to-implement, and scalable data ingestion from various sources into Azure Databricks. See the Databricks integrations.

[MattM]

Thanks for your response. Can business user without the help of any script modify any rows in the table after loading it onetime from CSV fiels?