Source to Bronze Organization + Partition

ChristianRRL — Tue, 19 Dec 2023 15:55:21 GMT

Hi there, I hope I have what is effectively a simple question. I'd like to ask for a bit on guidance if I am structuring my source-to-bronze auto loader data properly. Here's what I have currently:

/adls_storage/<data_source_name>/<category>/autoloader/<oem_shortname>/<linted_database_shortname>/<linted_table_name>/source/<project_id_partition>/<full_file_name>.csv

For this example, I'm trying to set up a source-to-*bronze* pipeline. Some examples I've seen online are a bit closer to the following:

.../autoloader/<oem>/<database>/<table>/source/project_id={x}/*.csv (aka: original raw data)
.../autoloader/<oem>/<database>/<table>/bronze (aka: the ingested bronze data)
.../autoloader/<oem>/<database>/<table>/checkpoint (still a bit unfamiliar with this one)
.../autoloader/<oem>/<database>/<table>/schema (i.e. keep track of current or evolving schema)

At the end of the day, I'm essentially wanting to have a clean database.table format that looks like the following:

{data_source_name}_{category}.bronze_{oem}_{database}_{table}
...
{data_source_name}_{category}.silver_{database}_{table}

Does this seem right or am I missing anything?

topic Source to Bronze Organization + Partition in Get Started Discussions

Source to Bronze Organization + Partition