https://community.databricks.com/t5/data-engineering/writing-to-multiple-files-tables-from-data-held-within-a-single/m-p/65612#M32862 <P>I have a requirement to read and parse JSON files using autoloader where incoming JSON file has multiple sub entities. Each sub entity needs to go into its own delta table. Alternatively we can write each entity data to individual files. We can use DLT streaming or standard autoloader. Please suggest how can this be achieved with Azure Databricks.</P> Fri, 05 Apr 2024 13:52:19 GMT Anandsingh 2024-04-05T13:52:19Z https://community.databricks.com/t5/data-engineering/writing-to-multiple-files-tables-from-data-held-within-a-single/m-p/65612#M32862 <P>I have a requirement to read and parse JSON files using autoloader where incoming JSON file has multiple sub entities. Each sub entity needs to go into its own delta table. Alternatively we can write each entity data to individual files. We can use DLT streaming or standard autoloader. Please suggest how can this be achieved with Azure Databricks.</P> Fri, 05 Apr 2024 13:52:19 GMT https://community.databricks.com/t5/data-engineering/writing-to-multiple-files-tables-from-data-held-within-a-single/m-p/65612#M32862 Anandsingh 2024-04-05T13:52:19Z https://community.databricks.com/t5/data-engineering/writing-to-multiple-files-tables-from-data-held-within-a-single/m-p/65628#M32869 <P>I think using DLT's medallion architecture should be helpful in this scenario. You can write all the incoming data to one bronze table and one silver table. And you can have multiple gold tables based on the value of the sub-entities.</P> Fri, 05 Apr 2024 16:55:48 GMT https://community.databricks.com/t5/data-engineering/writing-to-multiple-files-tables-from-data-held-within-a-single/m-p/65628#M32869 Lakshay 2024-04-05T16:55:48Z