https://community.databricks.com/t5/administration-architecture/guidance-for-creating-dlt-pipeline-with-raw-json-call-logs/m-p/59824#M852 <P>Hi I am looking for some support on how to handle the following situation.&nbsp; I have a call center that generates call log files in json format that are sent to an s3 bucket.&nbsp; Some of the raw json files contain more than one call log object and they are not in correct json format.&nbsp; I.e. they are one large json file with objects right after each other and may contain up to 8 of these jsons.&nbsp;&nbsp;<BR /><BR />I want to&nbsp;create a delta live data processing pipeline that will<BR />1. ingest new files twice daily<BR />2. separate the individual json objects into one file for each object<BR />3. create a table with the json objects<BR />4. clean the data to get rid of incomplete data<BR />5. add some data columns such as callers who hung up and immediately called back<BR />6. produce aggregated statistics about the calls&nbsp;<BR /><BR />some of the questions I have are:<BR />1. should I ingest the raw json files into a bronze table without separating them out, or separate them before adding them to the bronze level<BR />2.&nbsp; My file live an an s3 bucket that is mounted to my db workspace, does this mean they are already in the "bronze" level, or do i need to copy them from the mounted directory into the DLT pipeline?</P> Fri, 09 Feb 2024 18:21:33 GMT chrisf_sts 2024-02-09T18:21:33Z https://community.databricks.com/t5/administration-architecture/guidance-for-creating-dlt-pipeline-with-raw-json-call-logs/m-p/59824#M852 <P>Hi I am looking for some support on how to handle the following situation.&nbsp; I have a call center that generates call log files in json format that are sent to an s3 bucket.&nbsp; Some of the raw json files contain more than one call log object and they are not in correct json format.&nbsp; I.e. they are one large json file with objects right after each other and may contain up to 8 of these jsons.&nbsp;&nbsp;<BR /><BR />I want to&nbsp;create a delta live data processing pipeline that will<BR />1. ingest new files twice daily<BR />2. separate the individual json objects into one file for each object<BR />3. create a table with the json objects<BR />4. clean the data to get rid of incomplete data<BR />5. add some data columns such as callers who hung up and immediately called back<BR />6. produce aggregated statistics about the calls&nbsp;<BR /><BR />some of the questions I have are:<BR />1. should I ingest the raw json files into a bronze table without separating them out, or separate them before adding them to the bronze level<BR />2.&nbsp; My file live an an s3 bucket that is mounted to my db workspace, does this mean they are already in the "bronze" level, or do i need to copy them from the mounted directory into the DLT pipeline?</P> Fri, 09 Feb 2024 18:21:33 GMT https://community.databricks.com/t5/administration-architecture/guidance-for-creating-dlt-pipeline-with-raw-json-call-logs/m-p/59824#M852 chrisf_sts 2024-02-09T18:21:33Z