issue in loading the json files in same container with different schemas

kickbuttowski — Sat, 16 Mar 2024 14:44:19 GMT

Could you tell whether this scenario will work or not

Scenario : i have a container which is having two different json files with diff schemas which will be coming in a streaming manner , i am using an auto loader here to load the files incrementally , can one autoloader the json files in the container which is having two diff schemas ? i've already tried with one file and one schema and its working , but i'm struck here while doing it for two types of json files. for loading two json files , i have stored the schema in the adls gen2 container and called it in my notebook ,but it didnt help me

Re: issue in loading the json files in same container with different schemas

MichTalebzadeh — Sat, 16 Mar 2024 16:07:26 GMT

Short answer is no. A single Spark AutoLoader typically cannot handle JSON files in a container with two different schemas by default.. AutoLoader relies on schema inference to determine the data structure. It analyses a sample of data from files assumed to have a consistent schema. If files have different schemas , AutoLoader's inferred schema would be inaccurate. This leads to errors when processing files with structures that dont match the inferred schema.

topic Re: issue in loading the json files in same container with different schemas in Data Engineering

issue in loading the json files in same container with different schemas

Re: issue in loading the json files in same container with different schemas