https://community.databricks.com/t5/data-engineering/load-an-explicit-schema-from-an-external-metadata-csv-file-or-a/m-p/17676#M11644 <P></P> <P>Hi,</P> <P>I have a metadata csv file which contains column name, and datatype such as</P> <P>Colm1: INT</P> <P>Colm2: String.</P> <P>I can also get the same in a json format as shown:</P> <P></P><P></P> <P>I can store this on ADLS. How can I convert this into a schema like: "Myschema" that I can then pass during spark.read.format("csv") method while reading the datafile for the same metadata? When I infer schema for the datafile csv for multiple incremental files , I get clashes while writing into delta such as </P> <P>"Failed to merge fields 'Colm1' and 'Colm1'. Failed to merge incompatible data types IntegerType and StringType </P> <P>Any pointers/notes would be appreciated.</P> <P>Thanks!</P> <P></P> Thu, 15 Jul 2021 11:45:44 GMT AnandNair 2021-07-15T11:45:44Z https://community.databricks.com/t5/data-engineering/load-an-explicit-schema-from-an-external-metadata-csv-file-or-a/m-p/17676#M11644 <P></P> <P>Hi,</P> <P>I have a metadata csv file which contains column name, and datatype such as</P> <P>Colm1: INT</P> <P>Colm2: String.</P> <P>I can also get the same in a json format as shown:</P> <P></P><P></P> <P>I can store this on ADLS. How can I convert this into a schema like: "Myschema" that I can then pass during spark.read.format("csv") method while reading the datafile for the same metadata? When I infer schema for the datafile csv for multiple incremental files , I get clashes while writing into delta such as </P> <P>"Failed to merge fields 'Colm1' and 'Colm1'. Failed to merge incompatible data types IntegerType and StringType </P> <P>Any pointers/notes would be appreciated.</P> <P>Thanks!</P> <P></P> Thu, 15 Jul 2021 11:45:44 GMT https://community.databricks.com/t5/data-engineering/load-an-explicit-schema-from-an-external-metadata-csv-file-or-a/m-p/17676#M11644 AnandNair 2021-07-15T11:45:44Z