Hi, I'm trying to read YAML files using pyyaml and convert them into a Spark DataFrame with createDataFrame, without specifying a schema—allowing flexibility for potential YAML schema changes over time. This approach worked as expected on Databricks runtime 13.3, but does not seem to function correctly on runtime 15.4. Any suggestions? 

My yaml schema is as below which I can read well in 13.3 runtime, however I get '[CANNOT_INFER_TYPE_FOR_FIELD] Unable to infer the type of the field `dataset`' on 15.4 runtime.