When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

[159312]

When trying to ingest parquet files with autoloader with the following code

df = (spark

   .readStream

   .format("cloudFiles")

   .option("cloudfiles.format","parquet")

   .load(filePath))

I get the following error:

java.lang.UnsupportedOperationException: Schema inference is not supported for format: parquet. Please specify the schema.

I find this strange because parquet files contain schema information. There is nothing to infer.

If I pull the schema from one of the existing parquet files autoloader works.

filePath = '/dbfs/mnt/dops/streamtest/public/streamme/'

files = os.listdir(filePath)

files.sort()

sdata = spark.read.parquet(os.path.join(file_path[5:], files[-1]))

df = (spark

   .readStream

   .format("cloudFiles")

   .option("cloudfiles.format","parquet")

   .schema(sdata.schema)

   .load(filePath))

This does work but eliminates one of the primary benefits of autoloader: no directory listing.

Is this expected behavior? I have trouble understanding why autoloader cannot read the schema from parquet files.

Thanks,

Ben

[Noopur_Nigam]

Hi @Ben Bogart​ This is supported in DBR 11.1 and above.The below document suggests the same:

https://docs.databricks.com/ingestion/auto-loader/schema.html#schema-inference-and-evolution-in-auto...

Please try in DBR 11.1 and please let us know if you still face the issue.

[jose_gonzalez]

Hi @Ben Bogart​,

Just a friendly follow-up. Did Noopur's response help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

[159312]

While I can confirm that schema inference is supported in DBR 11.1, it is still not supported in either the DLT Current or Preview runtimes which is where I need it. Womp womp.