topic Re: When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right? in Data Engineering

When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

159312 — Thu, 30 Jun 2022 17:19:50 GMT

When trying to ingest parquet files with autoloader with the following code

df = (spark

.readStream

.format("cloudFiles")

.option("cloudfiles.format","parquet")

.load(filePath))

I get the following error:

java.lang.UnsupportedOperationException: Schema inference is not supported for format: parquet. Please specify the schema.

I find this strange because parquet files contain schema information. There is nothing to infer.

If I pull the schema from one of the existing parquet files autoloader works.

filePath = '/dbfs/mnt/dops/streamtest/public/streamme/'

files = os.listdir(filePath)

files.sort()

sdata = spark.read.parquet(os.path.join(file_path[5:], files[-1]))

df = (spark

.readStream

.format("cloudFiles")

.option("cloudfiles.format","parquet")

.schema(sdata.schema)

.load(filePath))

This does work but eliminates one of the primary benefits of autoloader: no directory listing.

Is this expected behavior? I have trouble understanding why autoloader cannot read the schema from parquet files.

Thanks,

Ben

Re: When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

Noopur_Nigam — Mon, 25 Jul 2022 10:40:28 GMT

Hi @Ben Bogart This is supported in DBR 11.1 and above.The below document suggests the same:

https://docs.databricks.com/ingestion/auto-loader/schema.html#schema-inference-and-evolution-in-auto-loader

Please try in DBR 11.1 and please let us know if you still face the issue.

Re: When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

jose_gonzalez — Mon, 08 Aug 2022 18:58:43 GMT

Hi @Ben Bogart,

Just a friendly follow-up. Did Noopur's response help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Re: When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

159312 — Wed, 10 Aug 2022 15:02:03 GMT

While I can confirm that schema inference is supported in DBR 11.1, it is still not supported in either the DLT Current or Preview runtimes which is where I need it. Womp womp.