Delta Live Table & Autoloader adding a non-existent column

YSF — Sun, 30 Apr 2023 20:38:15 GMT

I'm trying to setup autoloader to read some csv files. I tried with both autoloader with the DLT decorator as well as just autoloader by itself. The first column of the data is called "run_id", when I do a spark.read.csv() directly on the file it comes in fine. When I use autoloader then it seems to see the first column as a curly brace "{" can't find anything online what that is or where it's coming from.

Here's a sample of the autoloader call:

df = (spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "csv")
      .option("cloudFiles.includeExistingFiles","true")
      .option("cloudFiles.schemaLocation", "/dbfs/schema_registry/")
      .load("/dbfs/mnt/folder/data_20230519.csv"))

Anyone know what's going on?

Re: Delta Live Table & Autoloader adding a non-existent column

Rishabh-Pandey — Mon, 01 May 2023 10:39:30 GMT

can you attach the exact output so that I can have a look on that .

topic Re: Delta Live Table & Autoloader adding a non-existent column in Data Engineering

Delta Live Table & Autoloader adding a non-existent column

Re: Delta Live Table & Autoloader adding a non-existent column