Delta Live Table & Autoloader adding a non-existent column
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-30-2023 01:38 PM
I'm trying to setup autoloader to read some csv files. I tried with both autoloader with the DLT decorator as well as just autoloader by itself. The first column of the data is called "run_id", when I do a spark.read.csv() directly on the file it comes in fine. When I use autoloader then it seems to see the first column as a curly brace "{" can't find anything online what that is or where it's coming from.
Here's a sample of the autoloader call:
df = (spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.includeExistingFiles","true")
.option("cloudFiles.schemaLocation", "/dbfs/schema_registry/")
.load("/dbfs/mnt/folder/data_20230519.csv"))
Anyone know what's going on?
Labels:
- Labels:
-
Autoloader
-
Delta
-
DLT
-
Live Table
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2023 03:39 AM
can you attach the exact output so that I can have a look on that .
Rishabh Pandey

