Hi @mits1,

Sorry. I jumped to conclusions based on the post header and its relation to the other one. However, I don't think @lingareddy_Alva's reason is accurate either. This is because there is a distinction between schema inference and the actual stream processing. During schema inference, Auto Loader samples up to 50 GB or 1,000 files and generates a schema JSON file, which is stored in _schemas under cloudFiles.schemaLocation. In the actual stream processing phase, it uses the inferred schema to read files and write to your Delta table. During schema inference, Auto Loader does not write any data rows to your target table. It only inspects files and saves the inferred schema. Nothing about that phase creates "placeholder partitions" that get materialized as NULL rows. Empty partitions in Spark simply produce zero output rows. They don’t generate rows full of NULLs.

In contrast, Spark's JSON reader, including Auto Loader, operates in permissive mode, treating each line as a single JSON record. In this mode, any malformed, blank, or non-JSON lines result in records with actual columns set to null. Also, the raw text from these lines is stored in the _corrupt_record or rescued-data column. I'm guessing these are the rows you’re seeing.

So the more likely explanation is that on the first run, Auto Loader processes all pre‑existing files in /Volumes/workspace/dev/input/. Some of those lines/files are empty, whitespace‑only, or otherwise invalid JSON --> 33 NULL rows. Those files are now marked processed in the checkpoint and schema location, so subsequent runs never re‑read them, hence no more null rows.

Just to narrow this down, can you run a batch read against the exact same path and share what you see?

df = (spark.read
      .format("json")
      .option("columnNameOfCorruptRecord", "_corrupt_record")
      .load("/Volumes/workspace/dev/input/"))

df.select("*").where("_corrupt_record IS NOT NULL").show(truncate=False)

Can you also double‑check to ensure there aren’t extra or zero‑byte files in that directory.

display(dbutils.fs.ls("/Volumes/workspace/dev/input/"))

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.