Autoloader fails when creating external Delta table in same notebook

yit
Databricks Partner

‎08-07-2025 12:49 AM

Hi everyone,

I’ve set up Databricks Autoloader to ingest data from ADLS into a Delta table. The table is defined as an external Delta table, with its location pointing to a path in ADLS.

Here’s the flow I’m using:

  • On the first run for a given data source, I create the external Delta table in a notebook.

  • Immediately after, I invoke Autoloader (within the same notebook) to start streaming data into the table.

However, I often (but not always) encounter the following error on the first run

 
Failed to write to the schema log at location abfss://{container}@{storage_account}.dfs.core.windows.net/my-path/schema. SQLSTATE: XXKST

As a workaround, I tried splitting the process:

  • I run one notebook to create the external table.

  • Then, I run another notebook separately to start the Autoloader.

With this approach, the error does not occur.

 

What could be causing this intermittent schema log write failure when creating the table and starting Autoloader in the same notebook? Is this a timing or locking issue due to table creation and Autoloader initialization being too close together?

 

0 Kudos