topic Re: add new column to a table and failing the previous jobs in Get Started Discussions

add new column to a table and failing the previous jobs

leticialima__ — Sat, 02 Aug 2025 21:12:34 GMT

Hello community! 👋 I’m new to Databricks and currently working on a project structured in Bronze / Silver / Gold layers using Delta Lake and Change Data Feed.

I recently added 3 new columns to a table and initially applied these changes via PySpark SQL commands within our generic job files that handle streaming between layers. I later realized this might not be the best approach, so I dropped and re-added the columns using a Databricks notebook instead.

However, now my job is failing without a specific error. The job run shows the following message:

Cannot read the Python file source_to_bronze_loader.py. Please check driver logs.

Has anyone encountered something similar? What might cause this issue or how can I further troubleshoot it?

Any guidance would be appreciated. Thanks!

Re: add new column to a table and failing the previous jobs

Khaja_Zaffer — Sun, 03 Aug 2025 02:45:43 GMT

Hello @leticialima__

Good day

Can you please share the error observed on the driver log.

is it : [Errno 13] Permission denied or No such file or directory? Please let me know the error on the driver log.

THank you.

Re: add new column to a table and failing the previous jobs

leticialima__ — Mon, 04 Aug 2025 12:30:36 GMT

Hello, I gave up trying to solve it on Saturday, and today when I checked the failure log of the scheduled job, a new error appeared. the link in the error: https://docs.databricks.com/aws/en/delta/column-mapping

Re: add new column to a table and failing the previous jobs

SP_6721 — Wed, 06 Aug 2025 12:04:40 GMT

Hi @leticialima__ ,

The failure is likely due to a non-additive schema change, such as dropping and re-adding columns. To handle such changes, you can set the schemaTrackingLocation option in your readStream query.
Also, ensure that column mapping is enabled on the table.

https://docs.databricks.com/aws/en/delta/column-mapping#streaming-with-column-mapping-and-schema-changes

Re: add new column to a table and failing the previous jobs

leticialima__ — Thu, 14 Aug 2025 12:25:06 GMT

Thanks!! It worked!