Databricks Community

keshavmonga22

SetupSingleton Python logger (logging.FileHandler) writing .log files during ADF-orchestrated notebook runs (one notebook per file). Files need to land in a UC Volume.Context on the migration path — this workflow has been reshaped repeatedly by the m...

keshavmonga22 · 07-23-2026

We have a Lakeflow (DLT) pipeline that runs as a task inside a job. A service principal is set as the run-as identity.The pipeline originally created its streaming tables and a metadata Delta table (which gets updated in a separate task in the job) i...

keshavmonga22

Hi @gowri_databrick , on this:"For an ADF-orchestrated workflow with multiple notebook runs, would it make sense to treat each run independently — create a unique log file using the run ID, keep the logging local during execution, and persist the com...

keshavmonga22

Hi @DoTA , I have been using a singleton logger as there are helper classes being used within the notebook which will also need to use the logger.

keshavmonga22

Hey @data_pulse , thanks for replying and trying these scenarios out!I want to add that the shutil.copy() was working fine for about 2 months and then recently started hanging.With regards to committing .log files directly to Unity Catalog, I started...

keshavmonga22 · 07-27-2026

Hi @binlogreader and @adnan_alvee , thank you for the replies. But I don't have access to system.access either.We had temporarily granted USE SCHEMA permissions on the older schema to run the pipelines.Since then, I have deleted the schema entirely s...

keshavmonga22 · 01-29-2025

@Jim_Anderson I've completed the 4 modules from the Data Engineer Learning Plan. Looking forward to receiving the voucher.

Databricks Community

User Stats

User Activity

shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log

Pipeline still needs USE SCHEMA on an old schema it no longer writes to

Re: shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for

Re: shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for

Re: shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for

Re: Pipeline still needs USE SCHEMA on an old schema it no longer writes to

Re: Databricks Learning Festival (Virtual): 15 January - 31 January 2025