Databricks Community

Sainath368

Hi all,We are in the process of migrating from directory listing to managed file events in Azure Databricks. Our data is stored in an Azure Data Lake container with the following folder structure:To enable file events in Unity Catalog (UC), I created...

Sainath368

Hi everyone,We are currently migrating from a directory listing-based streaming approach to managed file events in Databricks Auto Loader for processing our data in structured streaming.We have a function that handles structured streaming where we ar...

Sainath368 · 09-01-2025

Hi everyone,We’re currently reviewing the performance impact of enabling Photon acceleration on our Databricks jobs, particularly those involving table maintenance tasks. Our job includes three main operations: OPTIMIZE, VACUUM, and ANALYZE_COMPUTE_S...

Sainath368 · 08-04-2025

Hi everyone,I regularly (once a week) run the analyze table compute statistics command on all my tables in Databricks to keep statistics up to date for query optimization.In the Spark table UI catalog, I can see some statistics metadata like spark.st...

Sainath368 · 07-28-2025

Hi all,I need to run weekly maintenance on approximately 7,000 tables in my Databricks environment, involving OPTIMIZE, VACUUM, and ANALYZE TABLE (for statistics calculation) on all tables.My question is: between the Ev4, Edv4, and Fsv2 VM series, wh...

Sainath368

Hi @K_Anudeep, Thank you for the detailed explanation! That clears up the path configuration for using Auto Loader with managed file events.I have a follow-up question:If we proceed with one stream per table, will there be a single queue for all stre...

Sainath368

Hi @Raman_Unifeye @szymon_dybczak , Considering my setup which i already explained, does it make any difference between ManagedFileEvents and Legacy file notification mode ? if yes in what aspects.

Sainath368 · 07-08-2025

Hi @Waldar , The approximate count matches the result from the DESCRIBE EXTENDED query. This confirms that the query you provided helps verify the count correctly. But what would be the reason?

Sainath368 · 07-04-2025

Thank you for your earlier reply—which was helpful—but I’m still looking for a clearer explanation, so I’m asking this in more detail.I'm working with external Delta tables that don't use Predictive I/O optimization, so I set the delta.dataSkippingSt...

Sainath368 · 07-03-2025

I ran ANALYZE TABLE <table_name> COMPUTE DELTA STATISTICS on my tables to calculate column-level statistics at the file level for each Parquet file inside the Delta table. Now, I’m wondering which command is better to run next: ANALYZE TABLE <table_n...

Databricks Community

User Stats

User Activity

Autoloader Managed File events

Migrating from directory-listing to Autoloader Managed File events

Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STATS)?

How to Retrieve the spark.statistics.createdAt When Statistics Were Last Updated in Databricks?

E series vs F series VM's

Re: Autoloader Managed File events

Re: Migrating from directory-listing to Autoloader Managed File events

Re: ANALYZE command for stats collection - distinct_count difference for a column

Re: Clarification Needed: COMPUTE STATISTICS vs COMPUTE DELTA STATISTICS on Delta Tables

Re: Clarification Needed: COMPUTE STATISTICS vs COMPUTE DELTA STATISTICS on Delta Tables