cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

kjoth
by Contributor II
  • 686 Views
  • 0 replies
  • 0 kudos

Unmanaged Table - Newly added data directories are not reflected in the table We have created an unmanaged table with partitions on the dbfs location, using SQL. After creating the tables, via SQL we are running

We have created an unmanaged table with partitions on the dbfs location, using SQL.example: %sql CREATE TABLE EnterpriseDailyTrafficSummarytest(EnterpriseID String,ServiceLocationID String, ReportDate String ) USING parquet PARTITIONED BY(ReportDate)...

  • 686 Views
  • 0 replies
  • 0 kudos
User16869510359
by Esteemed Contributor
  • 634 Views
  • 1 replies
  • 0 kudos

Resolved! Does Delta perform listing of data directories?

I never ran VACUUM on the Delta table. Will Delta perform direct listing on those directories, I am afraid the query performance is going to get degraded over timeHow about the log directories. I have more than 100k JSON files in the log directory

  • 634 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed. Similarly with in the Delta log directory listing is performed. However, it's performed using a continu...

  • 0 kudos
Labels