cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

kjoth
by Contributor II
  • 977 Views
  • 0 replies
  • 0 kudos

Unmanaged Table - Newly added data directories are not reflected in the table We have created an unmanaged table with partitions on the dbfs location, using SQL. After creating the tables, via SQL we are running

We have created an unmanaged table with partitions on the dbfs location, using SQL.example: %sql CREATE TABLE EnterpriseDailyTrafficSummarytest(EnterpriseID String,ServiceLocationID String, ReportDate String ) USING parquet PARTITIONED BY(ReportDate)...

  • 977 Views
  • 0 replies
  • 0 kudos
brickster_2018
by Esteemed Contributor
  • 857 Views
  • 1 replies
  • 0 kudos

Resolved! Does Delta perform listing of data directories?

I never ran VACUUM on the Delta table. Will Delta perform direct listing on those directories, I am afraid the query performance is going to get degraded over timeHow about the log directories. I have more than 100k JSON files in the log directory

  • 857 Views
  • 1 replies
  • 0 kudos
Latest Reply
brickster_2018
Esteemed Contributor
  • 0 kudos

For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed. Similarly with in the Delta log directory listing is performed. However, it's performed using a continu...

  • 0 kudos
Labels