cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
kjoth
by • Contributor II
  • 1334 Views
  • 0 replies
  • 0 kudos

Unmanaged Table - Newly added data directories are not reflected in the table We have created an unmanaged table with partitions on the dbfs location, using SQL. After creating the tables, via SQL we are running

We have created an unmanaged table with partitions on the dbfs location, using SQL.example: %sql CREATE TABLE EnterpriseDailyTrafficSummarytest(EnterpriseID String,ServiceLocationID String, ReportDate String ) USING parquet PARTITIONED BY(ReportDate)...

Data Engineering
Reply
  • 1334 Views
  • 0 replies
  • 0 kudos
brickster_2018
by • Databricks Employee
  • 1223 Views
  • 1 replies
  • 0 kudos

Resolved! Does Delta perform listing of data directories?

I never ran VACUUM on the Delta table. Will Delta perform direct listing on those directories, I am afraid the query performance is going to get degraded over timeHow about the log directories. I have more than 100k JSON files in the log directory

Data Engineering
Reply
  • 1223 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed. Similarly with in the Delta log directory listing is performed. However, it's performed using a continu...

  • 0 kudos
Labels