Can you z-order a table with Spark Structured Streaming?

christian_acuna
New Contributor

Hi, can you do zordering as part of a structured streaming ingestion? 

peterlandis
New Contributor II

You can zorder by reading in the structured streaming data to a delta table and the zorder the part that is needed on the written delta table.  So you essentially leverage foreach batch writing out the structured stream to a delta table in small micro batches and then zorder the data after each batch.

Brian2
New Contributor III

You can also turn on auto optimize writes for your table or session.  This is especially good if your table is partitioned.  You should also turn on auto compaction to help control the size of your files.

Auto compaction can be enabled at the table or session level using the following settings:

  • Table property: delta.autoOptimize.autoCompact

  • SparkSession setting: spark.databricks.delta.autoCompact.enabled

Brian2
New Contributor III

And this is the setting to turn on auto optimization

 

Optimized writes can be enabled at the table or session level using the following settings:

  • Table setting: delta.autoOptimize.optimizeWrite

  • SparkSession setting: spark.databricks.delta.optimizeWrite.enabled