Can you z-order a table with Spark Structured Streaming?

[christian_acuna]

Hi, can you do zordering as part of a structured streaming ingestion? 

[peterlandis]

You can zorder by reading in the structured streaming data to a delta table and the zorder the part that is needed on the written delta table.  So you essentially leverage foreach batch writing out the structured stream to a delta table in small micro batches and then zorder the data after each batch.

[Brian2]

You can also turn on auto optimize writes for your table or session.  This is especially good if your table is partitioned.  You should also turn on auto compaction to help control the size of your files.

Auto compaction can be enabled at the table or session level using the following settings:

• Table property: delta.autoOptimize.autoCompact

• SparkSession setting: spark.databricks.delta.autoCompact.enabled

[Brian2]

And this is the setting to turn on auto optimization

Optimized writes can be enabled at the table or session level using the following settings:

• Table setting: delta.autoOptimize.optimizeWrite

• SparkSession setting: spark.databricks.delta.optimizeWrite.enabled