topic Re: Can you z-order a table with Spark Structured Streaming? in Data Engineering

Can you z-order a table with Spark Structured Streaming?

christian_acuna — Wed, 28 Jun 2023 22:32:01 GMT

Hi, can you do zordering as part of a structured streaming ingestion?

Re: Can you z-order a table with Spark Structured Streaming?

peterlandis — Wed, 28 Jun 2023 23:07:47 GMT

You can zorder by reading in the structured streaming data to a delta table and the zorder the part that is needed on the written delta table. So you essentially leverage foreach batch writing out the structured stream to a delta table in small micro batches and then zorder the data after each batch.

Re: Can you z-order a table with Spark Structured Streaming?

Brian2 — Wed, 28 Jun 2023 23:25:03 GMT

You can also turn on auto optimize writes for your table or session. This is especially good if your table is partitioned. You should also turn on auto compaction to help control the size of your files.

Auto compaction can be enabled at the table or session level using the following settings:

Table property: delta.autoOptimize.autoCompact
SparkSession setting: spark.databricks.delta.autoCompact.enabled

Re: Can you z-order a table with Spark Structured Streaming?

Brian2 — Wed, 28 Jun 2023 23:26:51 GMT

And this is the setting to turn on auto optimization

Optimized writes can be enabled at the table or session level using the following settings:

Table setting: delta.autoOptimize.optimizeWrite
SparkSession setting: spark.databricks.delta.optimizeWrite.enabled