topic Re: Bucketing on Delta Tables in Data Engineering

Bucketing on Delta Tables

Rahul_Samant — Mon, 14 Mar 2022 10:55:28 GMT

getting error as below while creating buckets on delta table.

Error in SQL statement: AnalysisException: Delta bucketed tables are not supported.

have fall back to parquet table due to this for some use cases. is their any alternative for this. i have to join over 30 tables and biggest table have over 300M Records . joining fields have high cardinality and can't be used for partition so need to do bucketing.

Re: Bucketing on Delta Tables

-werners- — Tue, 15 Mar 2022 07:36:03 GMT

Delta has Z-Ordering and Bloom filters, check it out, lots to read:

https://docs.databricks.com/delta/optimizations/index.html

Re: Bucketing on Delta Tables

Rahul_Samant — Tue, 15 Mar 2022 07:46:50 GMT

@Werner Stinckens , Yeah checked it but Z-Ordering has to be triggered separately and it doesn't work well with auto compaction enabled with delta table.

Re: Bucketing on Delta Tables

Rahul_Samant — Wed, 27 Apr 2022 08:05:32 GMT

Thanks @Kaniz Fatma for the follow-up , we are using mix of both types. depending on the use-case.

didn't find the alternative for bucketing

Re: Bucketing on Delta Tables

Anonymous — Tue, 10 May 2022 12:57:58 GMT

Hi @Rahul Samant , we checked internally on this due to certain limitations bucketing is not supported on delta tables, the only alternative for bucketing is to leverage the z ordering, below is the link for reference

https://docs.databricks.com/delta/optimizations/file-mgmt.html#z-ordering-multi-dimensional-clustering