cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Bucketing on Delta Tables

Rahul_Samant
Contributor

getting error as below while creating buckets on delta table.

Error in SQL statement: AnalysisException: Delta bucketed tables are not supported.

have fall back to parquet table due to this for some use cases. is their any alternative for this. i have to join over 30 tables and biggest table have over 300M Records . joining fields have high cardinality and can't be used for partition so need to do bucketing.

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

Hi @Rahul Samantโ€‹  , we checked internally on this due to certain limitations bucketing is not supported on delta tables, the only alternative for bucketing is to leverage the z ordering, below is the link for reference

https://docs.databricks.com/delta/optimizations/file-mgmt.html#z-ordering-multi-dimensional-clusteri...

View solution in original post

4 REPLIES 4

-werners-
Esteemed Contributor III

Delta has Z-Ordering and Bloom filters, check it out, lots to read:

https://docs.databricks.com/delta/optimizations/index.html

Rahul_Samant
Contributor

@Werner Stinckensโ€‹ , Yeah checked it but Z-Ordering has to be triggered separately and it doesn't work well with auto compaction enabled with delta table.

Rahul_Samant
Contributor

Thanks @Kaniz Fatmaโ€‹  for the follow-up , we are using mix of both types. depending on the use-case.

didn't find the alternative for bucketing

Anonymous
Not applicable

Hi @Rahul Samantโ€‹  , we checked internally on this due to certain limitations bucketing is not supported on delta tables, the only alternative for bucketing is to leverage the z ordering, below is the link for reference

https://docs.databricks.com/delta/optimizations/file-mgmt.html#z-ordering-multi-dimensional-clusteri...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group