cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to do bucketing in Databricks?

Arun_tsr
New Contributor III

We are migrating a job from onprem to databricks. We are trying to optimize the jobs but couldn't use bucketing because by default databricks stores all tables as delta table and it shows error that bucketing is not supported for delta. Is there anyway to do this?

6 REPLIES 6

Debayan
Esteemed Contributor III
Esteemed Contributor III

Arun_tsr
New Contributor III

Hi @Debayan Mukherjee​ We are following similar syntax for creating the bucketed table. But we are getting the following error,

Operation not allowed: `Bucketing` is not supported for Delta tables

Databricks by default considering the created tables as delta

Pat
Honored Contributor III

Hi @Arun Balaji​ ,

bucketing is not supported for the delta tables as you have noticed.

For the optimization and best practices with delta tables check this:

https://docs.databricks.com/optimizations/index.html

https://docs.databricks.com/delta/best-practices.html

Arun_tsr
New Contributor III

Is it possible to create a table without making it a delta table​?

Pat
Honored Contributor III

good question 😉 I was going to mention this.

You can still use external tables. Those tables will be stored outside the main (root) metastore bucket/container.

https://docs.databricks.com/data-governance/unity-catalog/create-tables.html

https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials....

then you can use parquet tables.

It seems like the way forward is to use managed tables with Unity Catalog, they are gaining some performance improvements over time.

Pat
Honored Contributor III
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!