can databricks support murmur hash function

gvvishnu — Wed, 22 Jan 2025 21:36:46 GMT

current project we are using murmur hash function in hadoop.we are planning for migration to databricks.can databricks support murmur hash function ?

Re: can databricks support murmur hash function

brockb — Sun, 26 Jan 2025 05:03:48 GMT

Hi @gvvishnu ,

Thanks for your question.

My understanding is that the Apache Spark `hash()` function implements the `org.apache.spark.sql.catalyst.expressions.Murmur3Hash` expression.

You can see this in the Spark source code here:

https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-functions/sql-expression-schema.md?plain=1#L232

The best way however to confirm that this is equivalent to the existing 'murmur hash function in hadoop' that you have previously used is to do some tests and comparisons. Please do that testing and verify that this function returns the expected results.

Thank you.

topic can databricks support murmur hash function in Data Engineering

can databricks support murmur hash function

Re: can databricks support murmur hash function