cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is it possible to enable encryption in between worker nodes?

User16826992666
Valued Contributor

I have a security requirement to encrypt all data when it is in transit. I am wondering if there is a setting I can use to enable encryption of the data during shuffles between the worker nodes.

3 REPLIES 3

User16783853906
Contributor III

If your environment requires that data be encrypted at all times, whether at rest or in transit, you can create an init script that configures your clusters to encrypt traffic between worker nodes, using AES 128-bit encryption over a TLS 1.2 connection.

More guidelines mentioned here - https://docs.databricks.com/security/encryption/encrypt-otw.html

sajith_appukutt
Honored Contributor II

If it is on aws, consider using Nitro instances which gives this automatically. For more details check https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/data-protection.html#encryption-transit

amr
New Contributor III
New Contributor III

Inter-node encryption is a requirement for HIPPA compliance, reach out to your account management team and ask them for HIPPA compliant shards.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!