connect RabbitMQ hosted on AWS EC2 instance

[ajaysh]

We are facing an issue. We created a cluster "dev_cluster_catalog_support" . When we are using compute policy "unrestricted" we are able to connect RabbitMQ hosted on AWS EC2 instance, but when we change the compute policy "Shared Compute" getting error "connection refused".

any suggestion or step to resolve above issue.

[BigRoux]

RabbitMQ, it brings me back to my IoT days 🙂

With that said, here is some helpful guidance:

The issue occurs because Shared Compute cluster policies enforce stricter network access controls compared to Unrestricted policies. When using the Unrestricted policy, your cluster has full network access with no restrictions on outbound connections. However, the Shared Compute policy implements network security controls that can block connections to external services like your RabbitMQ instance on AWS EC2.

Key Differences Between Policies

Unrestricted Policy:

• No restrictions on cluster configurations or network access

• Full flexibility for outbound connections to any destination

• Users can create clusters with any settings and network configurations

Shared Compute Policy:

• Designed for multi-user shared resources with security guardrails

• Implements network restrictions and access controls

• May limit outbound connectivity to external services for security purposes

Helpful Links to investigate further:

https://docs.databricks.com/aws/en/compute/cluster-config-best-practices

https://www.chaosgenius.io/blog/databricks-clusters/

Hope this helps.

Cheers, Louis.