- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-05-2024 03:25 PM - edited 08-05-2024 03:27 PM
Technically speaking, neither subnet contains VMs, but NICs. As stated in documentation, each cluster node needs 2 IP addresses:
- one IP address for the host in the host subnet
- and one IP address for the container in the container subnet.
The good illustration of this is in the picture that you've attached. One VM has 2 NICs, one in host subnet, another one in container subnet.
And why each node requires 2 IPs? For an optimal processing experience Databricks segregates the Spark application traffic from the management traffic to avoid Network contention. Spark application traffic communications between the Driver-Executor and the Executors themselves where the computation if happening. Management traffic includes things such as communication between the Control Plane <-> Data Plane, etc