topic Data engineering professional exam in Data Engineering

Data engineering professional exam

anshi_t_k — Tue, 03 Sep 2024 10:07:18 GMT

Each configuration below is identical in that each cluster has 400 GB total of RAM, 160 total cores, and only one Executor per VM.

Given an extremely long-running job for which completion must be guaranteed, which cluster configuration will be able to guarantee completion of the job in light of one or more VM failures?

A. Total VMs: 8, 50 GB per Executor, 20 Cores / Executor
B. Total VMs: 16, 25 GB per Executor, 10 Cores / Executor
C. Total VMs: 1, 400 GB per Executor, 160 Cores / Executor
D. Total VMs: 4, 100 GB per Executor, 40 Cores / Executor
E. Total VMs: 2, 200 GB per Executor, 80 Cores / Executor

Re: Data engineering professional exam

anshi_t_k — Tue, 03 Sep 2024 10:09:17 GMT

I am confused between option B and D according to different sites, can anyone provide a clarity regarding this

Re: Data engineering professional exam

filipniziol — Tue, 03 Sep 2024 10:32:26 GMT

Hi @anshi_t_k ,

The key consideration here is fault tolerance.
How do you protect against a VM failure? By having more VMs, as the impact of a single VM failure will be the lowest.

For example answer C - the crash of the VM is loosing 1/1 so 100% capacity: no fault tolerance.
In case of answer B - the crash of the VM means loosing 1/16, so only 6,25% capacity.
The other answer you mentioned B - the crash of the VM means loosing 1/4, so 25% capacity.

The answer is B:

It has the highest number of VMs, which spreads the workload more evenly.
The impact of any single VM failure is minimal, with only 6.25% of the total capacity lost per VM failure.
This configuration provides the best fault tolerance, as the cluster can sustain multiple VM failures and still continue to operate effectively, ensuring that the job can complete.

Re: Data engineering professional exam

anshi_t_k — Tue, 03 Sep 2024 11:31:38 GMT

Thank you for the information.