topic Re: Which cluster mode should I choose for most efficient graph modelling? in Data Engineering

Which cluster mode should I choose for most efficient graph modelling?

Direo — Tue, 01 Mar 2022 15:00:46 GMT

Is there a difference between cluster modes in this case? Can it be that Graphx would work better on single than on standart cluster or high concurrency cluster (for multiple users)? Does less concurrent cluster wourld be more efficient for graph modelling?

Re: Which cluster mode should I choose for most efficient graph modelling?

Hubert-Dudek — Tue, 01 Mar 2022 19:34:56 GMT

Single - for development purposes,

high concurrency - when multiple users run notebooks at the same time in parallel,

so usually standard is the best option.

Regarding VM I bet compute optimized.

I recommend also to read ebook "Spark GraphX in action" https://livebook.manning.com/book/spark-graphx-in-action/table-of-contents/

Re: Which cluster mode should I choose for most efficient graph modelling?

-werners- — Wed, 02 Mar 2022 08:19:04 GMT

I´d say start as cheap as possible and check the runtime.

Re: Which cluster mode should I choose for most efficient graph modelling?

Anonymous — Mon, 07 Mar 2022 00:16:27 GMT

@Direo Direo - What do you think of these answers? If either of them stands out as best, would you please mark it that way? If you have more questions, please, bring them on!