I was creating delta table from ADLS json input file. but the job was running long while creating delta table from json. Below is my cluster configuration. Is the issue related to cluster config ? Do I need to upgrade the cluster config ?

The cluster was created for non-prod environment and we have complex batch ETL ie.., join, aggregation. Shall i create a small cluster with 400GB memory and 50 cores ? Please advise on this.

Input JSON file size - 5 GB

standard_D3_V2

14 GB memory and 4 cores

worker node - min -2 and max -8

executor type -standard_D3_V2

14GB memory and 4 cores

Note- the cluster was ALLPURPOSE