Consider you have the below details regarding the cluster:
Number of nodes = 10
Number of cores in each node = 15 cores
RAM of each node = 61GB
To identify the number of cores, we follow the approach:
Number of Cores = number of concurrent tasks that can be run parallelly by the executor.
The optimal value as part of a general rule of thumb is 5.
Hence to calculate the number of executors, we follow the below approach :
Number of executors = Number of cores/Concurrent Task
= 15/5
= 3
Number of executors = Number of nodes * Number of executor in each node
= 10 * 3
= 30 executors per Spark job
