What do you understand by a cluster manager? What are the different cluster manager types supported by PySpark?
In PySpark, a cluster manager is a cluster mode platform that facilitates Spark to run by providing all resources to worker nodes according to their requirements.
A Spark cluster manager ecosystem contains a master node and multiple worker nodes. The master nodes provide the worker nodes with the resources like memory, processor allocation, etc., according to the nodes' requirements with the help of the cluster manager.
PySpark supports the following cluster manager types :
* Standalone : This is a simple cluster manager that comes with Spark.
* Apache Mesos : This cluster manager is used to run Hadoop MapReduce and PySpark apps.
* Hadoop YARN : This cluster manager is used in Hadoop2.
* Kubernetes : This cluster manager is an open-source cluster manager that helps automate deployment, scaling, and automatic management of containerized apps.
* local : This cluster manager is a mode for running Spark applications on laptops/desktops.
