PySpark Partition is a method of splitting a large dataset into smaller datasets based on one or more partition keys. It enhances the execution speed as transformations on partitioned data run quicker because each partition's transformations are executed in parallel. PySpark supports both partitioning in memory (DataFrame) and partitioning on disc (File system). When we make a DataFrame from a file or table, PySpark creates the DataFrame in memory with a specific number of divisions based on specified criteria.
It also facilitates us to create a partition on multiple columns using
partitionBy() by passing the columns you want to partition as an argument to this method.
Syntax :
partitionBy(self, *cols) ?
In PySpark, it is recommended to have 4x of partitions to the number of cores in the cluster available for application.
