Akka Cluster Interview Questions and Answers

Akka Cluster is a module in the Akka toolkit that allows you to build fault-tolerant, distributed systems by connecting multiple nodes (or instances) into a cluster. It enables nodes to communicate seamlessly with each other, providing resilience, scalability, and support for location transparency.

Key Features of Akka Cluster :

1. Distributed Systems Support:

   • Akka Cluster enables an application to run across multiple machines or instances, where each node participates as a part of the cluster.

2. Location Transparency:

   • Nodes can send messages to actors without knowing the actor's physical location, whether local or remote.

3. Resilience:

   • If a node fails, Akka Cluster can redistribute tasks and maintain functionality using fault-tolerant techniques.

4. Dynamic Membership:

   • Nodes can join or leave the cluster dynamically without disrupting the entire system.

5. Cluster Sharding:

   • Distributes entities (actors) across the cluster automatically, providing load balancing and scalability.

6. Cluster Singleton:

   • Ensures only a single instance of a specific actor runs in the cluster, even if nodes fail or restart.

7. Failure Detection:

   • Uses a gossip protocol and heartbeat messages to monitor node health and detect failures.

Use Cases of Akka Cluster :

1. Microservices:

   • Used to coordinate and communicate between distributed microservices.

2. Real-time Data Processing:

   • Handles large-scale data pipelines in distributed environments.

3. Scalable Systems:

   • For systems that need to scale horizontally by adding new nodes dynamically.

4. Resilient Actor Systems:

   • Provides redundancy and fault tolerance in actor-based applications.

How It Works :

1. Cluster Nodes:

   • Each instance of an Akka application participating in the cluster is called a node. These nodes communicate via the Akka remoting mechanism.

2. Gossip Protocol:

   • Nodes exchange state information using the gossip protocol to maintain cluster state consistency.

3. Membership State:

   • Nodes can have different statuses like Joining, Up, Leaving, or Down, depending on their role in the cluster.

4. Message Delivery:

   • Messages are routed automatically to the appropriate actors, regardless of their physical location.

import akka.actor.ActorSystem
import akka.cluster.Cluster
import com.typesafe.config.ConfigFactory

object ClusterNode extends App {
  // Load cluster configuration
  val config = ConfigFactory.load("application.conf")

  // Create an ActorSystem
  val system = ActorSystem("ClusterSystem", config)

  // Join the cluster
  val cluster = Cluster(system)

  println(s"Node [${cluster.selfAddress}] started and joined the cluster.")
}?

akka {
  actor.provider = cluster
  remote.artery.canonical.hostname = "127.0.0.1"
  remote.artery.canonical.port = 2551
  cluster.seed-nodes = [
    "akka://ClusterSystem@127.0.0.1:2551",
    "akka://ClusterSystem@127.0.0.1:2552"
  ]
}?

A seed node in Akka Cluster is one or more designated nodes used to bootstrap and initialize the cluster. Seed nodes are responsible for helping other nodes discover and join the cluster.

When a node starts, it contacts one or more seed nodes (defined in the configuration) to obtain the current cluster membership information. Once the new node communicates with a seed node, it can successfully join the cluster and participate in distributed operations.

Key Points About Seed Nodes :

1. Discovery Mechanism:

   • Seed nodes act as a discovery point for other nodes to join the cluster.
   • They are not a "special" type of node but regular cluster members with an additional responsibility for the join process.

2. Required for Initialization:

   • At least one seed node must be available for a cluster to form. Other nodes use the seed nodes to get the cluster state.

3. Redundancy:

   • Multiple seed nodes are typically defined in the configuration to provide fault tolerance. If one seed node is unavailable, new nodes can contact the next one in the list.

4. Not Persistent:

   • Seed nodes do not store persistent information about the cluster. They only assist in the discovery process.

How Seed Nodes Work :

1. When a new node starts, it uses the seed node list to find an existing member of the cluster.
2. The new node sends a Join message to the first reachable seed node.
3. If the seed node is part of the cluster, it forwards the Join request to the cluster leader.
4. The cluster leader decides whether the new node can join the cluster. If approved, the new node becomes part of the cluster.

Configuration of Seed Nodes :

Seed nodes are defined in the application.conf file under the akka.cluster.seed-nodes setting.

akka {
  actor.provider = cluster
  remote.artery.canonical.hostname = "127.0.0.1"
  remote.artery.canonical.port = 2551
  cluster {
    seed-nodes = [
      "akka://ClusterSystem@127.0.0.1:2551",
      "akka://ClusterSystem@127.0.0.1:2552"
    ]
  }
}

The gossip protocol in Akka Cluster is a key mechanism used to distribute cluster state information across all nodes in the cluster. It ensures that every node in the cluster has a consistent and up-to-date view of the cluster's current membership and health.

This decentralized approach is lightweight, scalable, and fault-tolerant, making it ideal for distributed systems.

How Gossip Protocol Works :

1. Cluster State Propagation:

   • Each node maintains a local view of the cluster's state, including information about other nodes (e.g., their status, reachability).
   • This cluster state is periodically "gossiped" (shared) between nodes to synchronize their views.

2. Randomized Communication:

   • Nodes communicate with a small, randomly selected subset of other nodes during each gossip cycle. This randomness ensures scalability and avoids overwhelming specific nodes.

3. Eventual Consistency:

   • The gossip protocol guarantees eventual consistency, meaning that all nodes will eventually converge to the same cluster state as information propagates.

4. Failure Detection:

   • Gossip messages include information about node health and reachability, allowing the cluster to detect node failures or network partitions.

5. Scalability:

   • The protocol is designed to scale well in large clusters, as each node communicates with only a subset of other nodes during each cycle.

Cluster Membership Life Cycle :

The gossip protocol manages the states of nodes in the cluster:

• Joining: A node sends a request to a seed node to join the cluster.
• Up: The cluster leader marks a node as fully available for work.
• Leaving: A node signals its intention to leave.
• Exiting: A node finishes its work and prepares to leave the cluster.
• Removed: The node is completely removed from the cluster state.
• Unreachable: Nodes that fail health checks are marked as unreachable.

Advantages of the Gossip Protocol :

1. Decentralized:

   • No single point of failure; all nodes participate equally in sharing state.

2. Fault Tolerance:

   • Handles node failures and network partitions gracefully.

3. Scalable:

   • Communication grows linearly with the number of nodes, making it efficient for large clusters.

4. Resilient:

   • Nodes eventually synchronize even if some messages are lost due to transient network issues.

Consider a cluster with three nodes: NodeA, NodeB, and NodeC.

1. Initial State:

   • NodeA is aware of itself but knows nothing about NodeB and NodeC.

2. Gossip Communication:

   • NodeA gossips its state to NodeB, and NodeB updates its local view.
   • NodeB gossips to NodeC, propagating the updated state.

3. Convergence:

   • Over multiple gossip cycles, all nodes learn about each other and converge to a consistent cluster state.

Example: Configuring Gossip Interval :

You can configure gossip settings in the application.conf file :

akka {
  cluster {
    gossip-interval = 1s          # Frequency of gossip messages
    failure-detector {
      threshold = 8.0            # Adjust failure detection sensitivity
      heartbeat-interval = 1s    # Frequency of heartbeat messages
    }
  }
}

A Cluster Singleton in Akka is a feature that ensures a single instance of an actor is running across the entire cluster at any given time. This is useful for scenarios where you need centralized processing or coordination within a distributed system, but still want fault tolerance and failover support.

Key Features of Cluster Singleton :

1. Single Instance Across the Cluster:

   • Only one instance of the singleton actor exists in the entire cluster, regardless of the number of nodes.

2. Automatic Failover:

   • If the node hosting the singleton actor fails or leaves the cluster, the actor is automatically recreated on another node.

3. Cluster Membership Awareness:

   • The singleton is tied to the cluster's lifecycle and automatically adapts to changes in the cluster (e.g., nodes joining or leaving).

4. Location Transparency:

   • Other actors or components can send messages to the singleton without needing to know where it is currently running.

How Cluster Singleton Works :

1. Singleton Manager:

   • The SingletonManager is responsible for managing the singleton actor across the cluster.
   • It ensures that only one instance of the singleton actor exists at any time.

2. Singleton Proxy:

   • The SingletonProxy allows other actors in the cluster to communicate with the singleton actor without worrying about its physical location.

3. Leader Node:

   • The cluster leader is typically responsible for hosting the singleton actor. If the leader changes, the singleton will move to the new leader node.

Configuration for Cluster Singleton :

You need to define a SingletonManager and (optionally) a SingletonProxy in your code.

import akka.actor.{Actor, ActorSystem, Props}
import akka.cluster.singleton._
import com.typesafe.config.ConfigFactory

object ClusterSingletonExample extends App {
  // Actor to be used as a singleton
  class SingletonActor extends Actor {
    override def receive: Receive = {
      case message => println(s"Singleton Actor received: $message")
    }
  }

  // Create ActorSystem with cluster configuration
  val config = ConfigFactory.load("application.conf")
  val system = ActorSystem("ClusterSystem", config)

  // SingletonManager definition
  val singletonManager = system.actorOf(
    ClusterSingletonManager.props(
      singletonProps = Props[SingletonActor], // Props for the singleton actor
      terminationMessage = PoisonPill,       // Message to terminate the singleton
      settings = ClusterSingletonManagerSettings(system)
    ),
    name = "singletonManager"
  )

  // SingletonProxy definition
  val singletonProxy = system.actorOf(
    ClusterSingletonProxy.props(
      singletonManagerPath = "/user/singletonManager", // Path to SingletonManager
      settings = ClusterSingletonProxySettings(system)
    ),
    name = "singletonProxy"
  )

  // Send a message to the singleton actor via the proxy
  singletonProxy ! "Hello, Singleton!"
}

In Akka Cluster Sharding, the Shard Region plays a critical role in distributing and managing entities (actors) across a cluster. It is responsible for routing messages to the appropriate shard and, ultimately, to the specific entity (actor) within the shard, regardless of where the entity resides in the cluster.

This allows location transparency and dynamic scaling of actors across the cluster while simplifying actor management.

Role of Shard Region in Cluster Sharding :

1. Message Routing:

   • The Shard Region serves as the main entry point for messages sent to sharded entities.
   • It determines which shard an entity belongs to using a shard ID and then routes the message to the appropriate shard.
   • If the entity for the message doesn't exist yet, the Shard Region creates it automatically.

2. Shard Management:

   • The Shard Region oversees the lifecycle of shards.
   • It ensures that shards are evenly distributed across cluster nodes and manages shard rebalancing when nodes join or leave the cluster.

3. Entity Lifecycle Management:

   • The Shard Region ensures that entities are created and terminated based on activity and lifecycle policies (e.g., passivation when entities are idle for a specific time).

4. Fault Tolerance:

   • If a node hosting a shard fails, the Shard Region detects the failure and relocates the shard to another available node in the cluster.

5. Location Transparency:

   • Actors and clients can send messages to entities without knowing their physical location in the cluster. The Shard Region handles the underlying routing transparently.

6. Scaling and Rebalancing:

   • The Shard Region dynamically adjusts shard distribution to ensure load balancing when new nodes join the cluster or existing nodes leave.

Akka Distributed Data is a module in Akka that provides a way to manage eventually consistent, replicated, and distributed data across nodes in a cluster. It is built on the foundation of Conflict-Free Replicated Data Types (CRDTs), which are data structures designed to resolve conflicts automatically when data is updated concurrently on different nodes.

This module is particularly useful in scenarios where strong consistency isn't required, but availability and partition tolerance are critical.

Key Features of Akka Distributed Data :

1. Eventual Consistency:

   • Updates to data on different nodes are eventually propagated and resolved across the cluster.

2. Conflict Resolution:

   • CRDTs handle conflict resolution automatically using predefined merge strategies (e.g., merging sets or counters).

3. High Availability:

   • The data is available for read and write operations even during network partitions.

4. Scalability:

   • Data is replicated across multiple nodes, and the system scales as the cluster grows.

5. Distributed State Management:

   • Ideal for scenarios where a shared, distributed state is needed, such as distributed caches, collaborative tools, or leaderboards.

6. Immutable and Persistent Data:

   • Data changes are propagated as immutable deltas, ensuring safe and consistent replication.

What are CRDTs?

Conflict-Free Replicated Data Types (CRDTs) are special data structures designed for distributed systems. They allow for concurrent updates from multiple nodes and ensure eventual consistency by resolving conflicts automatically without requiring coordination between nodes.

Common CRDT Types in Akka Distributed Data :

1. Counters:

   • GCounter (Grow-only Counter): Can only be incremented.
   • PNCounter (Positive-Negative Counter): Can be incremented and decremented.

2. Sets:

   • GSet (Grow-only Set): Allows adding elements but not removing them.
   • ORSet (Observed-Removed Set): Supports adding and removing elements.

3. Maps:

   • ORMap (Observed-Removed Map): A map of CRDTs that supports adding, updating, and removing entries.

4. Registers:

   • LWWRegister (Last-Write-Wins Register): Stores a single value with timestamp-based conflict resolution.

5. Flags:

   • Flag: A boolean flag that can be switched from false to true (one-way toggle).

A split-brain scenario in an Akka Cluster occurs when a network partition divides the cluster into two or more isolated sub-clusters that are unable to communicate with each other. Each partition may still function independently, but they are unaware of the other partitions. This situation can lead to inconsistent state, duplicate work, or data corruption, as multiple nodes may assume leadership or perform conflicting tasks simultaneously.

Why Does Split-Brain Happen?

Split-brain can occur due to:

1. Network Failures:
   • Temporary loss of communication between nodes in different partitions.
2. Infrastructure Issues:
   • Faulty switches, routers, or firewalls can block communication.
3. High Latency:
   • Delays in message delivery can be mistaken as node failures.
4. Node Crashes or Failures:
   • Unexpected failures can cause cluster instability.

Challenges of Split-Brain :

1. Duplicate Leadership:

   • Multiple sub-clusters may elect their own leaders, causing conflicting decisions.

2. Inconsistent State:

   • Sub-clusters may maintain diverging views of the system state.

3. Data Corruption:

   • Write operations performed independently by each partition can result in data inconsistencies.

4. Service Downtime:

   • Resolving split-brain can disrupt normal operations if improperly handled.

Akka provides mechanisms to detect and resolve split-brain scenarios automatically. These include:

1. Failure Detection :

• Akka Cluster uses heartbeat messages and a failure detector (based on the Phi Accrual Failure Detector) to monitor node availability.
• Nodes are marked as unreachable if they stop responding to heartbeats.

2. Split-Brain Resolver (SBR) :

• The Split-Brain Resolver (SBR) is a built-in Akka module that resolves split-brain situations by deciding which partition should survive.
• The SBR can terminate nodes in smaller or less-preferred partitions to restore the cluster to a healthy state.