Currently we only track trap exit when sending the reply message after a ? message.
Clustered actor has two parts:
- The client/ClusterActorRef on the "client" cluster node
- The remote instance(s) (that the client talks to) on another cluster node
Part 1 - The remote instance side of things
1. Create a hidden "system" Supervisor actor on the cluster node
2. Link the remote actor instance to this Supervisor
3. ZK is used to set a Watch (ephemeral node) for linked actor
4. The path for this Watch is send back to the ClusterActorRef that represents this instance (see part 2 for how that is managed on in the ClusterActorRef)
5. Now when remote actor fails the Watch should be removed (which will trigger the ClusterActorRef on the client side)
6. ... wait for ClusterActorRef to do its job
7. When Supervisor receives a Restart message from the ClusterActorRef then it sends an Exit message to its linked actor to restart the remote actor.
Part 2 - The client/ClusterActorRef side of things
1. The ClusterActorRef receives the path to the Watch from the remote instance
2. It then adds a Listener to this path (Watch)
3. When Watch is triggered, notifying our Listener then we send the remote actor a Restart message to the "system" Supervisor on this node (go to part 1 item 7)
Use the RemoteDaemonActor channel (in ClusterNode) for all "system" communication. Add necessary commands to the ClusterProtocol.proto.
All life-cycle messages (Restart, Exit, etc.) should be sent to a the EventHandler.