Updating tickets (#769, #774, #875, #889, #917, #920, #928, #929, #930, #931, #932, #933, #934, #935, #936, #938, #939, #940, #941, #942, #943, #944, #951, #958, #959, #960, #962, #630, #870, #891, #895)

I think the following solution currently is best under the circumstances.

The 'release' function is complete; it removes the actor from the cluster administration. And for the time being that is good enough, although in the future this also should
be self healing. You can not depend on a clean shutdown of a node.. all nodes are able to fail for whatever reason, so all their 'volatile' data they stored in the zk server needs to
be cleaned as well. Some kind of heartbeat based garbage collection would do the trick here.

What needs to be done is to enhance the clustered ref so that it deals with failing calls to a remote node. When it does a call to the node containing the actor,
and the call fails, the clustered ref can update its internals. In case of a round-robin/random based routing (so multiple instances), it can just kick out the dead node
and do its work using one node less. As soon as there are no nodes, you can throw some kind of error.

The advantage is that the ref is 'self healing', and doesn't depend on the rest of the world telling him what to do. Imagine the (distributed) currency problems otherwise you need to deal with.

In case of the direct node I would currently thrown an error since there is no node to fall over to. In the future, the cluster needs to be notified of this and a the actor can be reconstructed using the durable
mailbox.

What also can be useful in the future is to have some timeout mechanism here.. as soon as a node doesn't return its request in 60seconds for example, it is considered dead.

Sounds great. Lets do it like that. Can you do it?

I already did :)

But I wanted to verify that I'm on the right direction.

Personally I think that
- removal of nodes
- failover (when is something seen as a falure??)
- addition of other nodes

Are extremely close together. So it has no use to spend a terrible amount of time on one part. I would rather see one bigger task with 3 subgoals.

Viktor and your initial problem description suggest another solution however. Let the clustered ref listen to changes in its target actor and update its internal infrastructure information correspondingly.

But apart from all the listening, it also introduces various other problems. It could be that a node receives events multiple times, never, or out of order, e.g.

Node 1 goes down.
Node 1 goes up.

It could be that this is processed as:
Node 1 goes up (so either exception or ignore it..idempotency)
Node 1 goes down (remove stuff from internal state).

The problem here now is that a cluster ref believes node 1 is down, even though it still is up, just because the events where processed out of order.

So for the time being, this self healing stuff is the simplest to implement with the least amount of problems. As soon as the other stuff (node addition etc) needs to be implemented, we need to revise this code anyway.

Currently, sending a message to another node is a synchronous process. So you know.. if you didn't get a remoting error, you call must have made it to the other machine. In the future, this could also be an asynchronous process, meaning that you don't get any guarantee about if the call to the other node actually worked. This is something viktor pointed out. But personally I would deal with it when we get this form of asynchronous remoting.

I agree. This is the best approach. Most general.
Good thinking, thanks for taking a step back to look at the bigger picture. Very good.
Dynamic node addition is something that is scheduled for 2.0

https://github.com/jboner/akka/commit/96cc0a00b42fe6d428c3791c7b596a7df334a678

Reviewed!

All the review comments where already resolved. So this commit doesn't contain anything interesting (only an import cleanup).

https://github.com/jboner/akka/commit/a5ca2bad155ded81151d35a01ce7478d6c270891

If you're done you can close this

Updating tickets (#818, #821, #823, #836, #842, #854, #856, #865, #866, #867, #868, #869, #871, #872, #873, #877, #878, #879, #885, #888, #889, #890, #894, #910, #911, #917, #923, #924, #925, #926, #927, #928, #930, #931, #932, #933, #934, #936, #937, #952, #955, #957, #958, #959, #960, #961, #964, #965, #966)